#AuroraPostgreSQL

[govindhtech]

Aurora PostgreSQL Limitless Database: Unlimited Data Growth

Aurora PostgreSQL Limitless Database

The new serverless horizontal scaling (sharding) feature of Aurora PostgreSQL Limitless Database, is now generally available.

With Aurora PostgreSQL Limitless Database, you may distribute a database workload across several Aurora writer instances while still being able to utilize it as a single database, allowing you to extend beyond the current Aurora restrictions for write throughput and storage.

During the AWS re:Invent 2023 preview of Aurora PostgreSQL Limitless Database, to described how it employs a two-layer architecture made up of several database nodes in a DB shard group, which can be either routers or shards to grow according to the demand.Image Credit To AWS

Routers:��Nodes known as routers receive SQL connections from clients, transmit SQL commands to shards, keep the system consistent, and provide clients with the results.

Shards: Routers can query nodes that hold a fraction of tables and complete copies of data.

Your data will be listed in three different table types: sharded, reference, and standard.

Sharded tables: These tables are dispersed among several shards. Based on the values of specific table columns known as shard keys, data is divided among the shards. They are helpful for scaling your application’s biggest, most I/O-intensive tables.

Reference tables: These tables eliminate needless data travel by copying all of the data on each shard, allowing join queries to operate more quickly. For reference data that is rarely altered, such product catalogs and zip codes, they are widely utilized.

Standard tables: These are comparable to standard PostgreSQL tables in Aurora. To speed up join queries by removing needless data travel, standard tables are grouped together on a single shard. From normal tables, sharded and reference tables can be produced.

Massive volumes of data can be loaded into the Aurora PostgreSQL Limitless Database and queried using conventional PostgreSQL queries after the DB shard group and your sharded and reference tables have been formed.

Getting started with the Aurora PostgreSQL Limitless Database

An Aurora PostgreSQL Limitless Database DB cluster can be created, a DB shard group added to the cluster, and your data queried via the AWS Management Console and AWS Command Line Interface (AWS CLI).

Establish a Cluster of Aurora PostgreSQL Limitless Databases

Choose Create database when the Amazon Relational Database Service (Amazon RDS) console is open. Select Aurora PostgreSQL with Limitless Database (Compatible with PostgreSQL 16.4) and Aurora (PostgreSQL Compatible) from the engine choices.Image Credit To AWS

Enter a name for your DB shard group and the minimum and maximum capacity values for all routers and shards as determined by Aurora Capacity Units (ACUs) for the Aurora PostgreSQL Limitless Database. This maximum capacity determines how many routers and shards are initially present in a DB shard group. A node’s capacity is increased by Aurora PostgreSQL Limitless Database when its present utilization is insufficient to manage the load. When the node’s capacity is greater than what is required, it reduces it to a lower level.Image Credit To AWS

There are three options for DB shard group deployment: no compute redundancy, one compute standby in a different Availability Zone, or two compute standbys in one Availability Zone.

You can select Create database and adjust the remaining DB parameters as you see fit. The DB shard group appears on the Databases page when it has been formed.Image Credit To AWS

In addition to changing the capacity, splitting a shard, or adding a router, you can connect, restart, or remove a DB shard group.

Construct Limitless Database tables in Aurora PostgreSQL

As previously mentioned, the Aurora PostgreSQL Limitless Database contains three different types of tables: standard, reference, and sharded. You can make new sharded and reference tables or convert existing standard tables to sharded or reference tables for distribution or replication.

By specifying the table construction mode, you can use variables to create reference and sharded tables. Until a new mode is chosen, the tables you create will use this mode. The examples that follow demonstrate how to create reference and sharded tables using these variables.

For instance, make a sharded table called items and use the item_id and item_cat columns to build a shard key.SET rds_aurora.limitless_create_table_mode='sharded'; SET rds_aurora.limitless_create_table_shard_key='{"item_id", "item_cat"}'; CREATE TABLE items(item_id int, item_cat varchar, val int, item text);

Next, construct a sharded table called item_description and collocate it with the items table. The shard key should be made out of the item_id and item_cat columns.SET rds_aurora.limitless_create_table_collocate_with='items'; CREATE TABLE item_description(item_id int, item_cat varchar, color_id int, ...);

Using the rds_aurora.limitless_tables view, you may obtain information on Limitless Database tables, including how they are classified.SET rds_aurora.limitless_create_table_mode='reference'; CREATE TABLE colors(color_id int primary key, color varchar);

It is possible to transform normal tables into reference or sharded tables. The source standard table is removed after the data has been transferred from the standard table to the distributed table during the conversion. For additional information, see the Amazon Aurora User Guide’s Converting Standard Tables to Limitless Tables section.postgres_limitless=> SELECT * FROM rds_aurora.limitless_tables; table_gid | local_oid | schema_name | table_name | table_status | table_type | distribution_key -----------+-----------+-------------+-------------+--------------+-------------+------------------ 1 | 18797 | public | items | active | sharded | HASH (item_id, item_cat) 2 | 18641 | public | colors | active | reference | (2 rows)

Run queries on tables in the Aurora PostgreSQL Limitless Database

The Aurora PostgreSQL Limitless Database supports PostgreSQL query syntax. With PostgreSQL, you can use psql or any other connection tool to query your limitless database. You can use the COPY command or the data loading program to import data into Aurora Limitless Database tables prior to querying them.

Connect to the cluster endpoint, as indicated in Connecting to your Aurora Limitless Database DB cluster, in order to execute queries. The router to which the client submits the query and shards where the data is stored is where all PostgreSQL SELECT queries are executed.

Two querying techniques are used by Aurora PostgreSQL Limitless Database to accomplish a high degree of parallel processing:

Single-shard queries and distributed queries. The database identifies whether your query is single-shard or distributed and handles it appropriately.

Single-shard queries: All of the data required for the query is stored on a single shard in a single-shard query. One shard can handle the entire process, including any created result set. The router’s query planner forwards the complete SQL query to the appropriate shard when it comes across a query such as this.

Distributed query: A query that is executed over many shards and a router. One of the routers gets the request. The distributed transaction, which is transmitted to the participating shards, is created and managed by the router. With the router-provided context, the shards generate a local transaction and execute the query.

To configure the output from the EXPLAIN command for single-shard query examples, use the following arguments.postgres_limitless=> SET rds_aurora.limitless_explain_options = shard_plans, single_shard_optimization; SET postgres_limitless=> EXPLAIN SELECT * FROM items WHERE item_id = 25; QUERY PLAN -------------------------------------------------------------- Foreign Scan (cost=100.00..101.00 rows=100 width=0) Remote Plans from Shard postgres_s4: Index Scan using items_ts00287_id_idx on items_ts00287 items_fs00003 (cost=0.14..8.16 rows=1 width=15) Index Cond: (id = 25) Single Shard Optimized (5 rows)

You can add additional items with the names Book and Pen to the items table to demonstrate distributed queries.postgres_limitless=> INSERT INTO items(item_name)VALUES ('Book'),('Pen')

A distributed transaction on two shards is created as a result. The router passes the statement to the shards that possess Book and Pen after setting a snapshot time during the query execution. The client receives the outcome of the router’s coordination of an atomic commit across both shards.

The Aurora PostgreSQL Limitless Database has a function called distributed query tracing that may be used to track and correlate queries in PostgreSQL logs.

Important information

A few things you should be aware of about this functionality are as follows:

Compute: A DB shard group’s maximum capacity can be specified between 16 and 6144 ACUs, and each DB cluster can only have one DB shard group. Get in touch with us if you require more than 6144 ACUs. The maximum capacity you provide when creating a DB shard group determines the initial number of routers and shards. When you update a DB shard group’s maximum capacity, the number of routers and shards remains unchanged.

Storage: The only cluster storage configuration that Aurora PostgreSQL Limitless Database offers is Amazon Aurora I/O-Optimized DB. 128 TiB is the maximum capacity of each shard. For the entire DB shard group, reference tables can only be 32 TiB in size.

Monitoring: PostgreSQL’s vacuuming tool can help you free up storage space by cleaning up your data. Aurora PostgreSQL Limitless Database monitoring can be done with Amazon CloudWatch, Amazon CloudWatch Logs, or Performance Insights. For monitoring and diagnostics, you can also utilize the new statistics functions, views, and wait events for the Aurora PostgreSQL Limitless Database.

Available now

PostgreSQL 16.4 works with the AWS Aurora PostgreSQL Limitless Database. These regions are included: Asia Pacific (Hong Kong), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), US East (N. Virginia), US East (Ohio), and US West (Oregon). Try the Amazon RDS console’s Aurora PostgreSQL Limitless Database.

Read more on Govindhtech.com

[globalmediacampaign]

Recap of Amazon RDS and Aurora features launched in 2019

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity. At the same time, it automates time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups. It frees you to focus on your applications so you can give them the fast performance, high availability, security, and compatibility they need. Moving self-managed databases to managed database services is the new norm, and we’re continuing to add features to Amazon RDS at a rapid pace. 2019 was a busy year, so let’s do a recap of the features launched across the different database engines. Amazon RDS first launched back in October of 2009, over 10 years ago! We started with Amazon RDS for MySQL; since then we’ve reached a total of seven database engine options: Amazon Aurora MySQL, Amazon Aurora PostgreSQL, PostgreSQL, MySQL, MariaDB, Oracle Database, and Microsoft SQL Server. In 2019, we launched over 100 features across all Amazon RDS and Aurora database engines. For a quick reference, visit the 2018 recap, and the 2017 recap. We’ll start by covering each database engine and the key releases that we think will have the biggest impact on your database strategy and operations. We then list all of the features that we launched in 2019, categorized for convenience: New instance types, Regions, and versions– Providing you with a variety of database deployment options Manageability – Simplifying database management and providing expert recommendations Developer productivity – Enabling builders to focus on tasks meaningful to the business Performance – Improving database performance and scale to meet the application’s needs Availability/Disaster Recovery – Deploying highly available databases across Availability Zones and AWS Regions Security – Facilitating secure database operation Key feature launches of 2019 Amazon Aurora MySQL Aurora Global Database, originally launched at re:Invent 2018, expands your database into multiple Regions for disaster recovery and faster global reads. In 2019, this feature gained support for up to five secondary Regions, MySQL 5.7, and in-place upgrades from single-region databases. The other two key features that we launched for Aurora MySQL in 2019 were Aurora Multi-Master and Aurora Machine Learning. Aurora Multi-Master increases availability by enabling you to create multiple read/write instances of your Aurora database across multiple Availability Zones. Aurora Machine Learning lets you add machine learning (ML) based predictions to your applications directly in your SQL queries. Read more on the AWS News Blog. Amazon Aurora PostgreSQL For Aurora PostgreSQL, we announced support for Serverless, where the database automatically starts up, shuts down, and scales capacity up or down based on your application’s needs. Aurora Serverless benefits use cases such as infrequently used applications, new applications, variable workloads, unpredictable workloads, development and test databases, and multi-tenant applications. We also launched support for Logical Replication using PostgreSQL replication slots, enabling you to use the same logical replication tools that they use with RDS for PostgreSQL. We launched support for Database Activity Streams to provide detailed auditing information in an encrypted JSON stream, Cluster Cache Management to resume the previous database performance after a failover, S3 import to make it easy and fast to load data from CSV files (S3 export was added in early 2020), export of logs to CloudWatch to make it easy to monitor PostgreSQL logs, support for PostgreSQL version 11 to give you access to the latest features from the PostgreSQL community, and FedRAMP HIGH compliance. Amazon RDS for Oracle We improved availability and disaster recovery by launching In-Region and Cross-Region Read Replicas using Oracle Active Data Guard. With read replicas, you can easily create up to five fully managed Oracle Active Data Guard standby databases that can be used for read scaling and offloading of long-running analytical queries. You can create read replicas in the same Region or a different Region from the primary instance, and replicas can be promoted into full read/write databases for disaster recovery purposes. Last year, we also simplified migrations to RDS for Oracle with Amazon S3 Integration for Data Ingress and Egress Capabilities. With the S3 Integration option, you can easily set up fast and secure file transfer between an Amazon RDS for Oracle instance and Amazon Simple Storage Service (S3), significantly reducing the complexity of loading and unloading data. Amazon RDS for SQL Server By increasing the maximum number of databases per database instance from 30 to 100, we enable you to further consolidate database instances to save on costs. Another exciting enhancement was around migrations. When some of our customers perform Native Backups and Restores when migrating to RDS SQL Server, they sometimes experience longer downtime during the final stages of the migration process than they would prefer. With support for Native Differential and Log Backups in conjunction with Full Native Backups, you can reduce downtime to as little as 5 minutes. Last but not least, we launched Always On Availability Groups for SQL Server 2017 Enterprise Edition. With Always On, we also launched the Always On Listener Endpoint, which supports faster failover times. Releases across multiple Amazon RDS database engines We expanded deployment options for Amazon RDS by launching Amazon RDS on VMware for the following database engines: Microsoft SQL Server, MySQL, and PostgreSQL. If you need to run a hybrid (cloud and on-premises) database environment, this gives you the option to use the automation behind Amazon RDS in an on-premises VMware vSphere environments. Jeff Barr wrote a detailed AWS News Blog post highlighting the available features and how to get started. For simplified single sign-on, you can use Microsoft Active Directory (AD) via AWS Managed Active Directory Service for Amazon RDS PostgreSQL, RDS Oracle, and RDS MySQL (AD is also supported on SQL Server, and was launched on MySQL in early 2020). Now you can use the same AD for different VPCs within the same AWS Region. You can also join instances to a shared Active Directory domain owned by different accounts. Lastly, we announced the public preview of Amazon RDS Proxy for Amazon RDS MySQL and Aurora MySQL. As its name implies, Amazon RDS Proxy sits between your application and its database to pool and share database connections, improving database efficiency and application scalability. In case of a database failover, Amazon RDS Proxy automatically connects to a standby database instance while preserving connections from your application and reducing failover times for Amazon RDS and Aurora Multi-AZ databases by up to 66%. Lastly, database credentials and access can be managed through AWS Secrets Manager and AWS Identity and Access Management (IAM), eliminating the need to embed database credentials in application code. Support for Amazon RDS for PostgreSQL and Amazon Aurora with PostgreSQL compatibility is coming soon. You can learn more by reading Using Amazon RDS Proxy with AWS Lambda. Features by database engine Amazon Aurora MySQL New instances types, Regions, and versions Launched the T3, and R5 instance family Launched support for R5.8xl, R5.16xl, and R5.24xl instances Available in the AWS Americas (São Paulo) Region Aurora Global Database Expands Availability to 14 AWS Regions Amazon Aurora Serverless with MySQL Compatibility available in AWS China (Ningxia) Region, Operated by NWCD Expanded List of Supported Features to Improve Performance and Manageability Manageability Key 2019 launch: Multi-Master is Generally Available Aurora Serverless Supports Capacity of 1 Unit and a New Scaling Option Support for Cloning Across AWS Accounts Support for Cost Allocation Tags for Aurora Storage Support for GTID-Based Replication for MySQL 5.7 Developer productivity Key 2019 launch: Aurora Supports Machine Learning Directly from the Database Amazon Aurora Serverless MySQL 5.6 Now Supports Data API Faster Migration from MySQL 5.7 Databases to Amazon Aurora for MySQL Amazon RDS Recommendations Provide Best Practice Guidance for Amazon Aurora Performance Performance Insights Availability/Disaster Recovery Aurora Global Database is Supported on Aurora MySQL 5.7 Aurora Global Database Supports Multiple Secondary Regions Aurora Supports In-Place Conversion to Global Database Amazon Aurora Serverless Supports Sharing and Cross-Region Copying of Snapshots Support for Zero-Downtime Patching Security Aurora Serverless Publishes Logs to Amazon CloudWatch Aurora Serverless Clusters can Launch in a Shared VPC Aurora is FedRAMP-High Compliant on AWS GovCloud (US) Amazon Aurora PostgreSQL New instances types, Regions, and versions Aurora PostgreSQL 2.2, compatible with PostgreSQL 10.6 Aurora PostgreSQL 2.3, compatible with PostgreSQL 10.7 Aurora PostgreSQL 3.0, compatible with PostgreSQL 11.4 Available in the AWS Americas (São Paulo) Region Support for R5, and Medium Instance Types Manageability Copy Tags from an Aurora PostgreSQL Cluster to a Database Snapshot Aurora Supports Cost Allocation Tags for Aurora Storage Support for Cloning Across AWS Accounts Aurora Serverless Now Supports Data API Developer productivity Key 2019 launch: Amazon Aurora with PostgreSQL Compatibility Supports Logical Replication Amazon RDS Recommendations Provide Best Practice Guidance for Amazon Aurora Support for Data Import from Amazon S3 Performance Support for Cluster Cache Management Performance Insights Support for SQL-level Metrics Availability/Disaster Recovery Key 2019 launch: Support for AuroraPostgreSQL Serverless Security Database Activity Streams for Real-time Monitoring Restore an Encrypted Aurora PostgreSQL Database from an Unencrypted Snapshot FedRAMP-High Compliant on AWS GovCloud (US) Aurora Serverless Clusters can Launch in a Shared VPC Publishing PostgreSQL Log Files to Amazon CloudWatch Logs Amazon RDS for MySQL/MariaDB New Instances Types, Regions, and Versions MySQL and MariaDB Support the R5 and T3 Instance Types Support for MySQL Versions 5.7.25, 5.7.24, and MariaDB Version 10.2.21 Compatibility Checks for Upgrades from MySQL 5.7 to MySQL 8.0 MySQL Support for Minor Version 8.0.16 Manageability Performance Insights is Generally Available for MariaDB Developer productivity Key 2019 launch: Introducing Amazon RDS Proxy (Preview) Performance Amazon RDS for MySQL and Maria DB increases maximum storage size to 64 TiB and maximum I/O performance to 80,000 IOPS Amazon Aurora and Amazon RDS Enable Faster Migration from MySQL 5.7 Databases Performance Insights is Generally Available on Amazon RDS for MariaDB Security Amazon RDS for MySQL Supports Password Validation Amazon RDS for PostgreSQL New instances types, Regions, and versions PostgreSQL 11 now Supported in Amazon RDS Supports T3 Instance Types Support for Minor Versions 11.2, 10.7, 9.6.12, 9.5.16, and 9.4.21 and Minor Versions 11.4, 10.9, 9.6.14, 9.5.18, and 9.4.23 Support for PostgreSQL 12 Beta 2 version in Preview Support for PostgreSQL 11 and Minor Versions 11.4, 10.9, 9.6.14, 9.5.18, and 9.4.23 AWS GovCloud (US) Regions Support for PG 12 Beta 3 in Preview Support for Minor Versions 11.5, 10.10, 9.6.15, 9.5.19, 9.4.24, adds Transportable Database Feature Support for Minor Versions 11.5 and 10.10, adds Transportable Database Feature in AWS GovCloud (US) Regions Manageability Amazon RDS for PostgreSQL Supports Customer Initiated Snapshot Upgrades Amazon RDS for PostgreSQL Now Supports Multi Major Version Upgrades to PostgreSQL 11 Developer productivity Support for Data Import from Amazon S3 Performance Amazon RDS for PostgreSQL increases maximum storage size to 64 TiB and I/O performance to 80,000 IOPS Security Key 2019 launch: Amazon RDS for PostgreSQL Supports User Authentication with Kerberos and Microsoft Active Directory Amazon RDS for Oracle New instances types, Regions, and versions Support for January 2019 Oracle Patch Set Updates (PSU) and Release Updates (RU), April PSU and RU, July 2019 PSU and RU Support for Oracle Database 18c and Oracle Database 19c Support for October 2019 Oracle Patch Set Updates (PSU) and Release Updates (RU) Support for SQLT Diagnostics Tool Version 12.2.180725 Support for Oracle Application Express (APEX) Versions 18.1 and 18.2 and Version 19.1 Support for Oracle Management Agent (OMA) version 13.3 for Oracle Enterprise Manager Cloud Control 13c Support for T3 and Z1d Instance Types Support for M5, T3, and R5 Instances Types are now available in the AWS China (Ningxia) Region, operated by NWCD, and the AWS China (Beijing) Region, operated by Sinnet Support for new Instance Sizes (R5/M5 8xl/16xl) Support for Z1d, X1, and X1e Instance Types in additional Regions Manageability Key 2019 launch: Support for Amazon S3 Integration For Data Ingress and Egress Capabilities Developer productivity Support for Oracle adds support to invoke EMCTL commands for Oracle Enterprise Manager Cloud Control Support for ALLOWED_LOGON_VERSION_SERVER and ALLOWED_LOGON_VERSION_CLIENT sqlnet.ora Parameters Performance Amazon RDS for Oracle increases maximum storage size to 64 TiB and I/O performance to 80,000 IOPS Amazon RDS Performance Insights Supports SQL-level Metrics on Amazon RDS for Oracle Availability/Disaster Recovery Key 2019 launch: Amazon RDS for Oracle Now Supports In-region Read Replicas with Active Data Guard for Read Scalability and Availability Key 2019 launch: Amazon RDS for Oracle Now Supports Managed Disaster Recovery and Data Proximity with Cross-region Read Replicas Security Key 2019 launch: Support for User Authentication with Kerberos and Microsoft Active Directory Amazon RDS for SQL Server New instances types, Regions, and versions High Availability support in additional AWS Regions Supports for R5 and T3, X1, and X1e Instance Types Support for additional Instance Sizes (M5/R5 8xl/16xl) Manageability Key 2019 launch: Increased the Database Limit Per Database Instance to up to 100 Key 2019 launch: Support for Differential Restores and Log Restores Support for Multi-File Native Restores Supports changing the server-level collation Developer productivity Support for Outbound Network Access Performance Support for Performance Insights Availability/Disaster Recovery Key 2019 launch: Support for Always On Availability Groups for SQL Server 2017 Security Publish log files to Amazon CloudWatch Support for SQL Server Audit Across Amazon RDS database engines New instances types, Regions, and versions Key 2019 launch: Amazon RDS on VMware is now generally available AWS Asia Pacific (Hong Kong) Region AWS Middle East (Bahrain) Manageability Amazon RDS now supports per-second billing Amazon RDS now supports Storage Auto Scaling Amazon RDS Enables Detailed Backup Storage Billing Performance Performance Insights Now Supports Counter Metrics on Amazon RDS for MySQL, RDS for PostgreSQL, Aurora MySQL, Amazon RDS for SQL Server, and Amazon RDS for Oracle Performance Insights Supports T2 and T3 Instance Types Security Amazon RDS Enhanced Monitoring Adds New Storage and Host Metrics Summary While the last 10 years have been extremely exciting, it is still Day 1 for our service and we are excited to keep innovating on behalf of our customers! If you haven’t tried Amazon RDS yet, you can try it for free via the Amazon RDS Free Tier. If you have any questions, feel free to comment on this blog post!   About the Authors   Justin Benton is a Sr Product Manager at Amazon Web Services.         Yoav Eilat is a Sr Product Manager at Amazon Web Services.         https://probdm.com/site/MTkxMDU

[ericvanderburg]

SD Times news digest: Flutter 1.7, Amazon AuroraPostgreSQL Serverless, and WhiteSource and Codota partner on open-source security vulnerabilities

http://i.securitythinkingcap.com/R88pRg

[govindhtech]

Aurora PostgreSQL zero-ETL Integration With Amazon Redshift

The general availability of Amazon Aurora PostgreSQL and Amazon DynamoDB zero-ETL integrations with Amazon Redshift.

The Amazon Aurora PostgreSQL-Compatible Edition zero-ETL integrations with Amazon Redshift are now generally available. By eliminating the need to create and maintain intricate data pipelines that carry out extract, transform, and load (ETL) activities, zero-ETL integration effortlessly makes transactional or operational data available in Amazon Redshift. It updates source data for you to use in Amazon Redshift for analytics and machine learning (ML) skills to extract timely insights and efficiently respond to important, time-sensitive events while automating source data replication to Amazon Redshift.

With these new zero-ETL integrations, you can conduct unified analytics on your data from various applications, eliminating the need to create and maintain separate data pipelines to write data from multiple relational and non-relational data sources into a single data warehouse.

Amazon Redshift is the target and a source is specified in order to construct a zero-ETL integration. The integration keeps an eye on the pipeline’s condition while replicating data from the source to the target data warehouse and making it easily accessible in Amazon Redshift.

Amazon Redshift integration of Aurora PostgreSQL zero-ETL

Near real-time analytics on petabytes of transactional data are made possible by the integration of Amazon Redshift and Amazon Aurora zero-ETL.

Why Aurora zero-ETL integration with Amazon Redshift?

Near real-time analytics and machine learning (ML) on petabytes of transactional data are made possible by the integration of Amazon Redshift with Amazon Aurora zero-ETL. Zero-ETL eliminates the need to create and maintain intricate data pipelines that carry out extract, transform, and load (ETL) activities by effortlessly making transactional data available in Amazon Redshift a few seconds after it was entered into Amazon Aurora.

Advantages

Access to data in almost real time

Run near real-time analytics and machine learning on petabytes of data by accessing transactional data from Aurora in Amazon Redshift in a matter of seconds.

Simple to use

Without having to create and maintain ETL pipelines to transfer transactional data to analytics platforms, you can quickly examine your transactional data in almost real time.

Smooth integration of data

To perform unified analytics across numerous apps and data sources, combine several tables from different Aurora database clusters and replicate your data to a single Amazon Redshift data warehouse.

Absence of infrastructure management

Using both Amazon Redshift Serverless and Amazon Aurora Serverless v2, you can do analytics on transactional data in almost real-time without managing any infrastructure.

Use cases

Operational analytics in near real time

To effectively respond to important, time-sensitive events, use Amazon Redshift analytics and machine learning capabilities to extract insights from transactional and other data in almost real-time. For use cases like fraud detection, data quality monitoring, content targeting, better gaming experiences, and customer behavior research, near real-time analytics can help you obtain more precise and timely insights.

Large-scale analytics

Petabytes of your transactional data pooled from several Aurora database clusters can be analyzed using Amazon Redshift’s capabilities thanks to the Aurora zero-ETL connector. You can benefit from Amazon Redshift’s extensive analytical features, including federated access to numerous data stores and data lakes, materialized views, built-in machine learning, and data sharing. With Amazon Redshift ML’s native integration into Amazon SageMaker, you can use simple SQL queries to generate billions of predictions.

lessen the operational load

It is frequently necessary to create, oversee, and run a sophisticated data pipeline ETL system in order to move data from a transactional database into a central data warehouse. You may easily transfer the schema, current data, and data modifications from your Aurora database to a new or existing Amazon Redshift cluster with a zero-ETL integration. Complex data pipeline management is no longer necessary with zero-ETL integration.

How to begin

You designate an Amazon Redshift data warehouse as the target and an Aurora DB cluster as the data source when creating your zero-ETL interface between Aurora and Redshift. Data from the source database is replicated into the target data warehouse through the integration. Within seconds, the data is accessible in Amazon Redshift, enabling data analysts to start utilizing the analytics and machine learning features of the platform.

Cost

Aurora zero-ETL integration with Amazon Redshift is free of charge via AWS. The change data produced by a zero-ETL integration is created and processed using pre-existing Aurora and Amazon Redshift resources, which you pay for. These resources could consist of:

By turning on change data capture, more I/O and storage are used.

For the first data export to seed your Amazon Redshift databases, the snapshot export costs

Extra Amazon Redshift storage for data replication

Extra Amazon Redshift computation for data replication processing

Cross-AZ data transfer fees for transferring data between sources and destinations.

Continuous data change processing via zero-ETL integration is provided at no extra cost. Please visit the Aurora price page for additional details.

Availability

The AWS regions for the US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm) now offer Aurora PostgreSQL zero-ETL integration with Amazon Redshift.

Read more on Govindhtech.com