AWS re:Invent 2025 - Universal data connectivity with ETL and SQL queries (ANT209)

Universal Data Connectivity with ETL and SQL Queries

Challenges with Fragmented Data Environments

• Organizations today struggle with data silos, where customer data, sales data, and other critical information is spread across disparate systems and locations.
• This makes it difficult and time-consuming to get a unified view of the business and answer basic questions like monthly sales performance or new product traction.
• The complexity increases as organizations need to converge analytics and ML workloads on the same data, accessed by different personas like data scientists, analysts, and engineers.

AWS's Unified Data Connectivity Approach

• AWS recognizes there is no one-size-fits-all solution, as different scenarios require different tools, control levels, and trade-offs between flexibility and simplicity.
• AWS provides a unified data connectivity strategy with purpose-built tools and solutions to address the needs of various user personas.

Empowering the Data Engineer: AWS Glue Connectors

• AWS Glue is a serverless data integration service that provides an all-in-one solution for data quality management, data cataloging, and supporting diverse user personas.
• AWS Glue Connectors offer over 100 pre-built connectors to AWS and third-party applications, allowing data engineers to easily ingest data from multiple sources.
  • Native connectors are managed by AWS, while custom and marketplace connectors provide flexibility.
  • Connectors handle API-level integration, pagination, and other complexities, so data engineers don't have to.
  • Glue also provides data quality checks, anomaly detection, and data aggregation capabilities.
• Example: Wire Medical used AWS Glue to transform hundreds of ETL jobs, achieving 50% cost savings and significantly improving developer productivity.

Simplifying Data Injection with AWS Zero ETL

• Alex, a data engineer, wants to keep his analytical data in sync with operational databases without added operational overhead.
• AWS Zero ETL is a managed service that allows ingesting data from multiple sources to target systems like Amazon Redshift or S3 without building complex ETL pipelines.
  • Zero ETL handles the initial data load, change data capture, schema translation, and error handling automatically.
  • It supports 23 data sources, including AWS databases, third-party SaaS apps, and self-managed databases.
  • Zero ETL can deliver data to targets within minutes, reducing time to insight.
• Example: Pinex, a crypto exchange, saw a 98% reduction in data latency, 80% reduction in maintenance costs, and 66% reduction in operational overhead by using Zero ETL.

Enabling Ad-Hoc Analysis with Federated Queries

• Alex also needs to handle ad-hoc requests from executives and analysts that require combining data from multiple sources quickly.
• AWS Federated Queries allow running SQL queries across disparate data sources, including relational databases, data lakes, and SaaS apps, without moving the data.
  • Federated Queries are available in Amazon Athena and Amazon Redshift, providing flexibility in the query engine.
  • The service supports over 30 connectors to access data where it resides, without replicating it.
• Example: Alex used Federated Queries in Amazon Athena to quickly join data from Amazon Redshift, Amazon RDS, and SAP to provide the requested customer insights.

Key Takeaways

• AWS provides a comprehensive, unified data connectivity strategy with purpose-built tools like Glue Connectors, Zero ETL, and Federated Queries.
• These solutions empower data engineers to simplify data pipelines, reduce operational overhead, and enable faster access to insights.
• Customers have seen significant benefits, including cost savings, improved developer productivity, reduced data latency, and better agility in responding to ad-hoc requests.
• The breadth of supported data sources, managed capabilities, and performance enhancements make AWS's data connectivity offerings a compelling choice for enterprises.