AWS re:Invent 2025 - What's new in Amazon Redshift and Amazon Athena (ANT206)

Summary of AWS re:Invent 2025 - What's new in Amazon Redshift and Amazon Athena (ANT206)

Overview of Customer Analytics Needs

• Customers have three main use cases for analytics:
  1. Storing and analyzing structured business data using a cloud data warehouse
  2. Analyzing vast amounts of raw unstructured or semi-structured data using a cloud data lake
  3. Combining data warehouse and data lake data to get richer insights

Amazon Redshift: Innovations and Enhancements

Cloud Data Warehouse Fundamentals

• Redshift continues to invest heavily in performance, security, and global availability to provide a robust cloud data warehouse platform.
• Key performance enhancements:
  • Materialized views with incremental auto-refresh, cascading refreshes, and support for shared data sets
  • Multi-dimensional data layouts (MDDL) that automatically optimize table layouts based on query patterns
• Security improvements:
  • Redshift clusters are now private by default, fully encrypted, and require SSL connections

Distributed Warehouse Architecture

• Redshift has introduced a "hub and spoke" or "data mesh" distributed warehouse architecture.
• This allows different analytics workloads (BI, ETL, real-time, etc.) to run on dedicated optimized clusters that share a common data set.
• Provides workload isolation, cost visibility, and secure data sharing between clusters.

Amazon Redshift Serverless Enhancements

• Trailing tracks for testing new releases before production deployment
• FedRAMP authorization for government and regulated industry use
• Support for 4 RPU instances for smaller workloads
• Serverless networking improvements for constrained environments
• Serverless reservations with up to 24% discounts

Apache Iceberg Support

• Redshift now supports reading and writing Iceberg tables natively.
• Provides 2x better price performance on Iceberg-based data lake queries compared to previous versions.
• Supports auto-refresh of materialized views built on Iceberg data.

Twilio's Use of Redshift and Athena

• Twilio rebuilt their billing engine on Redshift and Athena to address challenges with scale, flexibility, and analytics.
• Key architectural decisions:
  • Leveraged Redshift's serializable transactions and materialized views for real-time aggregation of billions of events.
  • Built a distributed data warehouse with multiple workgroups sharing data through live data shares.
  • Used dbt to manage complex pricing logic and provide transparency into invoice calculations.
  • Integrated Redshift with Aurora databases to capture full historical pricing changes.
• Outcomes:
  • 75% cost reduction compared to the previous system
  • 6-hour monthly billing recalculation reduced to 30 minutes
  • Enabled new analytical capabilities and pricing flexibility for the business

Amazon Athena Enhancements

Iceberg Performance Improvements

• Athena is now 1.5x faster on Iceberg tables compared to last year, through optimizations like enhanced Iceberg statistics handling.
• Also launched Parquet column indexing and Lake Formation optimizations to further improve Iceberg query performance.

Materialized Views for Athena

• Athena now supports managed Iceberg-based materialized views that are automatically updated when source data changes.
• Enables building pre-computed SQL pipelines that can be easily queried through Athena.

S3 Tables GA and Enhancements

• S3 Tables, Athena's fully managed Iceberg offering, went GA this year with expanded DDL support and a new console wizard.
• Also added "Create Table As Select" (CTAS) functionality to easily convert data formats.

Performance Optimizations

• Athena engine optimizations have improved performance for popular data formats like Parquet (1.2x), JSON (2x), and CSV (1.8x).
• Query result reuse caching has been enhanced to increase cache hit ratios.
• Memory management logic has been rewritten to improve stability for queries with heavy shuffling and joins.

Administrative Enhancements

Capacity Reservations

• Athena now offers capacity reservations to guarantee serverless compute for mission-critical or SLA-sensitive workloads.
• Includes new features for capacity and cost controls, as well as observability into DPU consumption.

Autoscaling

• Athena launched a serverless autoscaling solution that automatically adjusts capacity based on workload utilization.

Managed Query Results

• Athena can now automatically store and manage query result files, simplifying administration and lifecycle management.

Other Announcements

• SageMaker Notebooks integration with Athena SQL and Spark engines
• Trusted Identity Propagation (TIP) for end-to-end identity-based access and auditing
• Managed Compute Platform (MCP) servers for building autonomous data agents