AWS re:Invent 2025 - LSEG: Transforming market intelligence at massive scale (IND3305)

AWS re:Invent 2025 - LSEG: Transforming market intelligence at massive scale

Overview

• The London Stock Exchange Group (LSEG) is a leading global financial market infrastructure company delivering data to 44,000 clients in 170 countries.
• LSEG's real-time business processes 274 billion messages per day, ingesting data from 575 venues and exchanges globally.
• LSEG has been storing every trade tick since 1996, resulting in a 75 petabyte historical data set.
• The unprecedented scale and growth of modern financial markets has created significant challenges for LSEG's legacy on-premises infrastructure.

Business Drivers for Transformation

• Legacy on-premises infrastructure unable to keep up with exponential data growth and market volatility
• Customer demand for real-time insights and faster time-to-market for new products
• Cost pressures in competitive markets
• Need for increased innovation velocity to stay ahead of the market

LSEG's Transformation Journey

1. Historical Data Migration:

   • Migrated 75PB of tick history data from CSV files to more efficient Parquet format in Amazon S3
   • Enabled direct S3 access for clients to leverage the historical data

2. Real-Time Platform Development:

   • Developed a real-time optimized product leveraging AWS services for low-latency, high-throughput delivery
   • Deployed managed services like RTMDS in public cloud to scale infrastructure on-demand

3. Matrix Signals Platform:

   • Built a serverless, cloud-native solution to detect market hotspots and signals at scale
   • Leveraged AWS services like Amazon Athena, Amazon OpenSearch, and Amazon Bedrock

Technical Architecture

1. Transforming Trillions of Ticks to Billions of Optimized Data Points:

   • Used AWS Glue serverless ETL to aggregate tick-by-tick data from S3 into optimized time series formats
   • Leveraged S3 Access Points for zero-copy data access and cost-effective storage

2. Flexible Modeling and Insights Generation:

   • Queried the optimized data using Amazon Athena to build flexible models and detect market patterns
   • Automatically indexed Athena query results into Amazon OpenSearch for API-based access and integration

3. Leveraging Generative AI for Deeper Insights:

   • Integrated Amazon Bedrock to correlate market hotspots with relevant news and events using generative AI
   • Enabled faster, more contextual decision-making for operational teams

Key Outcomes and Benefits

• 5x reduction in infrastructure costs for signal detection and hotspot analysis
• 97% reduction in storage costs by leveraging S3 for historical data
• 82% reduction in infrastructure costs through serverless, cloud-native architecture
• 30-45 minute time-to-signal for market hotspots, down from 3 days previously
• Enabled new business capabilities like real-time optimized data products and managed services
• Improved customer experience through faster onboarding and on-demand capacity scaling
• Accelerated innovation cycles by leveraging the latest AWS services and technologies

Conclusion

LSEG's transformation journey demonstrates the power of a cloud-native, serverless architecture to tackle the challenges of massive scale and growth in the capital markets industry. By leveraging AWS services like Amazon S3, Amazon Athena, Amazon OpenSearch, and Amazon Bedrock, LSEG was able to build a highly scalable, cost-effective, and innovative market intelligence platform that delivers real-time insights and enables faster decision-making. This transformation has positioned LSEG for continued leadership in the market data aggregation and distribution business.