AWS re:Invent 2025 - How to prep Telemetry data for AI consumption (DVT222)

Summary of AWS re:Invent 2025 Presentation: "How to prep Telemetry data for AI consumption (DVT222)"

The AI for Telemetry Problem

• Observability data (logs, traces, metrics) has grown exponentially, but most of it is "noise" that is not useful for AI and analytics
• Feeding all this raw data into AI models leads to "garbage in, garbage out" - the signal is lost in the noise
• The key challenge is being able to effectively denoise and concentrate the observability data to extract the valuable signal

Greer: Automated Observability Data Preparation

• Greer is a solution that sits between observability data sources and downstream AI/analytics systems
• It automatically analyzes the observability data in real-time to:
  • Identify patterns and extract the key signals
  • Compress and summarize the data to reduce volume while maintaining full coverage
  • Selectively pass through only the high-value data for AI consumption

How Greer Works for Logs and Traces

Logs

• Greer analyzes log data in real-time to identify common patterns (e.g. GET, POST requests)
• It passes through initial samples of each pattern, then summarizes and compresses the subsequent log messages
• This allows preserving the key information while reducing overall log volume by over 90%

Traces

• Greer goes beyond just analyzing trace endpoints, and maps out the full structure of each trace
• It identifies different execution paths (e.g. cached vs. non-cached) and tracks the performance of each path separately
• This allows Greer to provide full sampling coverage of the application while dropping unnecessary "noise" data

The Observability Data Lake

• Greer stores all the raw observability data in a low-cost data lake, ensuring no data is ever lost
• This data can be queried directly or selectively backfilled into observability tools as needed
• It enables use cases like anomaly detection, customer troubleshooting, and historical analysis

Business Impact and Results

• Greer can reduce observability data volume by over 90% with minimal impact to developer workflows
• It enables AI-powered observability and operations by providing clean, high-signal data for consumption
• Customers report a significant reduction in observability costs and improved mean time to resolution (MTTR)
• Greer can be set up in about 30 minutes, changing the conversation from complex manual tuning to an automated, low-risk solution

Example Use Cases

• Automatically identifying and preserving the logs and traces that power critical dashboards and alerts
• Triggering selective backfill of relevant observability data when a customer opens a support ticket
• Integrating with anomaly detection systems to provide the necessary observability data for investigation