AWS re:Invent 2025 - Live logs & prosper: Use AI to make logs a primary observability signal(AIM346)

Summary of AWS re:Invent 2025 - Live logs & prosper: Use AI to make logs a primary observability signal (AIM346)

The Challenge of Observability Today

• Observability has been built on data collection, proprietary agents, and dashboards, but customers struggle to get answers and context from the alerts.
• Significant time and effort is spent building data pipelines to get logs into observability solutions, often resulting in data loss.
• As systems have become more complex (e.g., microservices, Kubernetes), logs have become the most voluminous observability signal, but they are often underutilized due to being messy, unstructured, and coming from disparate sources.

The Power of Logs and the Need for a Scalable Platform

• Logs contain valuable context and information that can be used to solve problems, but managing the scale and complexity of logs has been a challenge.
• A scalable, cost-effective platform is needed to ingest, store, and analyze the massive volume of logs generated by modern systems.
• Elastic provides a highly scalable platform with features like efficient log storage, powerful search and analytics capabilities, and machine learning-driven pattern analysis.

Elastic Streams: AI-Driven Log Analytics

1. Automated System Identification: An AI-powered engine analyzes logs to automatically identify the systems they come from and partition the data accordingly (e.g., Hadoop logs, Spark logs, order processing logs).
2. Structured Data Extraction: The AI generates parsing patterns to extract structured data from unstructured log entries, enabling more effective analytics.
3. Proactive Problem Detection: The AI identifies common problems for each system (e.g., Spark out-of-memory errors) and sets up continuous monitoring to alert on these issues.
4. Contextual Insights: The AI stores information about the identified systems in a knowledge base, which can be used to provide contextual insights during root cause analysis.
5. Flexible Ingestion: Customers can continue using existing log collection methods (e.g., Elastic Agent, Logstash) or simply send logs directly to the new Elastic Logs index, with the AI-powered processing happening transparently.

Technical Details and Benefits

• Elastic Logs DB index mode provides up to 70% storage cost savings through compression and indexing techniques.
• A forthcoming compressed log processor can provide an additional 50% storage savings by templating repeated log lines.
• The AI-powered partitioning, structuring, and problem detection capabilities are built on Elastic's vector database and retrieval-augmented generation capabilities.
• These features enable customers to spend less time managing log pipelines and more time solving problems, with the AI surfacing insights and actionable information.

Business Impact and Use Cases

• Reduces the time and effort required to get value from log data, allowing teams to focus on solving problems rather than managing observability pipelines.
• Proactively identifies issues that customers may not have been aware of, preventing downtime and improving system reliability.
• Provides contextual insights that accelerate root cause analysis and problem resolution.
• Applicable across a wide range of industries and use cases, from e-commerce and financial services to cloud infrastructure and IoT.

Example Walkthrough

The presentation includes a live demo showcasing the Elastic Streams capabilities:

• Automatic partitioning of logs by system (e.g., Spark, Hadoop)
• AI-generated parsing patterns to structure unstructured log data
• Proactive detection of Spark out-of-memory issues and automated alert creation
• Simplified data quality monitoring and retention management

Key Takeaways

1. Logs are a valuable but underutilized observability signal due to the scale and complexity of modern systems.
2. Elastic provides a scalable platform and AI-powered capabilities to help customers extract insights from their log data more effectively.
3. Elastic Streams automates key log management tasks, including system identification, data structuring, and problem detection, allowing teams to focus on solving issues rather than managing observability pipelines.
4. The combination of efficient log storage, powerful analytics, and AI-driven insights can significantly improve observability and reduce downtime for Elastic customers.
5. These capabilities are applicable across a wide range of industries and use cases, making logs a primary observability signal for the future.