AWS re:Invent 2025 - Intelligent Observability & Modernization w/ Amazon OpenSearch Service (ANT315)

Intelligent Observability & Modernization with Amazon OpenSearch Service

Overview

• Presentation covers how AWS customers can leverage Amazon OpenSearch Service for observability and modernization of their cloud-native applications.
• Presented by Suhibqatari Walla, Senior OpenSearch Specialist SA, and Joshua Brightite, Product Manager for OpenSearch Service.
• Focuses on addressing common observability challenges faced by a fictional company "Any Company" - a fast-growing e-commerce platform.

Observability Challenges for Any Company

• Microservices Complexity: Visibility challenges across numerous microservices, making it difficult to identify root causes of issues like checkout failures.
• Data Pipeline Chaos: Constant challenges in calibrating and maintaining data pipelines from multiple sources and formats.
• Developer Productivity: DevOps teams spend significant time manually analyzing logs, traces, and metrics to understand application behavior.
• Cost Management: Exponential growth in observability data volume makes traditional monitoring solutions cost-prohibitive.

Amazon OpenSearch Service for Observability

• Provides a comprehensive observability platform to ingest, store, analyze, and visualize telemetry data from modern cloud-native applications.
• Integrates with various AWS monitoring services (CloudWatch, Managed Prometheus) and supports open standards like OpenTelemetry for data collection.
• Offers features for data ingestion, transformation, storage, and analysis through OpenSearch and OpenSearch Dashboards.

Intelligent Observability with OpenSearch Service

1. Data Ingestion and Storage:

   • Any Company uses OpenSearch Ingestion to collect and transform data from various sources (logs, metrics, traces).
   • Data is stored in OpenSearch clusters or OpenSearch Serverless for scalable, cost-effective long-term retention.

2. Observability Insights and Analysis:

   • OpenSearch Dashboards provides a purpose-built user experience for exploring and visualizing observability data.
   • Developers can use the Discover experience to query data using the powerful Pipe Processing Language (PPL), SQL, or natural language.
   • Robust monitoring and anomaly detection capabilities automatically identify and alert on unusual patterns in the data.

3. Centralized Observability with OpenSearch UI:

   • The next-generation OpenSearch UI consolidates observability insights from multiple OpenSearch deployments into a single, unified view.
   • Workspaces feature allows organizing dashboards, alerts, and queries based on specific use cases and access control.

New Observability Capabilities in OpenSearch Service

1. Simplified Onboarding and Data Ingestion:

   • Pre-built blueprints for ingesting popular AWS and third-party logs (e.g., ALB, CloudTrail, OpenTelemetry) with automatic configuration.
   • New "Get Started" workflow in the OpenSearch console to quickly set up an OpenSearch UI instance and ingest data.

2. Enhanced PPL and Analytical Capabilities:

   • Expanded Pipe Processing Language (PPL) with new commands for joins, lookups, time analysis, and unstructured data extraction.
   • Enables sophisticated queries to correlate data across services and identify root causes (e.g., connecting checkout failures to authentication and payment issues).

3. Integrated Visualization and Exploration:

   • Consolidated "Discover" experience for querying, visualizing, and building dashboards, all within a single workflow.
   • Ability to easily migrate existing queries and dashboards from other tools into OpenSearch UI.

Agentic AI for Observability

• Any Company's engineering teams want to leverage AI agents to automate observability tasks and speed up incident response.
• The OpenSearch Model Context Protocol (MCP) enables AI agents to easily connect to OpenSearch data sources and perform advanced analysis.
• Demo showcases an AI agent using MCP to:
  • Automatically investigate and identify root causes of checkout service failures during a traffic surge.
  • Quantify the business impact and provide a comprehensive remediation plan.
  • Recommend preventative measures to avoid similar issues in the future.

Key Takeaways

• Amazon OpenSearch Service provides a comprehensive, scalable observability platform for modern cloud-native applications.
• Recent enhancements focus on simplifying onboarding, enhancing analytical capabilities, and improving the user experience.
• Agentic AI, powered by the OpenSearch MCP, enables automated observability workflows to accelerate incident response and improve overall system reliability.
• OpenSearch Service and its advanced features help organizations like Any Company address common observability challenges and modernize their application monitoring.