AWS re:Invent 2025 -What’s new in search, observability, and vector databases w/ OpenSearch (ANT201)

AWS re:Invent 2025 - What's New in Search, Observability, and Vector Databases with OpenSearch

Overview of OpenSearch

• OpenSearch is a popular open-source platform for a variety of use cases:
  • Search applications, including AI-powered search backends
  • Analytics and observability, providing fast search and aggregation of data
  • Security analytics and real-time data analysis
• OpenSearch is composed of several open-source components:
  • Data Prepper: Ingestion pipeline that feeds data into OpenSearch
  • OpenSearch: The core data engine and analysis engine
  • OpenSearch Dashboards: The front-end for data visualization and exploration

OpenSearch Community and Adoption

• OpenSearch transitioned to the Linux Foundation in the past year, with AWS as a premier member
• The project has seen rapid growth, with:
  • 1.3 billion downloads in 4 years
  • Over 3,000 active contributions from 400 organizations in the past year
  • Thriving user communities around the world
• Reasons for OpenSearch's popularity:
  • Innovation and new features added regularly
  • Improved performance, with 11x improvement in search and aggregations, and 2.5x improvement in vector search
  • AWS's delivery of OpenSearch as a fully managed service

Data Ingestion with OpenSearch Ingestion Service

• OpenSearch Ingestion Service is a serverless pipeline that simplifies data ingestion into OpenSearch
• Key capabilities:
  • Supports a wide range of data sources, including push and pull sources
  • Provides data processing and transformation capabilities
  • Integrates with AWS services like DynamoDB, RDS, and S3
  • Offers a visual workflow for building ingestion pipelines
  • Supports batch AI inference to create vector embeddings

Observability and Log Analytics

• OpenSearch Dashboards provides a unified view for observability, integrating with CloudWatch and Security Lake
• Powerful log analysis capabilities with the Pipe Programming Language (PPL):
  • Supports complex queries, joins, and analytical functions
  • Provides natural language query support for easier exploration
  • Enables rapid root cause analysis and debugging of issues

Search and Vector Databases

• Hybrid search, combining keyword and semantic search, is becoming increasingly popular
• OpenSearch provides automatic semantic enrichment, using a sparse neural model to enhance document indexing
• OpenSearch as a vector database:
  • Supports up to 1 trillion vectors, with various optimization techniques for cost-efficiency
    • Disk-optimized vector mode for reduced memory usage
    • S3 Vectors integration for storing vectors in S3
  • Collaborated with NVIDIA to leverage GPUs for faster index building (up to 20x) and lower costs (up to 12x)

Agentic Search and AI-powered Applications

• OpenSearch provides tools for building agentic search applications:
  • MCP server for integrating OpenSearch with AI models and agents
  • Agentic memory capabilities for storing short-term and long-term context
  • Pre-built agents like the Flow Agent, Conversational Agent, and Plan-Execute-Reflect Agent

Infrastructure Enhancements in Amazon OpenSearch Service

• Supports up to 1,000 nodes and 25 PB of data in a single cluster, with high availability and durability
• Launched OpenSearch Optimized instances with significant performance improvements
• Introduced Derived Source feature to reduce storage requirements by up to 40%
• Expanded the Amazon OpenSearch Serverless offering with increased scalability and features

Key Takeaways

• OpenSearch has seen rapid growth and adoption, driven by its open-source innovation and AWS's managed service offerings
• The platform provides a comprehensive set of capabilities for search, observability, vector databases, and agentic AI applications
• Significant performance and cost optimizations have been achieved through collaborations with partners like NVIDIA
• Amazon OpenSearch Service offers enterprise-grade features, scalability, and manageability, making it a compelling choice for organizations looking to leverage open-source search and analytics technologies