AWS re:Invent 2025 - Observability for Reliable Agentic AI with Strands SDK & OpenTelemetry (NTA406)

Observability for Reliable Agentic AI with Strands SDK & OpenTelemetry

Challenges with Deploying AI Agents in Production

• Agents can become a "black box" with limited visibility into their behavior and decision-making
• Issues with model evolution, contradictory prompts, and integration with external tools
• Difficulty identifying why agents are making specific decisions, especially in multi-agent architectures

Leveraging AWS Agent Core for Reliable Agent Deployment

• Agent Core provides a modular, component-based approach to building and hosting AI agents
• Reduces time-to-market by handling infrastructure, hosting, and integration with native/third-party services
• Enables the use of different agent frameworks (e.g., Strands) and observability tools (e.g., OpenTelemetry)

Implementing a Multi-Agent Financial Advisor

• Developed a financial advisor agent using the Strands SDK and running on AWS Agent Core
• Comprised of several specialized agents (rate checker, loan calculator, eligibility checker) orchestrated by a central agent
• Agents integrated with external knowledge bases to provide personalized financial recommendations

Observability Challenges and Improvements

1. Latency Issues:

   • Initial agent response times were slow (13 seconds)
   • Identified high "temperature" parameter causing the model to be overly creative
   • Reduced temperature and max tokens to make the model more constrained
   • Upgraded to newer language models (Haiku 4.5) to improve performance

2. Multilingual Support:

   • Encountered issues with Spanish language prompts due to parameter misconfiguration
   • Leveraged AWS Bedrock Prompt Management to store and version language-specific prompts
   • Deployed multiple agent endpoints (e.g., Spanish, English) to handle different user preferences

3. Knowledge Base Integration:

   • Discovered that the knowledge base data was outdated, leading to inaccurate recommendations
   • Investigated the observability traces to identify the knowledge base retrieval process
   • Configured the observability sampling rate to 100% to ensure all relevant data was captured

Optimizing Multi-Agent Architectures

• The initial architecture used a "swarm" of interconnected agents, leading to a complex full-mesh communication pattern
• Explored simplifying the architecture to a "workflow orchestrator" or "principal agent with subordinates" model
  • This can reduce the number of agent-to-agent hops and improve overall response times

Key Observability Metrics and Features

• Latency, error rates, token usage, and other runtime-specific metrics available in AWS Agent Core Observability
• Ability to trace individual user sessions and understand the flow of agent interactions
• Option to send logs, metrics, and traces to external observability tools using OpenTelemetry

Business Impact and Real-World Applications

• Improved observability and visibility into AI agent behavior can help organizations deploy reliable, production-ready agents
• Ability to quickly identify and resolve issues like latency, prompt conflicts, and outdated knowledge bases
• Optimized multi-agent architectures can enhance performance and cost-efficiency for complex AI-powered applications
• Demonstrated use case in the financial services industry, but applicable to any domain leveraging agentic AI

Resources and Next Steps

• AWS Agent Core Toolkit for simplified agent deployment and management
• Code samples for integrating other agent frameworks and observability tools with Agent Core
• AWS re:Invent workshop on "Deep Dive on AWS Agent Core"