Building a Well-Architected Foundation for Scaling Generative AI and Agentic Applications

Overview

This presentation from AWS re:Invent 2025 discusses the challenges and complexities of taking generative AI and agentic applications into production at scale. The speakers outline a comprehensive foundation for building these applications, covering key components like gateways, orchestration, observability, and security.

Challenges of Generative AI and Agentic Applications

Complexity Beyond Simple Chatbots

• Generative AI and agentic applications involve complex orchestration, agents, and safeguards - not just simple language models.
• Scaling these applications to production introduces new challenges around performance, scalability, security, and context management.

Key Challenges

• Performance and Latency: Agents need to reason, plan, and act with low latency over extended sessions.
• Scalability: Deploying thousands of agents requires purpose-built orchestration and infrastructure.
• Security and Isolation: Agents need secure access to sensitive data and tools, with proper controls in place.
• Context Management: Agents require evolving context and secure, scalable memory storage.
• Alignment and Oversight: Continuous monitoring, evaluation, and human oversight are needed to ensure agent alignment.

Comprehensive Foundation

Core Components

• Model and Agent Hub: Centralized access to models, agents, and tools via gateways.
• Data Pipelines: Ingestion, indexing, and vector storage for relevant data.
• Orchestration: Workflows, templates, and agent coordination.
• Clients: Applications and interfaces accessing the foundation.
• Operational Excellence, Observability, and Security: Critical cross-cutting concerns.

Key Services

• Amazon Bedrock: Fully managed service for building generative AI applications.
• Amazon Bedrock Agent Core: Fully managed platform for building and scaling agents.

Gateways and Registries

LLM Gateway

• Provides secure, unified access to a catalog of language models, handling cost attribution and guardrails.

Tool and Agent Gateways

• Secure and standardize access to external tools and agents, with registries for discovery and management.
• Can be combined into a comprehensive gateway and registry solution.

Observability

Importance of Observability

• Monitoring quality, cost, and risk - not just infrastructure metrics.
• Evaluating model outputs, tool selection, and agent convergence.
• Detecting issues like bias, toxicity, and PII leakage.

Observability Approach

• Collecting operational and semantic signals (logs, traces, metrics, user feedback).
• Aggregating and visualizing data in an observability platform.
• Implementing offline and online evaluation to optimize the system.

Agent Operations (Agent Ops)

Lifecycle Management

• Prototyping: Experimentation with models and frameworks.
• Development: Sandbox environments, observability integration, and gateway/registry integration.
• Testing: End-to-end QA and continued evaluation.
• Production: Live monitoring, security, governance, and audit.

DevOps Pipeline Example

• Git-based workflows for building, testing, and deploying agents.
• Leveraging agent registries and observability tooling (e.g., Langfuse) across environments.

Operating at Enterprise Scale

Centralized Platform Approach

• Federated model with a core platform providing shared services (e.g., model catalog, gateways).
• Consuming applications manage their own data and customizations.
• Emphasis on governance, risk management, and flexibility.

Resources

• Blog: Comprehensive Foundation for Generative AI and Agentic Applications
• Blog: Deploying an LLM Gateway on AWS
• Open Source: Agentic AI Foundation Accelerator
• Open Source: MCP Registry and Agent Registry
• AWS Skill Builder: Developing Agentic AI Skills