AWS re:Invent 2025 - Your AI Agent Is Expensive and Wrong (Let's Fix Both) (AIM264)

Summary of "Your AI Agent Is Expensive and Wrong (Let's Fix Both)"

Introduction

• The presentation focused on the need for a different approach to building high-quality, cost-effective AI agents that can be deployed in mission-critical applications.
• The speakers were Craig Wy, Head of Product Management for AI and ML at Databricks, and representatives from Condé Nast, a global media company.

Challenges with Current AI Agents

• Current AI agents often lack the necessary governance, control, and evaluation to be deployed in mission-critical applications.
• Businesses need to be able to understand and quantify the capabilities and performance of their AI agents, which is often lacking in today's solutions.
• There is a trade-off between the ease of use and the quality/cost of AI agents, with most current solutions favoring ease of use over high-quality, cost-effective deployment.

Databricks' Approach: Unity Catalog and Agent Bricks

Unity Catalog

• Unity Catalog is the foundation for governing and monitoring all data, models, and capabilities used by AI agents on the Databricks platform.
• It provides comprehensive governance and logging of all data access, API calls, and model usage by AI agents.
• This allows businesses to have full visibility and control over what their AI agents are doing and accessing.

Agent Bricks

• Agent Bricks is Databricks' solution for building high-quality, cost-effective AI agents.
• It provides a set of pre-built agent templates (e.g., information extraction, knowledge assistant, multi-agent supervisor) that can be easily customized and tuned.
• The key focus is on making it easy to build agents that can be evaluated, monitored, and optimized for performance and cost.
• Agent Bricks includes features like:
  • Automatic benchmarking and evaluation of agent performance
  • Ability to fine-tune agents based on user feedback
  • Optimization of agent models for cost-effectiveness

Condé Nast's Use Case

• Condé Nast, a global media company, faced the challenge of democratizing data access and insights across their diverse business units and brands.
• They partnered with Databricks to build a centralized data lakehouse and leverage Databricks' text-to-SQL and agent capabilities.
• Condé Nast's key objectives were to:
  • Enable self-service data access and analysis for executives and business users
  • Unify data and analytics across their disparate business units and brands
  • Leverage AI and machine learning to drive insights and personalization

Results and Lessons Learned

• Condé Nast was able to empower their business users to build their own text-to-SQL agents, enabling faster access to data and insights.
• The multi-agent supervisor capability allowed them to connect these individual agents, providing a unified interface for users to access data and insights.
• Key lessons learned:
  • Preparing curated data sets and ensuring data quality is crucial for effective text-to-SQL and agent-based solutions
  • Providing instructions and guard rails for the multi-agent supervisor is important to ensure reliable and appropriate responses
  • Tracking user questions and feedback helps identify areas for improvement and new data/agent requirements

Conclusion

• Databricks and Condé Nast demonstrated a new approach to building high-quality, cost-effective AI agents that can be deployed in mission-critical applications.
• The combination of Unity Catalog for governance and Agent Bricks for agent development and optimization provides a powerful platform for businesses to leverage AI at scale.
• The Condé Nast use case showcased the real-world benefits of this approach, including increased self-service data access, faster insights, and improved data-driven decision making.