AWS re:Invent 2025 - Building AI That Customers Actually Trust (AIM2203)

Building AI That Customers Actually Trust

Qualrix's Approach to Improving AI Agent Experiences

• Qualrix is an experience management organization focused on the human connection between brands, employees, and customers.
• They have developed a methodology to help organizations integrate their brand, products/services, employees, and customer experience into a cohesive experience.
• This same approach has been applied to improving the performance and customer experience of AI agents deployed by organizations.

Challenges with AI Agents in Customer Service

• Organizations are rushing to automate with AI tools, but this is leading to a loss of human connection.
• Customers are often sent down "dead ends" with AI agents, and employees are overwhelmed by the new technologies.
• Customers want to feel seen, heard, and understood, not just receive faster responses.
• Recent research shows customer comfort with AI has rebounded, but AI in customer service has the highest failure rate of any AI application.
• AI agents often underperform on key metrics like ease of use, time savings, and overall benefits to the customer.

Qualrix's Solution: Agent Efficacy

• Qualrix provides an "intelligence layer" to measure and improve the customer experience (CX) delivered by AI agents.
• They diagnose four key problems organizations face with their AI agents:
  1. CX Blind Spots: Inability to easily assess if AI agents are improving CX.
  2. Inconsistent Benchmarking: No way to compare human vs. AI agent performance.
  3. Ineffective Optimization: Lack of root cause analysis when AI agents fail.
  4. Silent Reporting: Disparate data sources make it hard to get a holistic view.
• Qualrix's methodology measures AI agents against human agents and other channels to:
  • Improve the business impact of AI agents
  • Ensure positive customer experiences
  • Avoid AI agent risks

Measuring AI Agent Effectiveness

• Qualrix measures AI agents across four key areas:
  1. CX and Outcome Metrics: Customer satisfaction, sentiment, effort, resolution rate.
  2. Risk, Security, and Quality Assurance: Toxicity, biases, data leakage.
  3. Operational Efficiency: Containment rate, average handling time.
  4. Agent Quality and Performance: Barriers, agent confusion, customer engagement.
• They use a proprietary survey methodology to gather detailed feedback, including open-ended responses analyzed by conversational AI.
• This provides a comprehensive view of AI agent performance across CX, operational, and agent-specific metrics.

The Future of Experience Agents

• Qualrix sees the future of experience management moving from measurement to an "activation engine".
• This involves proactively spotting risks, forecasting churn, and testing strategies before implementation.
• Experience agents can act as a system of orchestration, routing information to the right owners and resolving issues automatically.
• Qualrix's experience agents aim to increase resolution time, improve customer loyalty, and decrease attrition.
• They position the experience agent as the "experience layer" that connects operational data to the human connection and customer experience.