Scoring AI: A practical approach to validating generated content (TNC121)

Leveraging Generative AI: Ensuring Trust and Validation in Content Creation

Challenges of Rapid Innovation

  • AWS has seen tremendous growth, with over 4,200 services and features released since 2011.
  • The training and certification team at AWS has over 600 different digital and learning experiences and 1,000+ builder labs, making it challenging to keep up with the pace of change.
  • The vast amount of information available on the internet (45 billion publicly indexed web pages) makes it difficult to navigate and find relevant content.

The Role of Generative AI

  • Generative AI can be a powerful asset, but it's not a magical solution for everything.
  • Generative AI can produce non-deterministic results, meaning the same prompt can yield slightly different answers.
  • Using generative AI doesn't guarantee success, and it's essential to validate and trust the content produced.

Importance of Scoring and Validation

  • The quote "if you can't measure it, you can't manage it" highlights the importance of having a repeatable process to measure and validate the content.
  • Inaccurate information can have downstream effects, especially in mission-critical systems or heavily regulated industries.
  • Scoring introduces discipline and a repeatable process, helping to deal with the non-deterministic nature of generative AI.
  • Defining quality and aligning on expectations is crucial when using generative AI with a team.

AWS Training and Certification's Journey with Generative AI

  1. Defining Quality: The team established the "Four C's" criteria - correct, compliant, complete, and consistent - to assess content quality.
  2. Integrating Scoring: A scoring system was developed, using a scale of 1 to 5 to evaluate the usability of the content.
  3. Automating and Scaling: The "Geni Workbench" was created, allowing content creators to access prompts, adapt them, and get content scored.
  4. Feedback Loops: Mechanisms were implemented to capture user feedback and continuously improve the scoring service.

Key Lessons and Recommendations

  • Embrace experimentation and a distributed approach to prompt engineering.
  • Use scoring to drive meaningful discussions and align on quality expectations.
  • Beware of magical thinking around generative AI - progress takes time, so celebrate small wins.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us