Ensuring ethical use of AI models through consistent observability (AIM268)

Here is a detailed summary of the video transcript in markdown format, broken into sections for better readability. The key takeaways are preserved without losing important details.

Introduction and Overview

  • The session is about building applications with AI, specifically in the media industry.
  • The presenters are Jason Ye, a Staff Technical Advocate at Datadog, and Joe Cronin, the CTO of Arc XP, a division of The Washington Post.
  • Arc XP was born out of The Washington Post's digital transformation in 2011, and it now runs thousands of other media companies on its SaaS platform.
  • The media industry is interested in using generative AI but is also concerned about the implications for journalists, storytelling, and their business.

Arc XP's Approach to Generative AI

  • Arc XP believes in the power of storytelling to change the world and sees generative AI as a significant part of that.
  • They have taken a three-step approach:
    1. Improving workflows and productivity for newsrooms
    2. Providing personalized content experiences for consumers
    3. Developing new products that offer a unique experience for every reader/viewer
  • They started with an API-first approach, focusing on the fundamental building blocks and observability rather than jumping straight to user experience design.

Observability and Ethical Considerations

  • Observability was crucial for Arc XP to ensure the quality, workflow efficiency, cost, autonomy, and human oversight of their AI-powered features.
  • They needed to reliably identify hallucinations, protect against abuse, and ensure the model doesn't leak sensitive data.
  • Datadog's LLM observability tools were developed to provide end-to-end tracing, input/output evaluation, and trend analysis, going beyond traditional application performance monitoring.
  • Arc XP's key concerns around observability were quality, workflow efficiency, cost, AI autonomy, and human oversight to uphold their ethical standards.

Lessons Learned

  • Ensure your tech stack is ready before experimenting with generative AI.
  • Define a clear mission statement for using AI effectively in your organization.
  • Stay agile and humble, as mistakes and challenges are inevitable in the early stages.
  • Expect conflicts of interest, regulatory concerns, and risk management issues that can slow down your efforts.
  • Recognize that AI is often good at doing what you don't want it to do, and it can be challenging to get it to do what you need.

Conclusion and Resources

  • Datadog has a dedicated booth (1728) at the conference to showcase their LLM observability tools.
  • Arc XP offers generative AI-powered tools for media companies at ARXxp.com.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us