AWS re:Invent 2025 - GenAI game coach: Real-time gameplay feedback (DEV201)

AWS re:Invent 2025 - GenAI Game Coach: Real-time Gameplay Feedback

Introduction

• The speaker, Gagan Jot (Gagen), is a staff software engineer at Sony Interactive Entertainment, focused on backend engineering and AI enablement.
• The presentation explores using generative AI to provide real-time gameplay feedback and coaching for video game players.
• The goal is to create an "intelligent agent" that can analyze gameplay videos and provide personalized guidance to help players improve their skills.

Challenges in the Gaming Industry

• Gameplay in many games is very fast-paced and chaotic, with attacks resolving in 100-150 milliseconds, faster than the human reaction time of 250 milliseconds.
• Players often misdiagnose their own mistakes, as the gameplay happens too quickly for them to identify the root causes.
• Studios and players lack affordable tools and resources to analyze gameplay and provide meaningful feedback.

Opportunity for Generative AI

• Millions of players want to improve their skills, but lack access to personal coaches and advanced analytics tools.
• Generative AI models can be leveraged to analyze gameplay videos and provide real-time feedback and coaching.

Architectural Solution

1. User Interface:

   • A React-based single-page application hosted on Amazon CloudFront for low-latency global access.
   • Allows users to upload gameplay videos and view the analysis report.
   • Includes a conversational agent interface to interact with the AI coach.

2. API Layer:

   • API Gateway and Lambda URL functions to handle video uploads, analysis triggers, and chat interactions.
   • Leverages S3 for video storage.

3. Compute Layer:

   • Analysis Lambda: Generates pre-signed S3 URLs for video uploads and processes the videos.
   • Chat Handler Lambda: Invokes the Bedrock Agent to power the conversational coaching experience.
   • Agent Action Lambda: Executes the Bedrock Agent's action group functions.

4. AI/ML Layer:

   • Amazon Bedrock: Used for both direct video analysis and building the interactive coaching agent.
   • Amazon Recognition: Performs computer vision analysis on the gameplay videos.
   • Amazon Transcribe: Provides speech-to-text transcription of any spoken dialogue in the videos.
   • Amazon Comprehend: Conducts sentiment analysis on the gameplay videos.

Prompts and Personas

• The AI coach's personality and feedback style can be customized using prompts.
  • Example prompt: "You are an expert gaming coach with years of experience helping players improve their skills. You have access to the player's gameplay and are here to help them level up their game in a friendly and motivating way."

Demo and Results

• The speaker demonstrated the system using a 5-minute gameplay video from the game Destiny 2.
• The analysis provided insights on the player's performance, including:
  • Computer vision analysis of the scene, objects, and characters
  • Gameplay analysis identifying moments for improvement in situational awareness and enemy engagement
  • Overall assessment of the player's performance, highlighting strengths in aiming and cover usage
  • Sentiment analysis indicating a friendly and motivating tone in the feedback

Cost and Latency

• The total cost for analyzing the 5-minute video was $0.90.
• The latency was approximately 10 minutes, with 8 minutes for the video upload and 2 minutes for the AI analysis.

Next Steps and Future Opportunities

• Improve latency by exploring video compression and caching technologies.
• Integrate with live streaming platforms to provide real-time coaching during gameplay.

Key Takeaways

1. Generative AI can be leveraged to provide personalized, real-time feedback and coaching for video game players, addressing a key challenge in the gaming industry.
2. The proposed architecture utilizes a range of AWS services, including Bedrock, Recognition, Transcribe, and Comprehend, to power the gameplay analysis and coaching capabilities.
3. Customizable prompts and personas allow the AI coach to tailor its feedback style and tone to the player's preferences.
4. The demo showcased the system's ability to provide detailed, actionable insights on a player's performance, highlighting areas for improvement and strengths.
5. While the current implementation has some latency challenges, the speaker outlined plans to optimize the system for near-real-time performance and integration with live streaming platforms.

Conclusion

The presentation demonstrated a compelling use case for applying generative AI to the gaming industry, addressing the need for affordable and accessible tools to help players improve their skills. The proposed architecture leverages a range of AWS services to create an intelligent agent that can analyze gameplay videos and provide personalized, real-time coaching feedback. This solution has the potential to transform the way players approach skill development in video games, empowering them to level up their gameplay with the guidance of an AI-powered coach.