Scaling generative AI models for millions of users in Roblox (GAM310)

Here is a detailed summary of the key takeaways from the video transcription, broken down into sections for better readability:

Introduction to Roblox

• Roblox is not a game, but a platform for developers to build their own games and experiences.
• Roblox has a large ecosystem with around 2.5 million developers, 5.3 million experiences, and 79 million daily active users.

Generative AI at Roblox

• Roblox uses generative AI in various areas, including safety, creator tools, and player experiences.
• Roblox has built an internal platform to develop and deploy AI models, and now aims to externalize these tools to their 2.5 million developers.

Scaling Challenges

1. Load Multipliers: When scaling from internal teams to a large developer ecosystem, the load increases exponentially due to the way developers use the tools, leading to new challenges in capacity planning and load projections.

2. Managing Different AI Inference Use Cases: Roblox uses a mix of hosted AI providers, open-source models, and custom-trained models. They built a gateway architecture to abstract and manage this complexity for their developers.

3. Auto-scaling with Capacity Constraints: Roblox faced challenges with GPU capacity constraints, leading them to build a "load spill-over" system that dynamically routes traffic across pre-provisioned clusters.

4. Densifying Clusters: Roblox found that maximizing throughput (TRPs) was more important than optimizing for latency or time-to-first-token. They used techniques like online and offline batching to improve cluster utilization.

5. Evaluating Models for Production: Roblox found that standard academic benchmarks did not always reflect their real-world use cases. They are building their own evaluation datasets and frameworks to better assess model quality and performance.

Lessons Learned

1. Open-source model quality is improving: Roblox found that open-source models like Llama can often be cost-effective alternatives to hosted solutions, with quality close enough for many production use cases.

2. Versioning and complexity management: Roblox used AWS Deep Learning Containers to simplify the management of different model versions, frameworks, and hardware configurations.

3. Cluster densification through batching: Batching inference requests, both online and offline, was key to improving cluster utilization and reducing costs.

4. Building custom evaluation frameworks: Roblox found that standard benchmarks did not always reflect their real-world needs, so they developed their own evaluation datasets and frameworks.