Generative AI meets multi-tenancy: Inside a working solution (SAS407)

The video transcript covers a detailed discussion on building various multi-tenant architectures for SAS (Software as a Service) solutions, with a focus on resolving common SAS architecture challenges. The key takeaways from the transcript are:

SAS Architecture Challenges

1. Scaling Language Models: Serving a single customer with a large language model (LLM) can be complex, and scaling it to thousands of tenants introduces even more challenges, such as managing multiple LLMs, customer data isolation, and cost optimization.

Two Popular Gen Architectures

1. Retrieval Augmented Generation (RAG): Leveraging a generic LLM and embedding customer data into a vector store for retrieval and augmentation of the LLM response.
2. Fine-tuning: Fine-tuning the LLM with customer data to have the knowledge already embedded, reducing the need for context in each request.

AWS Services for Multi-Tenant SAS

1. Amazon Bedrock Knowledge Base: A managed RAG service that abstracts the complexity of connecting data, vector store, and LLM.
2. Amazon Bedrock Customized Model: A feature to fine-tune LLMs for each tenant without the need to host and manage the models.

Basic Tier vs. Premium Tier Architectures

1. Basic Tier: Focuses on using shared services (pool pattern) and a RAG approach to optimize costs.
2. Premium Tier: Utilizes dedicated resources (silo pattern) and a combination of RAG and fine-tuning to provide the best user experience.

Key Architecture Challenges and Solutions

1. Tenant Isolation: Leveraging IAM roles, security token service, and data access policies to ensure each tenant can only access its own resources.
2. Cost per Tenant: Capturing metrics like input/output tokens with tenant context, aggregating them, and multiplying with the total service cost to derive the cost per tenant.
3. Noise Eater: Implementing tenant-specific token usage plans and real-time token usage tracking to provide a throttling experience at the API level.

The transcript also includes detailed code examples and explanations of how to implement these concepts, as well as links to GitHub repositories and a workshop that covers these topics in depth.