AWS re:Invent 2025 - Customize AI models & accelerate time to production with Amazon SageMaker AI

Customizing AI Models and Accelerating Time to Production with Amazon SageMaker AI

Key Takeaways

• Customizing AI models is becoming increasingly important as the pace of growth for custom model-based AI applications is twice that of off-the-shelf foundation model APIs.
• Amazon SageMaker AI introduces new capabilities to simplify the model customization process, including:
  • An AI-powered model customization agent to define use cases and success criteria
  • Automated data generation and curation with responsible AI controls
  • Serverless reinforcement learning and other customization techniques
  • Seamless integration with MLflow for experiment tracking
  • Streamlined model deployment to SageMaker or Bedrock

The Importance of Custom AI Models

• Using off-the-shelf foundation models may not provide the necessary performance and accuracy for specific business use cases.
• As AI applications move from proof-of-concept to production, scaling with just foundation model APIs becomes less efficient.
• Highly regulated industries have strict compliance and security requirements that make relying solely on foundation model APIs infeasible.
• Building custom models can provide sustainable differentiation and increased value for both internal and external customers.

The Model Customization Workflow

1. Define Goals and Evaluation Criteria: The SageMaker AI model customization agent helps translate business use cases into model customization specifications.
2. Data Gathering and Enhancement: The agent can generate synthetic data while applying responsible AI controls, or integrate with existing data sources.
3. Model Customization: SageMaker AI offers serverless reinforcement learning and other techniques, abstracting away infrastructure complexity.
   • Reinforcement learning with verifiable rewards (RLVR) is used to improve the precision of tool calling by generating multiple responses and scoring them based on a custom reward function.
4. Model Evaluation: Custom metrics and prompts can be used to evaluate model performance on specific use cases, with visual scorecards and integration with MLflow for experiment tracking.
5. Model Deployment: Customized models can be deployed to SageMaker for real-time inference or Bedrock for API-based access.

Example: Customizing a Tool Calling Model

• The use case is for a tool calling model, which enables AI agents to interact with external systems through API calls, database queries, and function executions.
• The base foundation model often struggles with tool calling precision, hallucinating non-existent tools or failing to call available tools correctly.
• Using the SageMaker AI customization workflow, the presenter was able to:
  • Fine-tune the Quant 2.5 7B Instruct model using RLVR and a custom reward function
  • Achieve a 20-point increase in tool call reward score and a 20-point increase in F1 score compared to the base model
  • Deploy the customized model to a SageMaker real-time inference endpoint for testing
• The entire process, from defining the use case to deployment, took less than an hour, compared to a typical multi-month timeline for model customization.

Business Impact

• The ability to quickly customize AI models for specific business use cases can provide a significant competitive advantage.
• Streamlining the model customization workflow reduces the time and effort required, enabling faster iteration and deployment of production-ready AI applications.
• Responsible AI controls in the data generation process help ensure ethical and unbiased model behavior.
• Seamless integration with MLflow and deployment options for SageMaker and Bedrock make the customized models readily available for real-world use.

Key Takeaways

• Custom AI models are becoming increasingly important as the pace of growth for custom model-based applications outpaces that of off-the-shelf foundation models.
• SageMaker AI introduces new capabilities to simplify the model customization process, including an AI-powered agent, automated data generation, serverless training, and streamlined deployment.
• The example of a tool calling model customization demonstrates how the SageMaker AI workflow can be used to quickly iterate and deploy a production-ready AI application in under an hour.
• The business impact of these capabilities includes faster time to market, competitive advantage, and the ability to deploy ethical and unbiased AI models.