AWS re:Invent 2025 - Why Your Processor Matters for AI Inference and General Compute (MAM210)

Leveraging AMD EPYC CPUs for AI Inference and General Compute

Overview of AMD's End-to-End Solutions

• AMD is positioning itself as an end-to-end solutions provider, offering CPUs, GPUs, networking, and software to address a broad range of workloads.
• The CPU plays a crucial role in the AI pipeline, handling data input, cleaning, pre-processing, model training, and deployment, in conjunction with GPUs.

The Rise of AI Inference

• The cost of AI inference has dropped dramatically, from $20 per million tokens in 2022 to just 7 cents in 2024, a 280x reduction.
• This rapid decrease in inference costs has led to an insatiable demand for compute resources, driving massive data center buildouts to support the growth of AI.
• Inference workloads have become more complex, requiring significant pre-processing, post-processing, and iterative loops between CPUs and GPUs.

AMD EPYC CPU Performance and Efficiency

• AMD EPYC CPUs have consistently delivered high performance and power efficiency across a broad range of workloads.
• AMD has grown its market share from low single digits in 2017 to 41% with the latest Zen 5 (Turin) processors.
• The latest EPYC 9575F CPU is designed to optimize GPU performance by minimizing CPU bottlenecks, delivering up to 20% more GPU performance.

EPYC CPUs for AI Inference

• EPYC CPUs excel at a variety of AI inference workloads, including machine learning models, recommendation systems, and mixed workloads with both general compute and AI components.
• For smaller-scale AI deployments, such as chatbots, the large memory capacity of EPYC CPUs can be particularly beneficial.
• AMD provides optimized software plugins (ZenDNN) that can further improve performance on standard AI frameworks like PyTorch, TensorFlow, and ONNX.

Cost Efficiency and Utilization Benefits

• Choosing the right CPU platform can have a significant impact on cost efficiency, with AMD EPYC instances delivering up to 50% lower costs compared to alternatives.
• The high performance of EPYC CPUs allows for smaller instance sizes, reducing software licensing costs that are often tied to the number of cores.
• EPYC CPUs are highly available, geographically distributed, and can be easily repurposed for other workloads, making them well-suited for batch processing of AI inference during off-peak hours.

Real-World Examples and Results

• CVS Health, a Fortune 5 healthcare company, has leveraged AMD EPYC-powered instances to drive significant cost savings and operational efficiency through their FinOps practices.
• By standardizing on optimal instance types, consolidating reserved instances, and partnering with application teams, CVS has been able to achieve substantial cost reductions and better align cloud spend with business value.
• CVS is also exploring the use of AI and automation to further enhance their FinOps capabilities, including generating insights and optimizing container-based workloads.

Key Takeaways

• AMD EPYC CPUs provide a high-performance, power-efficient, and cost-effective platform for a wide range of workloads, including AI inference.
• The accessibility, utilization opportunities, scalability, and operational simplicity of EPYC CPUs make them a compelling choice for AI inference and general compute.
• Careful selection of the right CPU platform can lead to significant cost savings and performance improvements, as demonstrated by real-world examples like CVS Health.
• AMD's software optimizations and partnerships with leading AI frameworks further enhance the value proposition of EPYC CPUs for AI workloads.