AWS re:Invent 2025 - Building Zero-CVE Container Images at Scale: Patterns and Pitfalls (MAM215)

Building Zero-CVE Container Images at Scale: Patterns and Pitfalls

Overview of Chain Guard

• Chain Guard's mission is to be the secure source for open-source software (OSS)
• They provide over 1,818 container images with 134,000+ versions, all built to be secure and compliant
• Key tools used include:
  • Malange - Package manager for building APKO packages
  • Octo STS - GitHub security tool for managing security tokens
  • SigStore - For signing container images
  • Malcontent - Scans container images for malware

Building from Source for Trust and Security

• Building from source code provides deeper visibility and "depth in defense" compared to using pre-built images
• Allows using safer compiler options to reduce vulnerabilities in the software stack
• Enables trusting the provenance and integrity of the built artifacts

The Chain Guard Factory

• Automated pipeline that downloads OSS projects, generates package builds, remediates CVEs, and publishes secure container images
• Runs 24/7, rebuilding all 1,818 projects daily to address zero-day vulnerabilities and keep packages up-to-date
• Utilizes massive scale, spinning up over 1,000 CPU cores to handle the build workload

Automating the Build Process

• Heavily leverages GitHub pull requests to manage version updates and CVE remediations
• Automates package building, testing, and image publishing using Kubernetes and the Malange tool
• Employs AI to diagnose and fix build errors, improving the testing and quality of the output

Rigorous Testing and Validation

• Images are not released until they pass comprehensive functional testing on real Kubernetes clusters
• Uses tools like Helm, Docker, and custom test harnesses to validate the images beyond just "hello world"
• Ensures the built artifacts are truly production-ready and meet the required standards

Secure Distribution and Customization

• All images are signed with SigStore and pushed to customers' private repositories
• Customers can clone the images and further customize them with their own packages and configurations
• Chain Guard also provides a set of 54 free, zero-CVE base and application images for common use cases

The Chain Guard Software Stack

• Chain Guard OS - A minimal, hardened operating system distribution maintained by Chain Guard
• Chain Guard Virtual Machines - Secure VM images built using the same principles as the container images
• Chain Guard Containers - The core container image offering, with FIPS-compliant versions available
• Chain Guard Helm Charts - Open-source Helm charts for popular applications, maintained by Chain Guard
• Chain Guard Libraries - Secure, hardened versions of popular open-source libraries (Python, Java, JavaScript)

Key Takeaways

• Chain Guard has built a highly scalable, automated factory to produce secure, zero-CVE container images from open-source projects
• Their approach emphasizes building from source, using safer compiler options, and rigorous testing to ensure trust and security
• The Chain Guard software stack provides a comprehensive set of secure, hardened options for containers, VMs, operating systems, Helm charts, and libraries
• By offloading the burden of building and maintaining secure OSS artifacts, Chain Guard aims to help organizations focus on their core business while ensuring their software supply chain is secure