AWS re:Invent 2025 - Building for Efficiency & Reliability with Performance Testing on AWS (CMP351)

Building for Efficiency & Reliability with Performance Testing on AWS

Understanding Key Concepts: Resiliency, Reliability, and Efficiency

• Resiliency: The ability to recover from a crash or failure, whether it's seamless or takes some time.
• Reliability: Everything you can do to prevent a failure in the first place, reducing the need for resiliency.
• Efficiency: Building systems that meet service level objectives (SLOs) while consuming the least amount of resources.

The Role of Performance Testing

• Performance testing is the ability to understand how your system or application will behave under different load conditions.
• It goes beyond just load testing, encompassing:
  • Stress Testing: Identifying performance degradation and breaking points, testing at 2x, 5x, or 10x expected load.
  • Endurance/Soak Testing: Ensuring the system can sustain a particular load for an extended period, like during a major event.
  • Scalability Testing: Verifying the system can scale from zero to the expected peak load.
  • Spike Testing: Validating the system can handle sudden spikes in traffic, both on the way up and the way down.
  • Volume Testing: Understanding how the system behaves under massive amounts of data ingestion or processing.

Benefits of Performance Testing

1. Scaling and Deployment Confidence: Performance testing helps identify the right compute and database offerings for your workload.
2. Reliability and Efficiency: Investing in performance testing reduces the need for reactive resiliency measures.
3. Better User Experiences: Ensuring your application responds within target thresholds (e.g., 3 seconds) to retain customers.
4. Cost Optimization: Finding the right-sized infrastructure to meet performance goals without over-provisioning.

When to Perform Performance Testing

• Performance testing should be done:
  • Before major events or launches to validate readiness
  • During development, as part of unit testing
  • When making infrastructure changes, like switching to Graviton
  • Regularly as part of an ongoing monitoring and optimization process

Key Metrics to Monitor

• Percentiles: Measure the experience of your users or internal systems, focusing on the 95th and 99th percentiles.
• Transactions per Minute/Second: Track the overall throughput of your system.
• Latency: Measure response times, especially from different geographic regions for globally distributed systems.
• Errors: Identify and investigate any errors occurring during the tests.
• Resource Utilization: Monitor CPU, memory, and other resource consumption to aid in root cause analysis.

Performance Testing Frameworks on AWS

• Popular open-source frameworks include JMeter, K6, and Locust.
• AWS provides a managed solution called Distributed Load Testing on AWS:
  • Fully supported, open-source solution with a CloudFormation template for quick deployment.
  • Allows using JMeter, K6, or Locust scripts within a single platform.
  • Automatically provisions the necessary infrastructure across multiple AWS regions to simulate global traffic.
  • Cleans up the resources after the test completes, with no ongoing infrastructure management required.

Real-World Example

• The presenter demonstrated a global performance test for a major sports vendor, ramping up from 0 to 1 million users in 1 minute, then sustaining that load for an additional minute.
• The test showed real-time metrics, including average response time, successful requests, and errors triggered by the autoscaling system.
• The presenter encouraged the audience to try the Distributed Load Testing on AWS solution, as it can be deployed in 5 minutes and provides a deeper demo available at the AWS Village.