AWS re:Invent 2025 - Load balancing evolved: ALB Target Optimizer (NET336)

AWS re:Invent 2025 - Load Balancing Evolved: ALB Target Optimizer (NET336)

Problem Statement

• Traditional load balancing algorithms like round-robin and least outstanding requests fall short for "low concurrency" applications.
• Low concurrency applications are compute-intensive, where each deployed instance can only process a small number of requests at a time (e.g., 1, 5, 10 requests).
• With traditional load balancing, requests can get routed to busy targets, leading to:
  • Decreased request success rate and increased error rate
  • Higher perceived latency as rejected requests need to be retried
  • Underutilization of the target group, with some targets sitting idle

The Solution: Target Optimizer

• The load balancer should know which targets are idle and how many requests each target can still process.
• This information should be distributed across the load balancer nodes in a way that avoids overloading any single target.

How Target Optimizer Works

• An agent is installed on each target that acts as a proxy between the application and the load balancer.
• The agent tracks the number of concurrent requests being processed and sends signals to the load balancer nodes when capacity becomes available.
• Load balancer nodes use this information to route requests only to targets that have the capacity to process them.
• The max concurrent requests per target can be configured on the agent based on the target's resources (CPU, memory, etc.).

Benefits of Target Optimizer

1. High Request Success Rate: Requests are only routed to targets with available capacity, reducing rejections and retries.
2. High Target Group Utilization: Targets process requests as soon as capacity becomes available, minimizing idle time.
3. Automatic Load Shedding: If all targets are at capacity, the load balancer will automatically reject new requests to protect the targets.

Setup Process

1. Install the Target Optimizer agent on each target.
   • Configure the agent with the HTTP/HTTPS port, proxy port, and control port.
   • Specify the max concurrent requests per target.
2. Create a new "Optimized Target Group" and register the agent-enabled targets.
   • Specify the same control port as configured on the agents.
3. Add the new Optimized Target Group to an existing Application Load Balancer listener.
   • Traffic can then be shifted from the old target group to the new optimized one.

Supported Targets and Monitoring

• Target Optimizer supports EC2 instances, IP addresses, Amazon EKS, and Amazon ECS.
• New metrics are available to monitor the active control channels, errors, request volume, rejections, and control queue length.

Business Impact and Use Cases

• Target Optimizer enables efficient load balancing for compute-intensive, low concurrency applications like large language models.
• It helps maximize the utilization of the target group, potentially allowing for a smaller fleet size.
• The automatic load shedding protects targets from being overloaded, ensuring a more reliable application experience.
• Example use cases include media generation (video, text, image), scientific computing, and other AI/ML workloads.