AWS re:Invent 2025 - Managing Bots vs Humans with CloudFront and AWS WAF (NET324)

Managing Bots vs Humans with CloudFront and AWS WAF

Overview

• Presenters: Itav Arditi (AWS Edge Specialist) and Nick McCord (AWS Startup Solutions Architect)
• Goal: Discuss how to manage bots vs. humans using Amazon CloudFront and AWS WAF
• Presentation level: Advanced (300-level)
• Format: Live demo, code examples, and architecture diagrams

The Challenge: Bots vs. Humans

• Legitimate users (humans) vs. good bots (crawlers, indexers) vs. bad bots (attackers, scrapers)
• Bad bots can cause issues like DDoS attacks, content theft, and saturating infrastructure
• Need to differentiate between human and bot traffic to protect the application

AWS Edge Services Overview

• Amazon CloudFront: CDN with 700+ Points of Presence (PoPs) globally
• AWS Lambda@Edge: Compute at the edge for custom logic
• AWS WAF: Web Application Firewall for layer 7 protection

Request Flow and AWS Services

1. Viewer Request: Evaluated against AWS WAF rules before hitting the CloudFront cache
2. Origin Request: Cache miss triggers a request to the origin server
3. Origin Response: Populates the CloudFront cache for subsequent requests
4. Access Logs: Captured using Amazon CloudWatch, Amazon Kinesis, and Amazon S3

AWS WAF Deep Dive

• Protection Pack (formerly Web ACL): Scaffolding for rules and rule groups
• Protected Resources: APIs, user groups, etc. that need WAF protection
• Rules: Encapsulate business logic and actions (e.g., geo-based IP rules, query string checks)
• Rule Groups: Allow portability of rules across different Protection Packs
• Importance of rule priority and "terminating" rules

Example: Pet Adoption Platform

• Platform for aggregating pet adoption data from shelters and rescues
• Faced issue of bots saturating the adoption pipeline with fake requests
• Key metrics: website traffic, application review time, time to adopt

Phase 1: Enabling AWS WAF

• Implemented rate limiting, JWT fingerprinting, and bot control managed rule group
• Resulted in increased bot traffic trying to bypass the WAF protections

Phase 2: Smarter WAF Integration

• Switched from "block" to "count" action, adding custom request headers for bots
• Implemented caching policies to serve limited content to identified bots

Phase 3: Advanced Edge Techniques

1. Origin Modification: Used CloudFront Functions to route bot traffic to a different origin
2. Cache Busting: Selectively bypassed caching for a percentage of bot traffic to gather more insights

CloudFront Functions Deep Dive

• Lightweight compute at the edge, running JavaScript code
• Can read/modify request and response objects
• Supports async operations, native JavaScript methods, and custom logic
• Allows terminating the request or proxying it to the origin

Observability and Metrics

• Tracked metrics like cache hit rate, CloudFront requests, Lambda invocations, and adoption requests
• Correlated bot traffic and origin modifications in the observability data

Key Takeaways

1. Be Traffic-Aware: Understand the different types of traffic (humans, good bots, bad bots) and their characteristics
2. Simplify at the Edge: Move complexity to the edge using CloudFront, WAF, and Lambda@Edge instead of the application
3. Leverage AWS Service Synergy: Combine multiple AWS services (CloudFront, WAF, Lambda) to create a comprehensive solution

Additional Resources

• Articles on CloudFront request flow and origin routing techniques
• Information on the new AWS CloudFront flat-rate pricing plans

Call to Action

• Provide feedback on the session through the AWS Events app
• For each feedback, the presenters will donate to a pet-related nonprofit organization