AWS re:Invent 2025 - Hybrid connectivity at scale: A deep dive into AWS Direct Connect (NET403)

Hybrid Connectivity at Scale: A Deep Dive into AWS Direct Connect

Physical Connectivity

• AWS Direct Connect provides physical connections between customer infrastructure and the AWS network.
• Key decisions when establishing Direct Connect:
  • Location - Choose from the list of available Direct Connect locations worldwide, considering proximity to your infrastructure.
  • Bandwidth - Options include 1 Gbps, 10 Gbps, 100 Gbps, and 400 Gbps ports.
  • MAX Enabled Port - Enables encryption between the customer device and the AWS Direct Connect device.
• The "meet-me room" is where the AWS Direct Connect port is presented for the customer to connect to.
• Customers work with colocation providers to establish the cross-connect between the AWS port and the customer's infrastructure.
• This process often involves multiple parties (AWS, customer, colocation provider, telco) with different processes, which can introduce challenges.
• Tips for managing the process:
  • Draw a simple diagram of the end-to-end connectivity.
  • Identify the owner of each component.
  • Document all port IDs, circuit numbers, and contact details.

Logical Connectivity

• Virtual Interfaces (VIFs) provide the logical layer 2 and layer 3 connectivity over the physical Direct Connect.
• Types of Virtual Interfaces:
  1. Public VIFs: Connect to public AWS services like S3.
  2. Private VIFs: Connect to VPCs, using either a regional or global (Direct Connect Gateway) approach.
  3. Transit VIFs: Connect to Transit Gateways and Cloud WAN.
  4. Hosted VIFs: Virtual interfaces owned by another account's Direct Connect.
• Site-to-Site Connectivity:
  • Site Link allows routing traffic between Direct Connect locations over the AWS backbone.
  • Can be enabled and disabled as needed, providing flexibility.

Monitoring and Troubleshooting

• CloudWatch provides metrics for Direct Connect, including connection state, light levels, and bandwidth usage.
• Troubleshooting steps:
  1. Check the physical layer (e.g., light levels, cross-connect).
  2. Verify the layer 2 configuration (e.g., VLAN tags).
  3. Investigate the BGP layer 3 issues (e.g., BGP parameters, network connectivity).
• Common troubleshooting techniques:
  • "Rolling the fiber" to swap transmit and receive.
  • Putting a "loop" on the connection to test the path.

Routing and High Availability

• Routing considerations:
  • Route limits - Important to monitor and summarize routes to stay within limits.
  • Routing preference - Use BGP attributes like AS-path prepending or local preference communities to influence routing.
  • Active-active configurations - Use ECMP to load-share traffic across multiple connections.
• Failover testing:
  • AWS provides a failover testing tool to simulate BGP down events.
  • Different SLA models based on redundancy (single, multi-site non-redundant, multi-site redundant).

Encryption and Performance

• MAX Sec encryption:
  • Encrypts the layer 2 traffic between the customer device and the AWS Direct Connect device.
  • Customers manage the encryption keys in AWS Secrets Manager.
• IPSec Tunnels:
  • Alternative encryption option, using the Direct Connect as the transport.
  • Recent increases in tunnel bandwidth (up to 5 Gbps) make this a more viable option.
• Performance:
  • Bandwidth limited by the EC2 instance, not the Direct Connect.
  • Single flow performance capped at 5 Gbps, so recommend using multiple flows.
  • Enable jumbo frames on private and transit VIFs for better performance.

Pricing and Cost Optimization

• Key pricing components:
  • Direct Connect port hourly charge.
  • Data transfer charges (in and out of AWS).
  • Transit Gateway attachment and data processing charges (if used).
  • Site-to-Site data transfer charges.
• AWS provides an open-source Networking Pricing Calculator to help estimate costs.

Emerging Capabilities

• Database Interconnect Multi-Cloud:
  • Allows creating high-bandwidth, resilient connections to other cloud providers (e.g., Google Cloud).
  • Leverages AWS's pre-built "maximum resiliency" Direct Connect model.

Key Takeaways

• Understand the physical and logical connectivity components of Direct Connect.
• Leverage monitoring and troubleshooting tools to identify and resolve issues.
• Design for resilience and high availability using features like Site Link and redundant connections.
• Optimize costs by understanding the pricing model and using the Networking Pricing Calculator.
• Explore emerging capabilities like Database Interconnect Multi-Cloud to expand hybrid connectivity options.