AWS Direct Connect for SysOps

AWS Direct Connect (DX) is a cloud service that makes it easy to establish a dedicated network connection from an on-premises data center to AWS. Unlike a VPN over the internet, a Direct Connect link is a physical, private, and dedicated connection that bypasses the public internet. This results in more consistent network performance, lower latency, and reduced bandwidth costs for high-volume data transfer.

Organizations with hybrid workloads – where some infrastructure runs on-premises and some on AWS – often need reliable, predictable connectivity. Over the internet, traffic can be subject to congestion, variable latency, and security risks. Direct Connect provides a private path that is more secure (no internet exposure) and offers a Service Level Agreement (SLA) of up to 99.99% availability for the connection.

At a high level, the process is:

- Port Speeds: 1 Gbps, 10 Gbps, and 100 Gbps (the latter only at select locations). You can also order a hosted connection from a Direct Connect Partner at speeds like 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, and 10 Gbps. - Virtual Interfaces (VIFs): - Private VIF: Used to connect to a single VPC. You must associate it with a Virtual Private Gateway (VGW). Supports VLAN tagging (802.1Q) with a VLAN ID between 1 and 4094, excluding reserved VLANs (0 and 4095). BGP authentication is optional but recommended. BGP timers: Keepalive interval = 30 seconds, Hold time = 90 seconds (default). - Public VIF: Used to access all AWS public services (e.g., S3, DynamoDB, EC2 public endpoints, CloudFront). You cannot use a public VIF to reach a VPC directly – that requires a private VIF. Public VIFs also use BGP. The BGP ASN can be public or private (you must own the public ASN). - Transit VIF: Used with AWS Transit Gateway to connect multiple VPCs and on-premises networks. A transit VIF is associated with a Direct Connect Gateway, which then connects to a Transit Gateway. - Direct Connect Gateway: A global resource that allows you to connect multiple VPCs in different AWS Regions (same or different accounts) via a single private VIF. The Direct Connect Gateway must be in the same Region as the VGW for each VPC. - Virtual Private Gateway (VGW): A regional resource attached to a VPC. It supports BGP dynamic routing and can be used with VPN or Direct Connect. The VGW has an Amazon-side ASN (default 64512, but you can specify a private ASN when creating it). - BGP Attributes: - Local Preference: Default 100. Used to influence outbound traffic from AWS to on-premises. - AS Path Prepending: Used to influence inbound traffic (on-premises to AWS). You can prepend AS numbers to make a path less preferred. - Link Aggregation Group (LAG): You can bundle multiple Direct Connect connections into a LAG to increase bandwidth or provide redundancy. A LAG can contain up to 4 connections of the same bandwidth. - MACsec: You can enable MACsec (802.1AE) on dedicated 10 Gbps or 100 Gbps connections for encryption at Layer 2.

On your on-premises router (Cisco IOS example), you configure:

interface GigabitEthernet0/0/0
 description To AWS Direct Connect
 no shutdown
!
interface GigabitEthernet0/0/0.100
 encapsulation dot1Q 100
 ip address 169.254.10.1 255.255.255.252
!
router bgp 65000
 neighbor 169.254.10.2 remote-as 64512
 neighbor 169.254.10.2 timers 30 90
 neighbor 169.254.10.2 password MyBGPSecret
 address-family ipv4 unicast
  neighbor 169.254.10.2 activate
  network 10.0.0.0 mask 255.255.255.0
 exit-address-family

To verify the BGP session:

show ip bgp summary
show ip bgp neighbors 169.254.10.2 routes

On AWS side, you can verify using the AWS Management Console, CLI, or API. For example:

aws directconnect describe-virtual-interfaces --virtual-interface-id dxvif-xxx
aws directconnect describe-connections --connection-id dxcon-xxx

Common issues: - BGP session not coming up: Check IP connectivity (Layer 3), VLAN tagging, BGP neighbor IP, ASN, and authentication. Use ping from the on-premises router to the AWS BGP peer IP (169.254.x.x). - Routes not being advertised: Ensure the VPC route table has propagation enabled from the VGW. Check that the on-premises router is advertising the correct prefixes. - Performance issues: Verify bandwidth utilization on the port. Consider using jumbo frames if supported. Check for packet drops due to MTU mismatch. - Redundancy: For high availability, you should have at least two Direct Connect connections to different locations. Use BGP attributes (Local Preference, AS Path Prepending) to control traffic flow.

A financial services company runs its core banking application on-premises but uses AWS for analytics and machine learning. Daily, it transfers terabytes of transaction logs to S3 and runs compute jobs on EC2. Using the internet would be expensive and unpredictable. They deploy a 10 Gbps Direct Connect connection to an AWS Direct Connect location near their data center. They create a private VIF to a VPC containing their analytics cluster, and a public VIF to access S3 directly. The result: consistent latency (~2 ms), 40% lower data transfer costs, and no bandwidth contention. They also set up a VPN backup over the internet using the same VGW. BGP is configured with local preference to prefer the DX path. Monitoring via CloudWatch metrics (ConnectionState, VirtualInterfaceBgpStatus) ensures they detect BGP failures quickly.

A global e-commerce company has on-premises data centers in Virginia and Oregon. They want to replicate data to AWS in us-east-1 and us-west-2 for DR. They order two Direct Connect connections: one at a facility in Virginia (for us-east-1) and one in Oregon (for us-west-2). They use a Direct Connect Gateway (DX Gateway) to connect both VPCs (in different regions) to a single private VIF per location. The DX Gateway is created in us-east-1 and associated with both VGWs. This allows them to advertise a single set of on-premises routes to both VPCs. BGP AS path prepending is used to make the cross-region path less preferred for traffic from AWS to on-premises. They also enable jumbo frames on the 10 Gbps connections to maximize throughput for database replication.

At scale, monitor the port utilization. A single 10 Gbps connection can saturate. Use Link Aggregation Groups (LAGs) to combine multiple connections. For applications sensitive to packet loss, ensure jumbo frames are enabled end-to-end. The Direct Connect SLA requires at least 99.99% availability per connection (for dedicated connections).