This post is part of a multi-series blog to help folks prepare to take the AWS Certified Advanced Networking Exam. As the title indicates, this section is dedicated to Direct Connect, which is a major topic for the exam.
Introduction to BGP
BGP operates over tcp_179 and requires manual peering. By design, there is no auto-discovery. BPG is a path-vector – not link-state or distance-vector. BGP shares the best path to a destination with its peers – it does not share every path it knows. BGP will take the path with the least amount of hops. If two routes are equal, it will use the first route that it receives. A router only knows about its directly connected links or those learned from some other protocol. It will advertise those routes and prepend its own routes when advertising to its neighbors. Path choice can be flexibly changed as needed to ensure the desired path between networks. BGP does not care about the speed of the links to a network, therefore features such as AS_Path Prepending, MED, Local_Preference, and weights are critical to ensuring ideal routing.
  • AS Path Prepending – can be used to influence traffic patterns – basically it advertises the link to look like it has more hops, making the route look less desirable to the receiving router.
    • BGP prefers routes with the fewest hops, therefore lower is preferred
    • for Direct Connect, advertising more hops affect return traffic from AWS back to your premises
    • an example would be to advertise additional hops for 1GB Direct Connect to a DR site rather than the Direct Connect to the primary data center
  • MED – Multi-Exit Discriminator – used to influence inbound routing for a site with multiple links
    • lowest preferred
    • used to influence return traffic from AWS
  • Local_Pref – similar to weight, allows you to change the preference to the desired link over another – unlike weights, local_pref is advertised to other routes via iBGP
    • local to the AS using the same AS number (iBGP)
    • default value = 100
    • the route with highest local_pref is preferred
    • For DX, it affects outbound traffic to AWS
  • Weight – change preference to a link – local to only the router the weight is configured on and is not always supported – weights are not advertised to other routes
    • affects outbound traffic from the router
    • highest weight wins
    • Cisco default weight is 0 and the value can be from 0 to 65535
Direct Connect Introduction
Direct Connect is one of the few solutions on the AWS platform that is actually a hardware solution. However, because it requires physical hardware, it is not an instant setup. Through a Direct Connect location, you co-locate your own router which will physically connect to the AWS Direct Connect router with a cross-connect fiber. When ordering from AWS, you can only order a 1GB or a 10GB connection. You can, however, order slower speeds when working with a 3rd-party partner, like CenturyLink or Equinix. The location is generally a co-location facility where AWS owns its own racks and equipment. They are NOT located in an AWS region, however, each AWS Direct Connection is affiliated with one, and only one, AWS region and it’s important to ensure you’re selecting the correct region and location when requesting a connection through the AWS console. Once the request is made for a connection, AWS will provide a Letter of Authority (LOA) to connect to their router.
It’s important to note that Direct Connect is not resilient by default and if you require redundancy, you need to either order a second DX connection or combine it with a secondary solution, such as a VPN backup. Using BGP, these solutions can be configured to provide automatic failover. If you do order multiple connections through the same facility, AWS will ensure that the connections are provisioned on different routers to ensure the routers are a not single point of failure. To design an even better solution, use multiple DX locations, as most of the regions have multiple Direct Connection locations to allow customers to create connections through multiple facilities.
A DX connection is much more expensive than using VPN but you get additional benefits, such as reduced bandwidth costs, consistent network latency between your premises and the AWS VPC(s) you’re connecting to, as well as consistent network performance. Remember that a DX is a private, physical connection and does NOT traverse the public Internet (like VPN does). Due to the cost, many businesses choose to use a Direct Connect connection with multiple AWS accounts and multiple VPCs for connectivity. It’s also important to note that traffic traversing a Direct Connect is NOT encrypted by default unless you are encrypting it at the application level before transit. You can configure an IPSEC VPN over the Direct Connect (using public VIFs – more on that later) to ensure traffic is encrypted in transit, if needed.
A Direct Connect is actually trunk from a port on a router using 802.1q. Using that port, you’ll connect to your router, signifying a vlan for the connection. The connection between your router and the AWS router requires single-mode fiber:
  • 1GB = 1000Base-LX
  • 10GB = 10GBASE-LR
If the Direct Connect is acquired through a third-party partner, that connection is likely shared among multiple customers and the partner will use VLANs to segregate traffic between customers. The partner owns the cross-connect (and associated charges for it) and because you don’t own the link, you don’t get to modify or control the VLANs traversing that link. This connection will be referred to as a hosted connection and each hosted connection is a single VLAN for a public or private virtual interface (VIF). If you need multiple VIFs or connections to multiple VPCs, you’ll likely need multiple hosted interfaces from your partner.
By default, Direct Connect = single router, single port, single region.
In the console, a Direct Connect is identified as follows: DXConn-xxxxxxxx
Direct Connect – Virtual Interfaces
Virtual Interfaces are used to connect to your account/VPC (using private VIF) or to connect to public AWS resources such as S3, SES, SQS, etc (using a public VIF).
For a private VIF, there is a BGP session established between customer router and DX router. You provide the BGP AS number or you can have it auto-generated and the VLAN number which links the physical layer of the customer route with this virtual interface. You can also enter the router peer addresses (/30) or have AWS auto-generate one for you.
For a public VIF:
  • You should use a public AS number, preferably one that you own. You can use a private AS number but it removes capabilities over the connection (like AS path prepending).
  • You need to use public IP addresses for the peer addresses. You can use your own or log a ticket to have AWS assign you a /30.
  • Enter a VLAN
For a private VIF:
  • You can use a private ASN which is 64512 through 65535.
  • You can use your own private IP addresses or have it auto-generate a 169.x.x.x/30
  • Enter a VLAN
If you’re creating the VIF in your own account  (same as where the DX connection lives), you’ll need to specify the VPG when it will terminate. If you’re creating the VIF for another account, regardless if it’s another account you own or somebody else’s, you’ll need the account number and you will NOT need to specify the VPG which to terminate. The account owner will need to accept the hosted connection in their account as that account will be responsible for the data charges over that VIF (but won’t be responsible for port-charges for the DX itself).
For a public VIF, you will advertise your public IP addresses to AWS and AWS will advertise the public IP addresses of its public services to your DX router. In the USA, you will get IP addresses of ALL public services from ALL of the US regions, which means that you can connect to any public service over a public VIF in any US region, regardless of which region your DX terminates. AWS uses BPG communities to identify the order of a prefix, specifically the NO EXPORT entry, which allows the BPG advertisement to internal neighbors but not external neighbors. Other BGP communities uses are:
  • Internet – informs a BGP neighbor to advertise the prefix to all BGP neighbors
  • No Advertise – not to advertise the route to any BGP neighbors
In addition to the text entries, AWS also makes use of numerical entries to identify where advertisements came from:
  • 8100 – Prefix is local region
  • 8200 – Prefix is Local continent
  • 9100 – YOU advertise to local AWS region
  • 9200 – YOU advertise to local; AWS continent
Community string means that the prefix came from the local region indicated by community value and that it originated on this continent indicated by 8200 – since a region is in the continent it makes sense that 8100 & 8200 would be used together.
Direct Connect Billing
DirectConnect itself has two charges associated with it, the hourly port charge and data transfer.
Hourly Port Charge
There is an hourly port charge for the physical connection between your router and the DX router. The amount of this charge depends on the speed and location of the Direct Connect and it is billing to the account where the Direct Connect interface is added (the physical connection, not the VIF).
Data Transfer
The second charge is for data transferring over the virtual interfaces. Remember that data IN to AWS is always free and data charges are for data from the VPC going OUT of AWS, measured in GB/month. It’s important to notate that the account where the virtual interface appears in the account is charged for the data traversing the interface and not necessarily the DX owner (although they may be the same account in some/most instances).
Data Charges – A Flow Chart
Here’s a good example of where and when you should expect charges within your AWS account, VPCs, and external connections