Connect the Dots with Internet Peering at IX Points

This post by DigitalOcean network administrator Tim Vogler explains what IX points are, why they’re integral to the medium-sized networks of the Internet, how DigitalOcean uses them, and how you can encourage your local networks (such as your ISP) to be good neighbors.

All over the world, IX points connect the dots of the Internet.

IX points, short for Internet Exchange points, are where companies, schools, internet service providers (ISPs) and other organizations connect their traffic directly to each other over a single Local Area Network (LAN).

Some notable IXs include LINX in the UK, AMS-IX in the Netherlands, and NYIIX in New York City, to name a few. DE-CIX in Germany hits peak traffic at 4.7 Tbps each day with an average of 2.78 Tbps and the next biggest IX, AMS-IX, is no slouch with 4.27 Tbps peak and 2.49 Tbps average. That’s a whole lot of cat videos being pushed.

Cat video throughput is a whimsical way to measure traffic, but the Internet actually relies on these direct connections between providers. Let’s imagine, for a moment, what would happen if nobody used IX points.

If nobody used IX points, most Internet traffic would go through ISPs on the public Internet.

The routing protocol that connects different networks together to form the Internet is called BGP, or the Border Gateway Protocol. This protocol handles the giant web-like structure of the Internet and its 500,000+ routes. While necessary for the Internet to work, BGP does have its shortcomings. It leaves out the latency of a route when deciding where to send traffic, and it is also heavily tuned by engineers, who occasionally make mistakes.

BGP’s main decision-making mechanism is path length. For BGP, this means the fewer networks along the path from origin to destination, the better. Instead of counting router hops (number of physical devices in the path) it counts how many autonomous systems (or an organization’s network) it crosses to reach the end network. Some examples of autonomous systems include ISPs and large networked enterprises such as AT&T, DigitalOcean, CloudFlare, and NTT.

Here’s an example of a route that takes a very convoluted path to reach its end point. (A route is the path traffic takes from start to end):

Here’s a text copy of the same route:

89.32.120.0/22      \*\[BGP/170\] 00:46:26, MED 1000, localpref 100  
AS path: \*\*4637 3356 5588 48095\*\* I, validation-state:unverified  

The interesting part here is the list of providers after “AS path,” which is four different providers including the start and end points. So, traffic on this route needs to pass through two intermediary providers before reaching the end user’s ISP. Each hop can introduce latency or packet loss.

How can we address this issue? You might have guessed by the title of this post: IX points, of course!

IX points send data over self-contained layer 2 networks (that use fast layer 2 switching) instead, shortening the routes data needs to travel, and thus reducing the cost and latency of sending data over the Internet.

Here’s another route showing a direct connection between DigitalOcean and CloudFlare, unencumbered by middleman networks:

103.31.5.0/24      \*\[BGP/170\] 2w6d 18:21:37, MED 200, localpref 100  
AS path: \*\*13335\*\*  

As you can see, the traffic hops directly to Cloudflare’s network without having to wade through the depths of the public Internet.

IX points provide a better user experience because traffic follows a shorter, faster path that’s easier to control and provide network consistency on. In fact, the more IX points, the better the Internet functions.

Setting up a peering relationship on an exchange is extremely simple. You’ll need to contact the organization that runs the IX to get a LAN connection and IP address. Once you have that, it’s as simple as sending an email with your interface details and asking politely to peer.

With our newest datacenter in Toronto, for example, all it took was an email with our ASN and an IP address to set up a peering relationship with Cloudflare. We send a lot of traffic their direction, so the mutual benefits made it a no-brainer.

It’d be great if the entire Internet was fueled by IX points. Achieving 100% peering isn’t realistic, considering that there are 300 IXs worldwide, you need to be close enough to plug a cable in, and someone has to pay for the switching equipment (usually fueled by membership fees).

However, connecting any sizable network to its neighbors can help make the Internet better for everyone. If you have access to an IX, reach out to the other members to set up some sessions and connect those dots.

by Tim Vogler