Dr

Peering

Transit Illustration. Consider the picture below in which the ISP EastNet purchases transit from a Transit Provider (also called an “Upstream” ISP) that has access to the Internet (shown as many colored networks behind Upstream ISPs). As a result of this transit relationship, EastNet receives access to all network routes in the upstream ISP’s routing table. In the picture below this is shown as colored circles (route announcements) being conveyed with arrows. In return the upstream ISP receives and announces EastNet routes across all of its peering and transit interconnections. As a result, EastNet gains connectivity to the entire Internet as known to its upstream ISP.

Internet Transit. So why need anything else?  One issue is that the transit fees can get quite expensive as the transit traffic volume increases. Cable companies for example today purchase tens of Gigabits-per-second worth of transit, and many video-based content companies are approaching 100’s of Gbps. Even at $5/Mbps, these companies may be paying 10Gbps*$5/Mbps=$50,000 per month. 

The other motivations for exploring Internet Peering are related to Engineering, Marketing, and Business issues we will discuss next.

Enter Internet Peering...

Based on hundreds of conversations with ISPs, Cable companies, Content Providers, and others, four primary motivations for peering emerged:

1. A)Lower Transit Costs. Business models for ISPs and Content Providers are often dominated by telecommunications costs. Highest among these costs are Internet transit fees that provides the ISP with connectivity to the global Internet. As their largest expense, many of these companies are motivated focus energy on minimizes their transit bills.
   

2. B)Better Performance. As a side effect of interconnecting directly with their peers, ISP customers generally experience lower latency to the other ISP’s customers. Traffic destined for a local competitor’s customers may need to traverse a couple of transit providers and potentially across great distances (with high latency) before reaching the other customer. The worst example from the research highlighted that traffic (at the time) between the United Arab Emirates and Saudi Arabia traversing an overloaded exchange point all the way back in Washington DC ! Peering has become popular so we don’t tend to see such examples anymore. The point is, through direct interconnections, ISP customers realize better performance.  Content companies in particular cited this as their primary motivation.
   

3. C)Usage-based traffic billing. Some ISPs charge their customers based upon metered traffic. Since packet loss and latency slows traffic consumption, these ISPs benefit from a lower latency, lower packet loss Internet. It is in their best interest therefore to assure that customers use as much bandwidth as possible by minimizing loss and latency through effective traffic engineering.  Dave Rand (AboveNet) cited this as a primary motivation for “Open” peering at AboveNet.
   

4. D)Marketing Benefits. ISPs identified wide scale high capacity peering as a marketing pitch that large content providers found somewhat compelling. The ability to scale and handle spot events was cited as a motivation for large content distribution network companies to peer as well.
   

For these reasons, we will set a foundation of definitions throughout this site, starting with the basics:

Definition: The Big ‘I’ Internet is a network of networks that implement the Internet Protocol (IP).

Definition: An Internet Service Provider (ISP) is an entity that provides (usually sells) access to the Internet.

An ISP must therefore get connected to an entity that is already connected to the Internet. There are two ways to do this: Internet Transit and Internet Peering.

Definition: Internet Transit is a service in which ISP sells access to the global Internet.

Definition: Peering is the business relationship whereby ISPs reciprocally provide access to each others’ customers.

To illustrate peering, consider figure 1 below showing a much simplified Internet; an Internet with only three ISPs: WestNet, USNet, and EastNet. WestNet has customers shown as green circles. USNet has customers of its own (blue circles) and EastNet has its customers shown as yellow circles.

Internet Peering can be thought of as a local optimization -- a way of bypassing the transit meter.

Next Page: Who will peer with me?   Peering Inclinations & Peering Policies

The research showed that there are several reasons why ISPs and Content Providers may decide NOT to peer with others.

1. 1)There may be the potential for transit sales if they don’t peer. Some ISPs will not peer if there are any existing or pending customer-provider relationships between the parties, even if the sale is completely unrelated to interconnection (i.e. fiber sale or colo sale).
   

2. 2)Peering consumes resources (router interface slots, circuits, staff time, etc.) that could otherwise be applied to revenue generation. Router slots, cards, interconnection costs of circuits or Internet Exchange environments, staff install time are incremental expenditures. Further, operating costs, particularly for peering sessions with ISPs without the necessary on-call engineering talent, can require increased processing power for filters and absorb time better suited to paying customers.
   

3. 3)Motivation not to commoditize IP transit. Tier 1 ISPs compete on the basis of better performance. They accomplish better performance due to the large customer base of direct attachments and high-speed interconnections with other Tier 1 ISPs. Since peering with other ISPs improves the performance of the “peer” it effectively makes them a more powerful competitor. Therefore, there is a strong disincentive to peer and increase the number of top tier competitors.
   

4. 4)As a “peer” there are no Service Level Agreements (SLAs) to guarantee rapid repair of problems. Both parties benefit from the reparation of outages, and may even have clauses in the peering agreements to work diligently to repair the problem. However, a customer relationship (with or without SLAs) generally has more contractual teeth (financial repercussion for failure to perform).  Content Providers generally cite this as a motivation to prefer a customer relationship. They like the contractual “Teeth”.  (Some ISPs cited consider this a running joke - they price the SLA service based on the likelihood of not having to pay the SLA fees; the service is the same, just priced and perhaps given some priority to repair things when they break. )
   

5. 5)Tier 1 ISP Market Protection. John Milburn put it best when he said “The desire for Tier 1 ISPs to protect their home market far exceeds any benefits that Peering may provide.”
   

6. 6)Tier 1 ISPs have all the Peering they need. Tier 1 ISPs are defined as ISPs that have access to the entire Internet Region (domestic) solely from their free Peering relationships. Therefore, by definition, they can reach any destination in that Internet Regional routing table for free. Therefore, they don’t need any more domestic peering.

Figure 1 - Transit Relationship - selling access to entire routing table

Transit is a simple service from the customer perspective; all one needs to do is pay for the data service and all traffic sent to the upstream ISP is delivered to the Internet. The transit provider typically charge on a metered basis, the metered rate specified in a dollars per-Megabit-per-second ($/Mbps) basis.

The volume of traffic used as the multiplier is typically measured using a 95th percentile measure of 5-minute samples across an entire month. That is, byte count samples are taken every five minutes, and at the end of the month the deltas are stacked lowest to highest; the value at the 95th percentile is the volume upon which the transit customer is charged. Use formula to start your spreadsheet:

Monthly TransitFee ($) = TransitPrice($/Mbps) * TransitVolume95th(Mbps)

Transit providers often provide a tiered pricing schedule based on the level of commitment. The larger the volume the customer commits to, the lower the unit price for transit. If you commit to a large transit volume, but only use a small portion of that volume, you are still billed as if you used that minimum commit. So companies have a challenge to balance their commitment level and their expected growth rate to minimize their monthly transit bill.

Acknowledgements

In Internet Peering discussions, there is often more heat than light. 

There are several reasons for this.

First, there are misunderstandings based upon different uses of terminology, or the misuse of terminology. For example, the word ‘transit’ means something different in the telephony world than in the Internet industry. The author has on more than one occasion received the question

“When my browser places an Internet connection to Yahoo!, how to the intermediary ISPs get paid for transiting that traffic?”

This question often comes from folks well versed in the telephony industry and learning how things work in the Internet world.

Second, unlike the telephony industry, the Internet business models are continuously in flux. The services offered over the Internet have grown from simple static web pages over HTTP to rate adaptive video to peer-2-peer and Web 2.0 AJAX desktop replacement apps, with new business models emerging. All of this is evolving over a relatively consistent IP layer, but business models at this layer have evolved to include caching and CDN services. The point is, there is a continuous flow of changes to the ecosystem compared to the telephony world.  New terminology leads to information asymmetry which leads to misunderstanding.

Finally, there are commercial interests at the heart of these discussions, and combined with the lack of transparency (many companies are inclined to only disclose things like traffic volumes, future plans, etc. as necessary), people discussing peering matters are inclined to believe ill motivations are at play.  The reality is often that only rational and well-reasoned commercial interests are at play, although there are many cases of short-sided thinking at the root of commercial policies.

Next Page: Who will peer with me?   Peering Inclinations

Click here to see a tiered graph of the current market prices for transit along with a historical transit pricing trend graph and projections into the future.