This paper presents Internet Peering from the Internet Service Provider and Large Scale Content Provider perspective.
In 2010, Internet Peering remains one of the most important and effective ways for ISPs to improve the efficiency of their operation. Peering is defined as “a business relationship whereby ISPs reciprocally provide connectivity to each others’ customers.”
ISPs seek peering relationships primarily for two reasons. First, peering decreases the reliance on and therefore the cost of purchased Internet transit. As the single greatest operating expense, ISPs seek to minimize these telecommunications costs. Second, peering lowers inter-Autonomous System (AS) traffic latency. Peering traffic exchanged direct between two peering ISPs is likely to take the lowest latency path. So how is peering done?
This paper details the ISP peering decision-making process from selection of potential peers through implementation based on a decade of research with the peering coordinator community. Interviews with Internet Service Providers have highlighted three distinct phases in the peering process: Identification (Traffic Engineering Data Collection and Analysis), Contact & Qualification (Initial Peering Negotiation), and Implementation Discussion (Peering Methodology). The first phases identify the 'who' and the 'why', while the last phase focuses on the 'how'.
Updated August 23, 2010
Over ten years of research with the Peering Community have revealed that much of the misunderstanding surrounding Internet Peering topics is based on a lack of common lexicon, so we will begin with some of the basic definitions.
Definition: The “Internet Peering Ecosystem” is a community of loosely affiliated network service providers that interact and interconnect their networks in various business relationships.
In order for ISPs to offer this service they need to connect their network to the Internet.
Definition: Internet Service Providers (ISPs) connect end-users and businesses to the public Internet.
This interconnection can take one of two forms - Internet Transit and Internet Peering which will be discussed in turn.
Definition: Internet Transit is the business relationship whereby one ISP provides (usually sells) access to all destinations in its routing table .
Figure - Transit Relationship - selling access to the entire Internet.
Internet Transit can be thought of as a pipe in the wall that says "Internet this way".
In the illustration above, the Cyan ISP purchases transit from the Orange Transit Provider (sometimes called the "Upstream ISP"), who announces to the Cyan ISP reachability to the entire Internet (shown as many colored networks to the right of the Transit Providers). At the same time, the Transit Provider propagates the Cyan routes across the Internet so that all attached networks know to send packets to the Cyan ISP through the Orange Transit Provider. In this way, all Internet attachments know how to reach the Cyan ISP, and the Cyan ISP knows how to get to all the Internet routes.
Some Characteristics of Internet Transit
Transit is a simple service from the customer perspective. All one needs to do is pay for the Internet Transit service and all traffic sent to the upstream ISP is delivered to the Internet. The transit provider charges on a metered basis, measured on a per-Megabit-per-second basis using the 95th percentile measurement method.
Transit provides a Customer-Supplier Relationship. Some content providers shared with the author that they prefer a transit service (paying customer) relationship with ISPs for business reasons as well. They argue that they will get better service with a paid relationship than with any free or bartered relationship. They believe that the threat of lost revenue is greater than the threat of terminating a peering arrangement if performance of the interconnection is inadequate.
Transit may have SLAs. Service Level Agreement and rigorous contracts with financial penalties for failure to meet service levels may feel comforting but are widely dismissed by the ISPs that we spoke with as merely insurance policies. The ISPs said that it is common practice to simply price the service higher with SLAs, with increased pricing proportional to the liklihood of their failure to meet these requirements. Then the customer has to notice, file for the SLA credits, and check to see that they are indeed applied. There are many ways for SLAs to simply increase margins for the ISPs without needing to improve the service.
Transit Commits and Discounts. Upstream ISPs often provide volume discounts based on negotiated commit levels. Thus, if you commit to 10Gigabits-per-second of transit per month, you will likely get a better unit price than if you commit to only 1gigabit-per-second of transit per month. However, you are on the hook for (at least) the commit level worth of transit regardless of how much traffic you send.
Transit is a commodity. There is debate within the community on the differences between transit from a low cost provider and the Internet transit service delivered from a higher priced provider. The higher price providers argure that they have better quality equipment (routers vs. switches)
Transit is a metered Service. The more you send or receive, the more you pay. (There are other models such as all-you-can-eat, flat monthly rate plans, unmetered with bandwidth caps, etc. but at the scale of networking where Internet Peering makes sense the Internet Transit services are metered.)
So why need anything else? There are primarily two motivations (engineering and financial) that we will cover later in this document. Needless to say though, the transit fees can get large as the traffic volume increases. Cable companies for example today purchase tens of Gigabits-per-second worth of transit today ! Even at today's low market transit prices, these transit fees add up.
Definition: Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.
To illustrate peering, considerthe figure below showing a much simplified Internet; an Internet with only three ISPs: WestNet, MidNet, and EastNet.
WestNet is an ISP with green customers, MidNet is an ISP with blue customers, and EastNet is an ISP with red customers.
WestNet is in a Peering relationship with MidNet whereby WestNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach WestNet's green customers.
EastNet is in a Peering relationship with MidNet whereby EastNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach EastNet's red customers.
Peering is typically a free arrangement, with each side deriving about the same value from the reciprocally arrangement. If there is not equal value, sometime one party of the other pushes for a Paid Peering relationship.
Important as well is that peering is not a transitive relationship. WestNet peering with MidNet and EastNet peering with MidNet does not mean EastNet customers can reach WestNet customers. WestNet only knows how to get to blue and green customers, and EastNet knows how to reach only blue and red customers. The fact that they both peer with MidNet is inconsequential; peering is a non-transitive relationship.
1) In your own words, what are the definitions of Internet Transit and Internet Peering?
2) What is the difference between Internet Peering and Internet Transit?
3) What would be the effect of having Peering relationships with some local ISPs and a Transit relationship with a large upstream ISP? What traffic would go along which path?
Now that we understand peering and transit relationships and the differences between them, we will examine the first phase of the Peering Process.
We spoke with over 100 Peering Coordinators in this study to document how peering works, and how they approach peering from a practical perspective.
The Peering Coordinator is an individual that executes the tactics associated with a peering strategy. A Peering Coordinator is the person that travels to peering forums, negotiates peering, shepards peers through the peering process of their company and represts their company interests as they traverse the peer's peering processes. After peering is etsbalished the peering coordinator maintains the relationship with the peers through upgrades, expansions, reconfigurations and terminations.To the peerign community, the Peering Coordinator is the face of the company for matters related to interconnections.
A good peering coordinator attends peering forums, speaks at conferences about interconnection techniques, technologies, operations activities, etc. This multidisciplinary job crosses the boundaries of network architecture, technical (routing), business (making the business case), public speaking and evangelism, and legal (negotiating contracts). This job therefore requires a mix of skill sets to be executed effectively.
Some organizations have a "Peering Steering Committee" often consisting of a multi-disciplinary team that handles peering activities. You would find the peering requests discussed with network planning, network engineering, legal, business development and maybe even sales, all evaluating the implications of accepting or denying peering.
(It is interesting that there is typically no designated title of Peering Coordinator in Asia - this activity is done as a side job for a network engineer. )
If Internet Peering makes sense from a technical and financial perspective, the next question is, “With whom should we peer?”
To identify potential peers, ISPs use a variety of criteria.
1) Quantities of traffic distributed between networks often sets the pace of the negotiation; to quantify this, ISPs may systematically sample inbound and outbound traffic flows. Flows then are mapped to originating AS, and calculations are made to determine where peering (direct interconnections) would most reduce the load on the expensive transit paths. There is substantial work involved here, as this traffic sampling results in a large number of data.
Once the top traffic destinations are identified and associated with specific ISPs, these ISPs are targeted for potential peering relationship discussions. Below is a "Peering Top 50 list" in the form of a downloadable spreadsheet that can be used to document large volume peering prospects.
Often the top traffic destinations have a restrictive peering policy and are unlikely to be interested in peering. These restrictive peers are highlighted in red in the spreadsheet.
Prescription: do not waste a lot of time trying to convince the regional Tier 1 ISPs (AT&T, Sprint, Verizon, etc. in the U.S.) to peer with you in the U.S.!
Those that do remain however are potential peering targets. Peering coordinators add up the so-called "peerable" traffic exchanged with these companies to determine how much traffic they might be freely peer. Collectively these ISPs are targeted as ideal candidates for a peering relationship.
A typical peering destination histogram looks something like the graph below, with the top targets being restrictive peers and representing a large chunk of traffic. The remainder of the traffic is potentially peerable.
Below is a sample top peering target list shared with the author in 2003.
Anonymously shared NetFlow Top 50 list (2003)
When this study was first done in 2001, many ISPs did not do any analysis of traffic to determine whom to peer with they used intuition. They assumed since they sold a lot of transit to access networks that they might get a lot of peering traffic from the content heavy ISPs.
We use the following phrase to capture the essence of this heuristic:
"Eyeballs love content"
At this time there were few options for network collections and only one in 20 did any sort of data collection and analysis. In 2010 most companies in the peering space do some analysis, leveraging Arbor, Renesys, iDoxio, or related services.
Interestingly enough, by far, the most detailed analysis was done by content providers. While an ISP might do the analysis monthly, using a fast machine running scripts over the weekend, Yahoo! for example had a, almost real-time identification of bad paths and adjusted peering and transit routes accordingly.
In any case, the end result of this first phase is list of the top 50 ISP candidates for peering. Interviews with Peering Coordinators highlighted a few other considerations.
2) Broader business arrangements between ISPs may circumvent the peering negotiation phase and expedite discussions directly to Phase III, the peering methodology negotiation phase. For example, in the early days AOL had enough dial-in model dollar volume with Level 3 that peering was used a leverage point to bypass the negotiation phases.
3) Do you meet the requirements articulated in the Peering Policy? Peering policies range from “open peering policy” meaning “we will peer with anyone”, to restrictive peering policies described as “if you have to ask, we won’t peer with you.” To obfuscate things, Peering policies are often exposed only under Non-disclosure agreements, and these policies reduce the number and types of ISPs that are peering candidates.
In many cases, peering requires interconnections at multiple peering points, explicit specifications for routing, migration from public (shared switch) peering to private (non-shared switch) peering after a certain traffic volume is reached, etc. It is beyond the scope of this document to fully explore the technical and political motivation for peering policies; it is sufficient to be aware that these discussions can be cumbersome and require a combination of technical and financial negotiation. See "A Study of 28 Peering Policies" for examples of peering clauses that can be used for developing your own peering policy.
The greatly simplified peer qualification decision tree looks something like this:
Once the Top 50 list is made, and it appears to be beneficial to peer, the ISP enters into Phase 2, Contact & Qualification, Initial Peering Negotiation.
In the U.S., ISPs typically have a Peering Coordinator or a “Peering Steering Committee” to evaluate peering requests. Some variations of the following steps lead to the parties either a) leaving the negotiation, or b) proceeding to the peering methodology discussions.
Interviews have highlighted a key challenge for ISPs.
Step 1) Finding the right person to speak with at the target ISP remains a very difficult and time intensive process.
Peering Coordinators change jobs and there is no standard way to find out who handles this task. Mergers and acquisitions cloud lines of communication. Even if the name is known, Peering Coordinators are often traveling, way behind in e-mail, and prioritizing e-mail based on the subject or the sender. This is where “people networking” helps a great deal, and hiring expertise for their contacts speeds this initial contact process up quite a bit. In some cases, peering is expedited between ISPs simply because the decision makers have a previous relationship . This was the dominant mode of operation in the early days of the Internet.
In any case, peering contacts are initiated in one of the following ways:
Step 2) Mutual non-disclosures agreements (NDAs) may be negotiated and signed, and a discussion of peering policy and prerequisites follow. Note that NDAs are an optional step, and many ISPs do not require signed NDAs prior to discussions . Traffic engineering discussions and data disclosure may be used to justify the peering relationship. Each ISP typically has a set of requirements for peering that include peering at some number of geographically distributed locations, sometimes at public exchange points.
Traffic volume is usually a key determining factor. The decision rule hinges upon whether or not there is sufficient savings from peering to justify spending capital on a port on a router and/or a portion of the interconnection costs or augmenting existing capacity into an exchange point. A Bilateral Peering Agreement (BLPA) is the legal form that details each parties understanding of acceptable behavior, and defines the arms length interactions that each would agreed to.
Another motivation for peering to factor in includes lower latency and/or more regional distribution of traffic than existing connections allow.
This process is diagrammed below.
Figure 6 – Phase 2: Contact & Qualification Decision Tree
Since Internet Peering is seen as being a relationship of mutual benefit, both parties now explore the interconnection method(s) that will most effectively exchange traffic. The primary goal is to establish point(s) of interconnection, and secondarily detail optimal traffic exchange behavior (using Multi-Exit Discriminators (MEDs) or other traffic weighting techniques).
To interconnect, ISPs face two distinct options: Direct Circuit Interconnection or Exchange-Based Interconnection (or some global combination thereof).
The legacy “Interconnection Strategies for ISPs” white paper quantified the economics and technical tradeoffs between the first two options in 1999. To summarize this report, the preferred methodology depends on the number of peers participating in the region and bandwidth required for its regional interconnections. ISPs that expect to interconnect at high or rapidly increasing bandwidth within the region, or expect interconnections with more than five parties in the region prefer the exchange-based solution. Those that do not anticipate a large number of regional interconnects prefer direct-circuits and typically decide to split the costs of interconnection with the peer by region. On occasion the costs are covered in whole by one peer .
Figure 7 – ISP Physical Interconnection Methods
For direct-circuit interconnects, key issues center upon interconnection location(s) and who pays for and manages the interconnection. This becomes a material cost issue as traffic grows and circuits increase in size and cost.
In either case, ISPs generally have the following goals for establishing peering:
A related white paper is called "Public vs Private Peering - the Great Debate" that discusses the tradeoffs between the two forms of interconnect within an Internet Exchange Point.
This section details the selection criteria an ISP typically uses when selecting an Internet Exchange Point. Note that these issues are listed in no particular order. These issues are shown graphically as flowcharts and discussed in detail in the paragraphs below.
Figure 8 - Exchange Environment Selection
These issues have to do with getting telecommunications services into the exchange. How fast can circuits be brought into the interconnection environment? How many carriers compete for business for circuits back to my local Point of Presence (POP)? For facilities-based ISPs, what is the cost of trenching into the exchange (how far away and what obstacles present themselves)?
Are there nearby fiber providers that lease strands? These questions will help answer the most important question to ISPs: How fast can my peer and I get connectivity into the exchange? Multiple carriers lead to speed and cost efficiencies. Some ISPs have volume deals with certain carriers or otherwise prefer carriers and therefore prefer exchanges where these carriers can quickly provision circuits. These answers strongly impact the desirability of the exchange environment.
These issues have to do with getting equipment into the exchange.
The answers to these questions impact the deployment schedule for the ISP(s) engineers and the costs of the interconnection method.
This issue is based on the following observation by the peering coordinators: The most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange.
|#1||I am already Installed here||You are already installed here|
|#2||I am already Installed here||You have to build in|
|#3||I have to build in||You are already installed here|
|#4||I have to build in||You have to build in|
There is a hidden assumption here that there is sufficient capacity to interconnect at the exchange in either Public Peering or Private Peering arrangements. Cross-connects or switching fabrics can easily establish peering within a few hours or at most days. ISPs will prefer to interact where one or both ISP already has a presence.
These issues focus on the ongoing operations activities allowed within the exchange after initial installation. Does the exchange allow private network interconnections? Are there requirements to connect to a central switch? How is access and security handled at the facility ?
Is there sufficient power, HVAC, capacity at the switch, space for additional racks, real time staff support ? Is it easy to upgrade my presence over time? Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges over those with restrictions over future operations.
“Bandwidth, strategic partner alliances, and corporate ties often override the technical justification .” – Lauren Nowlin, (at the time, Peering Coordinator for Onyx Networks.)
Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)? A neutrally operated exchange (defined as one that is not owned or aligned with any carrier, fiber provider, or ISP) provides an open distortion-free marketplace for carrier and ISP services.
Market distortions often result when an exchange is owned by one of its participants. This often manifests itself in requirements (required use of their carrier or ISP services) that constrain the market for services within the exchange . Since it is difficult and disruptive to move equipment out of an exchange, ISPs will prefer a neutrally operated exchange environment that will not suffer from market distortions and limitations due to business conflicts of interest.
This broad issue crosses all other issues. What is the cost and value of this Internet Exchange Point? What are the rack fees, cross connect fees, port fees, installation fees? What are the future operating fees going to be? What are the motivations and parameters surrounding these fees? Cost issues shadow most of the other issues listed in this paper.
Below is an estimation of the value of participation in public peering at various IXes, based on publicly available data in late 2009. It shows that the massive amount of traffic that is being exchanged in peering relationships is substantially more cost effective than if that traffic traversed their upstream transit provider.
Source: Modeling the Value of an Internet Exchange Point, http://DrPeering.net
All else being equal, ISPs will seek to minimize the costs, particularly upfront costs, associated with the interconnection for peering.
The credibility issue is twofold.
First, credibility goes to the financial support of the exchange. Does the exchange exist today and will it exist tomorrow? During the early stages of the exchange, ISPs are asked to make a leap of faith when committing, and therefore prefer an exchange with strong backing and the credibility to survive.
Observation: The Internet Exchange Points exhibit the characteristics of what economists call the “Network Externality Effect”; the value of the product or service is proportional to the number of users of the product or service .
Internet Exchanges are a special case of this effect; the value of an Exchange Point is not the number of participants but a slightly more complex calculation including the number and uniqueness of the routes and volume of traffic peered. Since the value of the IX to an ISP is proportional to the amount of traffic the ISP can exchange in peering relationships at the IX, the value of the IX to the peering population follows the network externality graph as shown below.
On the Y-axis we see the value of the Internet Exchange Point. On the X-axis we see the independent variable that causes the value of the IX to increase; we will use the number of participants for now; as more participants connect to the exchange for peering, the more value an NSP could derive from peering at the IX
All Internet Exchange Points go through the following growth curve. They start out with zero or a few founding ISP members, and face the challenge of attracting additional peers into a building where there are not many peers to peer with. This is the "startup hump", and solutions to this problem were shared with the author in "The Art of Peering: The IX Playbook".
Once the IX reaches critical mass, where the value of participation exceeds the cost of participation, the a well positioned IX experiences exponential growth.
Second, does the exchange operator have the backing and credibility to attract the more valuable peering candidates? Since the value of the exchange (shown in the graph below) is proportional to the number and type of participants. Does this exchange have the backing to attract my peers? Who is managing the exchange and what technology is in use? These answer signal the credibility and survivability of the exchange. ISPs will prefer an exchange with credibility – one that is financially and technically well backed and likely to attract the most desirably peering candidates.
These issues focus on the side benefits to using this exchange. Are there other ISPs at this exchange that are peering candidates? Are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)? ISPs will prefer an established and well-populated exchange, particularly one with potential customers that can generate revenue.
There are many operational exchange points in each region of the U.S. There are also emerging (soon to exist) exchanges that may be considered as peering points. However, given the pace of ISP expansion, it is unlikely that emerging exchange offerings are differentiated or compelling enough to be preferred over existing exchanges. Chronic traffic congestion can influence the decision to plan to peer in an existing malfunctioning exchange or wait until a better exchange opens. Customers with heavy flows of regional traffic can also influence the decision. Long term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one.
The ISPs we spoke with shared with us varied weightings of the importance of each of these issues. To some, the most important issues were the business issues, and others weighted more heavily the operations issues. Each ISP places higher or lower importance on different issues and not surprisingly select their operations environment based on their specific criteria.
This paper describes the decision processes ISPs follow to identify and establish peering relationships. It explores the implementation phase and the criteria for exchange point selection.
The results of the interviews with ISP Peering Coordinators can be summarized with the following observations:
This paper was the result of interviews with key Internet peering coordinators, interactions with ISP support staff, and side conversations at various meetings and forums. Special thanks to a few folks who contributed their time and ideas to help create this early draft report: Ren Nowlin (Onyx networks), Joe Payne (IXC), Dave Diaz (Netrail), Jake Khuon and Alan Hannan (Global Crossing), Dan Gisi and Jeff Rizzo (Equinix), Patricia Taylor-Dolan (Level 3 Communications), Sean Donelan (AT&T Labs), Avi Freedman (AboveNet), Patrick Gilmore (Onyx), Geoff Huston (Telstra), Steve Meuse (GTE), Aaron Dudek (Sprint), Raza Rizvi (REDNET), Wouter van Hulten (InterXion), Michael Palmaffy (Equinix), and Beau Sackett (Telus).
 “Maturation in a Free Market: The Changing Dynamics of Peering in the ISP Market“ by Jennifer DePalma
 “NAPs, Exchange Points, and Interconnection of Internet Service Providers: Recent Trends, Part I: 1997 Survey of Worldwide NAPs and Exchange Points”, Mark Knopper
 Peering Contact Database, Restricted Access; for Peering Coordinators only. To received send e-mail to email@example.com with AS#, company name, peering e-mail address, mailing address, phone numbers, and contact names.
 List of ISPs and NOC contact info: http://puck.nether.net/netops/nocs.cgi
 ISP Survival Guide, Geoff Huston, Wiley Publishers
 Data Communications “Old Boys’ Network”, Robin Gareiss http://www.data.com/issue/991007/peering.html
 Australian’s take on Interconnection competition issues, http://www.accc.gov.au/
 IWS2000 Presentations on Internet Exchange futures: http://hsn-crl.koganei.wide.ad.jp/ipsj-hqi/prev_workshops/5.html
Internet Transit Pricing Historical and Projections
Index of other white papers on peering
WIlliam B. Norton is a consultant, author of The Internet Peering Playbook: Connecting to the Core of the Internet, and highly sought after public speaker and international recognized expert on Internet Peering. He is currently engaged as the Executive Director and Consultant for DrPeering.net, a leading Internet Peering portal and consultancy. With over twenty years of experience, Mr. Norton focuses his attention on sharing his knowledge with the broader community in the form of presentations, Internet white papers, and private on-site consultations.
From 1998-2008, Mr. Norton’s title was Co-Founder and Chief Technical Liaison for Equinix, a leading Internet Peering Exchange and colocation center provider. From startup to IPO and until 2008 when the market cap for Equinix was $3.6B, Mr. Norton spent 90% of his time working closely with the Peering Coordinator Community. As an established thought leader, his focus was on building a critical mass of carriers, ISPs and Content Providers and documenting the core values that Internet Peering provided, including the Peering Break Even Point and Estimating the Value of an Internet Exchange.
To this end, he created the white paper process, identifying interesting and important Internet Peering operations topics, and documenting what he learned from the peering community. He published and presented his research white papers in over 100 international operations and research forums. These activities helped establish the relationships necessary to attract the set of Tier 1 ISPs, Tier 2 ISPs, Cable Companies, and Content Providers necessary for a healthy Internet Exchange Point ecosystem.
Mr. Norton developed the first business plan for the North American Network Operator's Group (NANOG), the Operations forum for the North American Internet. He was chair of NANOG from May 1995 to June 1998 and was elected to the first NANOG Steering Committee post-NANOG revolution.
Connect on-line: http://DrPeering.net – RSS: http://ask.DrPeering.net – e-mail: wbn@TheCoreOfTheInter.net
The Peering White Papers are based on conversations with hundreds of Peering Coordinators and have gone through a validation process involving walking through the papers with hundreds of Peering Coordinators in public and private sessions.
While the price points cited in these papers are volatile and therefore out-of-date almost immediately, the definitions, the concepts and the logic remains valid.
If you have questions or comments, or would like a walk through any of the paper, please feel free to send email to consultants at DrPeering dot net
Please provide us with feedback on this white paper. Did you find it helpful? Were there errors or suggestions? Please tell us what you think using the form below.
Contact us by calling +1.650-847-1809 or sending e-mail to info (at) DrPeering.net