This paper is about when it makes sense to use an Internet Exchange Point and when it makes sense to use point to point circuits for interconnections.
Over the last few years, two models for ISP interconnection have emerged: the exchange-based interconnection model and the direct circuit interconnection model. The exchange-based model involves a meet point at which ISPs exchange traffic whereas the direct circuit interconnection model, as the name implies, involves a point-to-point circuit between the exchange parties. This paper compares these options from a technical and business (financial) standpoint by the introduction and application of the Neutral Internet Business Exchange as the basis for the exchange-based model comparison. Real world costs and revenue projections have been used in a financial model to quantify the cost savings and revenue generated by participants in both models.
last updated 5/21/99 - recreating this - still needs work...
The #1 ISP Challenge: Scaling Bandwidth
As the Internet has grown in number of users and types of use, a large number of companies have recognized that its continued success depends on a continually increasing supply of bandwidth. What most have failed to recognize is that the Internet, as a network of networks, depends on its interconnections as much as its network capacity. Scaling the Internet requires not just trenches full of dark fiber, but an infrastructure that permits a rich set of interconnections among the networks that make up the Internet. The Internet requires a mechanism to scale these interconnections in a technically efficient and cost efficient manner.
The #2 ISP Challenge: Operations Environment Not Controlled by Competitors
The Internet remains an environment of intertwined technical and business relationships. ISPs participate in mutually interdependent relationships with competitors in order to provide global connectivity. Exchange facilities provide the physical environment for interconnection, but are typically operated by a party with a vested interest; almost all exchanges in operation today are run by an ISP, a carrier, or a provider that participates in both markets. Circuits into a carrier-run exchange typically must be purchased from the carrier operating the exchange, and activities allowed are generally tightly restricted1. Collocation within an ISP facility typically requires buying transit from that ISP, thus reinforcing a competitor that will be competing against that customer in the future. These conflicts and exchange environment restrictions and requirements make it difficult for competitors to share infrastructure and realize economies of scale.
The #3 ISP Challenge: Increase Transit Sales
Finally, ISPs, particularly those aspiring to become nationals, are under increasing pressure to acquire prestige content provider accounts. This goal has strong implications spanning the financial (paying for peering or transit relationships), marketing (who buys from whom implies hierarchy), and inter-provider negotiation space (who brings more valuable content to the table). Premier content providers require the ability for ISPs to bundle collocation, access to multiple (potentially many) ISP networks, and the ability to offer a presence in many geographical areas. The combination of these forces (scaling requirements, exchange neutrality, and prestige content provider transit sales) lead to the need for a neutral third party to build and operate highly scalable neutral business exchanges for the ISP community. This paper next introduces the Neutral Internet Business Exchange model as the basis for comparison against these three goals, and as basis for comparison against the other interconnection strategy: direct circuit interconnection.
The model is based upon the basic notion of "a free market for Internet services". As shown in the graphic below, the Neutral Internet Business Exchange Model provides an environment for three types of participant to conduct business. Carriers bring in fiber and sell circuits to ISPs. ISPs are presented with a state-of-the-art operations environment from which they can purchase high bandwidth circuits from carriers as fast as running a piece of fiber between cages. This allows bandwidth to be provisioned within hours as compared to months to provision additional landline circuits. ISPs exchange traffic with other ISPs and content providers in a peering or transit sales relationship. All participants benefit from the business relationships enabled in this model, and all share in the cost savings from the exchange economies of scale.
Figure 1 - Neutral Internet Business Exchange Model - Facilitating Internet Business
This model satisfies the conflict of interest concerns described in ISP Challenge #2, but how does this model solve the other challenges highlighted (scaling bandwidth and transit sales)? How do ISPs operate more cost effectively and make money with this model? The rest of the paper will discuss solutions to ISP Challenge #1 (scaling bandwidth) and #3 (increasing transit sales) through comparison of the direct circuit interconnect model against the Neutral Internet Business Exchange.
ISP interconnection takes one of two forms: peering and transit. In either case, interconnection requires provisioning of some bandwidth between parties. One approach is to use a direct circuit interconnection approach.
Figure 2 – Direct-Circuit Interconnection Model
The direct circuit interconnection model requires the lease of point-to-point circuits between parties that scale linearly with the number of ISPs. Each party typically pays half the cost of the circuit by alternating who pays for each regional exchange circuits. From interviews with ISPs, the cost of these circuits is typically on the order of $11,400 for OC-3 interconnection and $23,000 for OC-12 circuit2. The cost of interconnecting with n other parties using the direct circuit interconnection model is therefore:
ISPDirectCircuitInterconnectionCost = (n −1) * CircuitCost /2
From a backbone management perspective, there are operational implications to be considered with this model. Traffic engineering and operations complexity increases over the increasing web of links, involving multiple local loops and IXCs. A larger number of machines need to add non-revenue-generating ports to sustain a mesh between participants. The circuits traverse miles of fiber underground, subject to outages due to construction (and the rapid proliferation of backhoes3). The previous picture shows a simple topology; imagine the additional troubleshooting burden when the number of interconnections grows to 100! As the bandwidth requirements grow, ISPs need to upgrade circuits to many different places; they are unable to take advantage of traffic aggregation over a very high bandwidth circuit. Through application of our financial model we will demonstrate that the cost of this interconnection strategy does not scale either.
Solution to Challenge #1 (Scaling Bandwidth): Large Pipes into a Neutral Internet Business Exchange
An alternative to the direct circuit interconnection model (shown below in Figure 3) is for ISPs to purchase instead a much larger circuit into the Neutral Internet Business Exchange to take advantage of the relatively inexpensive4 and the rich connectivity mesh within the exchange. The same interconnection that occurred using the direct circuit interconnection model can occur in the exchange-based model with some notable advantages.
Figure 3 - Exchange-based Interconnection Strategy
From the ISP perspective, very large pipes into the exchange scale the aggregate interconnection traffic more effectively than the POP-multiple direct-circuit interconnection alternative. ISPs typically realize a 2:1 to 3:1 economy from traffic aggregation of partially filled pipes5. ISPs understand the traffic aggregation model well. The key to the exchange-based model of interconnection is that it is far more efficient to scale a single high bandwidth, point-to-point circuit than many lower speed point-to-point circuits. We will quantify this saving in our financial model in the next section.
Operationally, ISPs can also make use of the robust infrastructure within an exchange, and scale their interconnection capacity as quickly as running a piece of fiber within the exchange (compared against ordering a new land line circuit). This highlights one of the central motivations for the Neutral Internet Business Exchange Model - the interconnection dependency tenet:
The greater the dependence on the interconnection, the more hardened and scalable that interconnection environment should be.
As a side effect of the Neutral Internet Business Exchange Model, ISPs have access to an additional source of revenue: transit sales to content providers and ISPs. We will expand and quantify additional transit opportunities later. We will first apply real world costs and compare the two strategies (direct circuit interconnection and exchange-based interconnection) using our financial model.
Cost Savings Analysis. Our financial model compares the cost of the direct circuit interconnection model against the cost of the exchange-based model. There is a potential for substantial savings for ISP interconnection under the exchange-based model. There are three drivers to the savings equation. First is the low-cost of the fiber cross connects for interconnection within the Neutral Internet Business Exchange . The second and offsetting cost is the additional expense of pulling a big circuit into the Neutral Internet Business Exchange. The third is the economies of the aggregation of small partially used pipes into a large pipe.
In our first examination of the models, consider a direct comparison of the direct circuit interconnection model and the exchange-based model as we saw side-by-side in Figure 3 above. In the direct circuit interconnection model above, the four half circuit costs are compared against a large pipe into the exchange with four cross connects between the participants. In our financial model, we use the more conservative 2:1 aggregation efficiency figure. The cost of both of these models varies over the number of participants. The plot in Figure 4 shows the cost savings from using the exchange-based model over using the direct circuit interconnection model.
Figure 4 - Cost Savings Curve - Exchange vs. Direct Circuit Interconnection Model
The first point and step in the cost savings curve is the cost of purchasing an OC-12 into the facility to accommodate up to 4-OC-3 (plus 2 more realizing the benefits of traffic aggregation) interconnections within the exchange6. Up to approximately five participants, there are no cost savings from the direct circuit interconnection model. When the sixth interconnection is made, there will be saving $3,000 per month in interconnection costs, and when the seventh interconnection is made, there will be a $8,500 per month savings7.
The second step in the curve shows the cost of upgrading the circuit to an OC-48 to accommodate the existing interconnections and scale to up to 16-OC-3s (plus 8 more realizing the benefits of aggregation). After the tenth interconnection is made, and so on throughout the plotted future, the exchange-based model is far more cost effective, saving up to $160,000 per month in interconnection costs.
Again, the key to this model is the scaling allowed through aggregation over a very large point-to-point circuit. It is also important to note that the savings escalate not only across the number of participants, but also as the bandwidth requirements increase!
One savings not quantified in our model is cost savings from the free market. Carriers offering circuits recognize the circuit sales potential in this model and take advantage of advances in fiber technology such as DWDM to more effectively scale the point-to-point fiber run from their POP into the exchange. This results in lower prices for these big circuits for ISPs. We ignored this savings in our financial model.
In the financial model, we also did not take into account the cost savings of using the Neutral Internet Business Exchange as a point of presence from which to launch ISP services. If a new POP were built directly into the Neutral Internet Business Exchange, then the ISP would not incur the cost of a POP elsewhere.
Facilities-based providers win big in the exchange-based model. As the capacity grows and Dense Wave Division Multiplexing (DWDM) becomes economically feasible, there will be an order of magnitude greater savings realized by facilities-based ISP. We'll explore the most ambitious facilities-based providers – those that trench and install their own fiber. The initial outlay of capital (estimated to be $2M in the case we explored8) is typically amortized over twenty years at 12%, resulting in a monthly cost of about $20,000/month - approximately half the monthly cost of a large circuit (OC-48) in our model. By applying DWDM technology, the facilities-based ISP can incrementally allocate lambdas to seamlessly9 scale the bandwidth into the exchange. As shown in the cost savings graph below, the facilities-based ISP that pulls fiber into the Neutral Internet Business Exchange will realize significant cost savings when interacting with at least 5 participants at OC-3 and can scale this large DWDM pipe to realize substantial savings.
Figure 5 - Cost Savings of Exchange-Based Model (using dark fiber) over Direct Circuit Interconnection Model
Again, this works well because it is far more efficient to scale a single large pipe than to scale many smaller pipes. This is due to the benefits of aggregation, and the scaling properties of advances in optical switching technology.
This analysis extends quite well into the transoceanic fiber provisioning into the exchange. Consider the following quote from Dave Rand regarding the AboveNet transatlantic fiber leased out of the AC-1 build:
"We paid $8.3 million for a trans-Atlantic OC-3 for 25 years. That breaks down to about $300,000 per year or less than $30,000 a month. Of course the reality is that the OC3 will be fully loaded within a few years and will therefore need to be upgraded. But the point is that the connectivity cost is significantly lower than buying it on a monthly basis."
Whether trenching and deploying dark fiber (as we did in our analysis) or leasing dark fiber as Dave did across the ocean, the same order of economies apply. The ability for ISPs to scale this bandwidth into the exchange provides substantial cost savings.
To close out this section, consider the summary graph below (Figure 6) comparing the cost of interconnection for each of the three models. As the direct circuit interconnection model scales linearly (shown as red squares), the exchange-based model scales more cost effectively. Using a large circuit to aggregate the traffic shows a step curve cost function (shown as blue triangles) resulting from the incremental cost of the larger pipes into the exchange. Dark fiber providers win big by scaling the large pipe seamlessly as exchange traffic scales by changing the speed of the fiber interconnects (shown as red X's ). For interconnection of a small number of participants (about five) the direct circuit interconnection strategy is rational from a financial standpoint. However, as the number of participants and the bandwidth requirements increase, either of the exchange-based models prove far more cost effective for interconnection. Given the Internet growth trends, the seemingly insatiable demands for cost-effective bandwidth, and considering the conflict of interest issues described earlier, ISPs will prefer the Neutral Internet Business Exchange model for scaling their ISP interconnections.
Figure 6 - Cost of Interconnection for 3 Interconnection Strategies
In the analysis so far we have ignored the transit sales opportunities afforded those participating in the Neutral Internet Business Exchange. We will discuss this next.
Solution to Challenge #3: Increase Transit Revenue.
Understanding the cost curves for ISP interconnection, ISPs can also gain incremental revenue through participation in the Neutral Internet Business Exchange. Consider that unlike the direct circuit interconnection model the Neutral Internet Business Exchange Model provides access to a market for additional transit sales opportunities: ISPs and large Content Providers. Consider that Content Provider requirements include:
Likewise ISPs (regional, international, etc.) seek:
Quite simply, the Neutral Internet Business Exchange Model supports both sets of requirements while the direct circuit interconnection model alternative can not. For example, consider that the time to obtain a terrestrial circuit can take upward of a year11 and the provisioning of a fiber cross connect between cages can be accomplished within 24 hours.
Figure 7 - Direct Circuit Interconnection Model vs. Exchange-Based Model for ISP to Content Provider Interconnection
Content providers can achieve their multi-homing goals in one of two ways. One way is to purchase circuits into their corporate headquarters and operate a data center of their own. If we assume that the central corporate headquarters has been built to include sufficient space conditioned with HVAC, electrical, security, backup generators and UPS infrastructure, then the issue becomes that of interconnection methods. A large pipe into the content provider site shows similar scaling properties to direct circuit interconnect ($11,400/month) except the cost of interconnection is fully the responsibility of the content provider. Unless multiple data centers are used, the geographic diversity property requirement is not met. While the simplicity of this model is appealing, the inability to aggregate traffic, the expense and time delay of incrementing bandwidth, and the inability to change ISPs easily make the direct circuit interconnection model less attractive for content providers.
The exchange-based approach meets the requirements by providing a market place with a low cost of entry for content providers. Assume that the content provider locates its content at one or more exchange centers. Using mirroring software, content providers realize high-speed low latency access to end-users on the Internet. In the financial model, the incremental cost for ISPs to connect to the content provider with fiber cross connects to ISPs within the exchange is $200/month. Under the direct circuit interconnection model, the incremental cost for transit sales would be at least the full OC-3 circuit cost: $11,400/month. Therefore, ISPs at the exchange can offer a much more cost-competitive bid. The key to this model is the centralization of content and multiple ISPs, and the scalability of the bandwidth into and within the exchange. In this way, ISPs can compete aggressively against those who are not taking advantage of the economies, and content providers benefit from the open market for bandwidth and connectivity.
Assume next that the ISP is at the exchange but the content provider is not at the exchange and needs to expand its presence into the exchange to achieve the economies discussed. The additional expense for the content provider is simply the racks (assume $3000 per rack for example) and servers required to host the content (necessary in all cases), and the cross connects to the ISPs at $200/month. This represents a modest investment for expansion compared with the cost of building a presence. Compared against the cost of the ISP selling an OC-3 worth of transit over a circuit back to the content provider's POP, the content provider should be indifferent between the two options, yet gains great scalability, connectivity options, and a hardened environment in which to operate. Content providers will prefer to make use of the Neutral Internet Business Exchange for these reasons. The ISP faces the same cost savings dynamics as described above, yet realizes the additional multi-homing transit revenue that was unavailable before. ISPs participating in the Neutral Internet Business Exchange model can offer very competitive pricing for content providers.
There is also a trend in the industry away from vertical integration towards outsourcing of value added services such as e-mail, usenet news, caching, web hosting, access (dial-in to DSL etc.) etc. so that ISPs can focus attention on branding, marketing, and customer relationships. This emerging trend enables companies like Covad and NorthPoint to focus on DSL and resell access services to ISP in the exchange. In this way ISPs can select from a diversity of service providers to bundle together and offer to their customer bases. The security and infrastructure of the exchange allows this flexible integration, and scales well through low-cost internal cross connects. The end result is that the exchange will grow into a net source of data, resulting in an even greater need for bandwidth scalability.
We will now examine a case study to quantify the short-term revenue opportunity at the exchange.
Equinix participation projections show a mix of 6 carriers, 50 ISPs, and 4 large autonomous Content Providers leading to about 50 or so peering and transit sales opportunities within the first year of operation13. (Note that the Neutral Internet Business Exchange Model allows for dual-homed content within ISP space. Equinix believes that the resell market will support a much larger number of interconnection opportunities for these content providers located within ISP space in the exchange. The financial model assumes transit opportunities only with the large autonomous content providers and uses these more conservative numbers.)
Assume that an ISP obtains only 7 transit sales and hands off traffic to 15 other ISPs through peering arrangements. We will examine the revenue generated and the cost savings realized through participation in the Equinix Internet Business Exchange (IBX).
For example, if the ISP in question achieves the following mix of interconnections, it can increase revenue through participation in an Equinix IBX within 12 months.
Based on these projections and the street price for transit, we see that the combination of cost savings and additional revenue opportunity result in a sustainable net cash flow increase of $6,456,000 per year, starting within the first year of participation. It is important to note that the revenue can be realized without collocation, however, the cost of the sale will be higher due to the higher cost of the circuit. Furthermore, ISP participants will be able to quickly turn up transit sales if they are already at the IBX. Finally, the most compelling reason to participate early is that the early ISP participants will become the incumbent ISPs and will have access to the greatest number of sales opportunities. In this way, those early participants will more quickly realize the cost savings and sales revenue of interconnection via the exchange-based model.
We have described the Neutral Internet Business Exchange Model and demonstrated the technical and business benefits of this approach over the direct circuit interconnection model for ISPs. In particular, the direct circuit interconnection model suffers from the scaling over both bandwidth and number of participants. As the bandwidth and number of participants grow, the number and size of circuits must increase in a way that disallows economies of scale through traffic aggregation.
We demonstrated through a financial model that the cost savings function is an incremental function leading to potentially significant savings. We further demonstrated the substantial revenue opportunity realized through participation in a Neutral Internet Business Exchange. In the conservative example, over $3 million of incremental cash flow was realized by the sample ISP. As the number of participants grows, the value of the exchange grows, the benefits to the participants grow, and the cost for bandwidth decreases. We further demonstrated that the first-to-market in a Neutral Internet Business Exchange will realize the greater revenue by being able to participate in a greater number of business opportunities.
Next Studies: With the exception of the fiber transmission gear, we did not include equipment cost for either interconnection strategy. These costs may be material at different stages of the cost curve for both strategies, however, additional cards and chassis characteristics and requirements vary widely across vendors. To take these into account would require engineering tradeoffs and architectural considerations that ultimately result in additional steps in the cost savings curve as new hardware is added to the configuration. Finally, this incremental hardware cost allocation would involve amortization calculations, which also vary by accounting practices. This study is expected to be both valuable and require a large set of assumptions.
Additional work needs to be done quantifying the benefits of managing traffic asymmetry. An exchange with the proper mix of net producers and net consumers of bandwidth allow an ISP to more effectively utilize expensive land lines by fully utilizing both directions of the link.
Peter Lothberg Rodney Joffe (Centergate) Dave Rand (AboveNet) Steve Feldman (WorldCom) Paul Vixie (M.I.B.H., LLC) Dave O'Leary (Juniper) Tony Bates (Cisco) Stephen Stuart (M.I.B.H., LLC) David Conrad (M.I.B.H., LLC) Doug Humphrey (SkyCache) Ted Hardie (Equinix) Bill Woodcock (Zocalo) Pat Binford-Walsh (UUNET) Lauren Nowlin (______) Vab Goel (Qwest) Joe Payne (IXC) Jeff Rizzo (Equinix) Bill Manning (USC/ISI) Jeff Payne (Real Networks) My Mom
Index of other white papers on peering
WIlliam B. Norton is the author of The Internet Peering Playbook: Connecting to the Core of the Internet, a highly sought after public speaker, and an international recognized expert on Internet Peering. He is currently employed as the Chief Strategy Officer and VP of Business Development for IIX, a peering solutions provider. He also maintains his position as Executive Director for DrPeering.net, a leading Internet Peering portal. With over twenty years of experience in the Internet operations arena, Mr. Norton focuses his attention on sharing his knowledge with the broader community in the form of presentations, Internet white papers, and most recently, in book form.
From 1998-2008, Mr. Norton’s title was Co-Founder and Chief Technical Liaison for Equinix, a global Internet data center and colocation provider. From startup to IPO and until 2008 when the market cap for Equinix was $3.6B, Mr. Norton spent 90% of his time working closely with the peering coordinator community. As an established thought leader, his focus was on building a critical mass of carriers, ISPs and content providers. At the same time, he documented the core values that Internet Peering provides, specifically, the Peering Break-Even Point and Estimating the Value of an Internet Exchange.
To this end, he created the white paper process, identifying interesting and important Internet Peering operations topics, and documenting what he learned from the peering community. He published and presented his research white papers in over 100 international operations and research forums. These activities helped establish the relationships necessary to attract the set of Tier 1 ISPs, Tier 2 ISPs, Cable Companies, and Content Providers necessary for a healthy Internet Exchange Point ecosystem.
Mr. Norton developed the first business plan for the North American Network Operator's Group (NANOG), the Operations forum for the North American Internet. He was chair of NANOG from May 1995 to June 1998 and was elected to the first NANOG Steering Committee post-NANOG revolution.
Read his monthly newsletter: http://Ask.DrPeering.net or e-mail: wbn (at) TheCoreOfTheInter (dot) net
The Peering White Papers are based on conversations with hundreds of Peering Coordinators and have gone through a validation process involving walking through the papers with hundreds of Peering Coordinators in public and private sessions.
While the price points cited in these papers are volatile and therefore out-of-date almost immediately, the definitions, the concepts and the logic remains valid.
If you have questions or comments, or would like a walk through any of the paper, please feel free to send email to consultants at DrPeering dot net
Please provide us with feedback on this white paper. Did you find it helpful? Were there errors or suggestions? Please tell us what you think using the form below.
Contact us by calling +1.650-614-5135 or sending e-mail to info (at) DrPeering.net