Q:

<Previously on Ask.DrPeering....>

DrPeering –

How should we think about expanding our peering and network footprint internationally?

Harry Zimm

Chili Palmer

Zimm Productions

A:

Harry -

In the previous article we discussed the three traffic types and corresponding revenue. Now we will exercise the model with a couple of peering and transit strategies for building into an emerging Internet Peering Ecosystems and why open peering is a sound strategy in new markets. We will use African peering ecosystems in our examples because of the rapid growth of these emerging Internet Regions.

Interconnection Strategies

The interconnection strategies usually consist of two parts: an Internet Transit strategy and an Internet Peering strategy.

Internet Transit Strategies

Transit strategies tend to be based on the company’s strategic intent and the market context.

Compete in the Transit Market. For example, if ISP A wants to be the leader in the wholesale Internet Transit across Africa, it might deploy a node and a small sales office in each region. In this case, ISP A might not care so much about the Internet Region itself, nor about the Peering Ecosystem dynamics in this Internet Region since it only wants to sell wholesale access to the Internet routes it picks up in Frankfurt for example.

Lead the Internet Market. On the other hand, if the strategic intent is to “own” or “lead” the market for certain Internet Regions, then they may want to accelerate the growth of this particular Internet Peering Ecosystem and take a more active role in the Internet Region. They might establish a larger local team and focus on sales and growing their own local hybrid peering and transit with an aggressive peering and customer attainment strategy. They may create an informal alliance with the local Internet Exchange Point, try to pull together the Tier 2 ISP population for peering, and perhaps establish a strategic alliance with an ISP that owns some local fiber.

Perform Transit Arbitrage. Let’s assume that ISP A seeks to rapidly build its customer base in several Internet Regions by arbitraging the transit markets between the inexpensive gateway country and the expensive home market. It wants to drive the price for Internet Transit down until the price of transit approaches the price of transit in the Gateway Internet Region.

The degree of competitive aggressiveness is a strategic decision. Consider the transit pricing trajectory:

The price of Internet transit in the U.S. began at $1200/Mbps in 1998 and was driven down to about $12/Mbps over ten years. The price for Internet Transit in South Africa, Nigeria and Ghana was hovering $1200/Mbps in 2008, so one might expect the price to follow a similar trajectory over the next ten years. If true, the African Internet Peering Ecosystem’s transit market might expect to see $10/Mbps by 2018.  The question is, how should ISP A ride this price curve down?

Figure 14-5 - History of Internet Transit pricing in the U.S.

Profit Maximization Internet Transit Pricing Strategy. One pricing strategy would be to offer Internet Transit at just below the market price, low enough to collect as many customers as possible at the highest possible price point. This is a conservative strategy that will be minimally disruptive.

Market Acquisition Internet Transit Pricing Strategy. An aggressive pricing strategy would be to undercut the market by the percentages shown in the graph (20%-50% per year) regardless of the market price. This is what Level 3 did in the early days and Cogent continues to do across the globe.

Next we turn our attention to Internet Peering in emerging Internet Peering Ecosystems and determining our peering inclinations and our peering policies, and how they may evolve as these ecosystems develop.

Peering in Emerging Internet Peering Ecosystems

Peering strategies are most relevant in the context of a particular Internet Region. For example, we need to understand:

1.How well developed is the Internet Peering Ecosystem?

2.Who are the Tier 1 ISPs in the Internet Peering Ecosystem?

3.Who are the Tier 2 ISPs and are they peering?

4.Where are the Internet Exchange Points (IXPs), who runs them, who is there, and what are the costs for participating (local loop, colocation, peering ports, membership fees, etc.)

5.What is the market price for Internet Transit?

6.What is the cost for transport capacity into the exchange?

7.What are the dominant traffic patterns?

8.How much traffic stays within the Internet Region? What are the popular destinations? Who are the leading access providers in the region?

We go through this data collection in each region under consideration. We then create the business case for peering in each region.

Next we will walk through a sample peering strategy case study based the model and discussion so far.

Case Study – African ISP A Expands into London

Strategic Intent. Let’s assume that ISP A is an African ISP that is interested in expanding into adjacent countries and more importantly, into London to purchase Internet Transit there for $2/Mbps instead of the $300/Mbps that is available in their home market. ISP A wants to provide better Internet Transit service for the home market while also being the price leader.

Our market research has uncovered some additional relevant context.

1) 98% of all African traffic is coming from the US and Europe. A trivially small percentage of African traffic is locally produced content.

2) With such a small percentage of traffic satisfied locally, it is not surprising that there is not much peering traffic exchanged within the Internet Regions across Africa.

Let’s enumerate some additional assumptions for the case study.

1)The cost of transport up to London is $275/Mbps.

2)Relevant content providers are in London and willing to peer openly.

ISP A’s Peering Policy in London

Since 98% of all African traffic comes from the U.S. and Europe, ISP A should peer openly in London. Given the original assumptions that the peers are all in the ISP A colocation center in London, that Traffic will either come from the Global Internet Transit interconnect or the free peer. We prefer this traffic from our free and local peer.

One could refine the analysis by performing a Peering vs. Transit calculation, but why bother. You could limit peering to those that meet certain criteria, or not bother with peering at all given the low price of transit. Those that will peer however will generally improve performance while simultaneously reducing costs.  Peering also provides a good sales and marketing tool since ISP A can then promote itself as having direct peering with the most popular content sites in the world up in London. Peering openly is aligned with our strategic intent.

ISP A should identify (or hire) a peering coordinator to build these peering relationships. This will require some travel budget, staff time, and a learning curve, but ultimately peering in Europe will be great preparation for peering in the African Internet Regions.

ISP A should take a community leadership role in each Internet Region

Many Internet Regions across Africa are just now starting to grow. There is little value derived from peering in these regions today. But these markets will not always be small. As the Internet Peering Ecosystem grows, the market for international transit grows. It is therefore in ISP A’s best interest to promote and help grow these ecosystems. Over time,

1.Broadband access (perhaps mobile) will become more widely and inexpensively available, and

2.Lower transit prices will lead to more local Internet attachments, and

3.More Tier 2 ISPs will therefore launch services for their local markets, and

4.The local Internet Peering Ecosystem will grow enough that local eyeballs will seek local content, and

5. The business case for peering will make sense.

If ISP A takes the leadership role now, promotes local peering and the local exchange points, it will not only help establish good will, but it will also grow a broader base of customer prospects that may ultimately prefer to purchase transit from ISP A.

In South Africa for example, MWEB won the hearts and minds in this Internet Region. They offered unlimited Internet access when others only offered metered service. They peered openly. They established a reputation for building the South African Internet. They then launched a public relations campaign to get the incumbent Tier 1 ISP to peer with them. The public relations rallied the populous to the cause. When the peering was initially denied (again), MWEB publicly announced that the traffic to Telekom South Africa was going to traverse London, and that people should complain to Telekom South Africa. Ultimately, MWEB was able to maneuver Telekom South Africa to peer according to some press coverage on the subject. The point is that this community leadership is a very powerful tool, and helped position South Africa as a continental Internet focal point. ISP A should do the same for each Internet Region of interest.

ISP A should adopt an Open Peering in each Internet Region of interest across Africa.

At this point during the peering workshops people usually ask, “Why will customer prospects buy from ISP A when they could just peer with ISP A for free?”

The answer is that Internet Transit and Internet Peering are not perfect substitutes.

ISP A’s peers only get access to ISP A’s downstream Internet Transit customers, which, as we have already established, is a very small percentage of the ecosystems Internet traffic (less than 5% of African traffic is local sourced and sunk). So initially there is very little value to peering with ISP A today. This peering discussion however can lead to this next discussion with prospects:

Internet Transit on the other hand provides access to all routes:

1.Routes learned from other ISP A customers, and

2.Routes from ISP A’s peers’ customers, and

3.Routes from ISP A’s upstream Internet Transit provider in London.

Ultimately, the local customers in each Internet Region should evaluate and compare the value of the ISP A transit network against others in the market.

The Value of an Internet Transit Network

When one of the local ISPs purchases Internet Transit from ISP A today, it is primarily buying access to the global Internet delivered out of London. Most of the value derived by purchasing Internet Transit from ISP A today is the quality of its network capacity up to London and the quality of the global routes it picks up there. The primary question customers should ask today is “How well does ISP A deliver traffic to me from the U.S. and Europe?”

Today, the secondary question is, “How well does ISP A meet my long-term local traffic needs?” Let’s consider the long-term effect of this open peering policy in the emerging local markets has on the value of the transit network (shown in Figure 14-6).

Figure 14-6 - The value of an Internet Transit network to its prospective customers.

Peering makes ISP A a more attractive Transit Provider

The ISP A network is a more attractive network to its transit customers when it peers its traffic locally. Why? Peering provides ISP A’s customers with direct access to ISP A’s peers’ customers. This means that customers get better performance when their traffic doesn’t have to go all the way up to London to be served! As the ecosystem develops, more local ISPs and more local content comes on-line, so the value of ISP A’s Internet Transit network increases since it peers this traffic better locally as shown in the figure above. From a marketing perspective, the performance benefits become an important differentiator, and from a public relations perspective, peering positions ISP A as promoting nationalistic interests. This will lead to more customers that want not only access to the global Internet but also want great performance to this emerging local content.

More Transit Customers makes ISP A a more Attractive Transit Provider

The ISP A network is a more attractive network to its transit customers as it obtains more customers. Why? First, the performance is even better when all customer traffic can be satisfied entirely on the same network. Second, these routes may be unique; if the customers are singly homed to ISP A, the next best alternative to reach these customers might require sending traffic through London. Therefore, a strong argument can be made that purchasing transit from ISP A will provide the best performing path to these local destinations.

A Strong Local Network of Customers and Peers will fend off Foreign Competitors

Over time, and as the Internet Peering Ecosystems across Africa develop, ISP A will be able to peer more of its traffic locally and therefore be able to demonstrate that much of this local traffic can be satisfied without leaving the country. These peering sessions will help it differentiate itself and help it compete aggressively against the foreign transit providers that build into this now more established Internet Peering Ecosystem. These foreign Internet Transit providers can initially only offer global routes from London, not the more desirable local routes that ISP A has cultivated in this emerging ecosystem. The competitor’s traffic will have to traverse London to get to ISP A’s customer base, and it may have to traverse London to get to ISP A’s peers’ customers as well. ISP A is in a more defensible market position from this strong local peering and customer acquisition strategy.

Over time, by driving down the price of Internet Transit, and perhaps by migrating to a selective peering policy, foreign competitors will see these markets as less lucrative arbitrage opportunities.

Emerge as a Tier 1 ISP in these emerging markets

At some point, a “Tier 1 ISP” club will probably emerge in these ecosystems similar to what we see in the rest of the world. These “Tier 1 ISPs” will peer only with each other and be able to reach all in-country routes through these free and reciprocal peering arrangements. They will adopt a restrictive peering policy in the country and de-peer all of those networks that are customers of the other Tier 1 ISPs. They will deny requests for peering with anybody else so competitors may have to send traffic through London to reach their customers. Every other Internet Region has gone through this transition and ISP A may, with this strategy, be well positioned to emerge as a Tier 1 ISP in various regions. The uniqueness of the routes and the greater volume of traffic may even allow ISP A to negotiate peering from a stronger power position in London.

Final Notes

We have been talking about ISP A and their interconnection strategy as a provider of Internet Transit, but we have not yet discussed how the model applies to content providers. Every developing Internet Region realizes the cost and performance issues when content is not available locally. As a result, the large-scale network savvy content providers are constantly invited to extend their presences into other Internet Regions. These content providers generally consider three things when evaluating these requests:

1)“How do they currently reach customers in this region, and what is the current price and performance for this traffic?” and,

2)“How might they better reach customers in this region and what would be the price and performance improvement?” and finally,

3)“What are the implications of expanding into the Internet Region?”

For content providers and content-heavy ISPs and CDNs, it is all about the end-user experience. They might even pay more for a substantial improvement in performance for an important user base. Companies like LinkedIn for example have a substantial, important, and growing user base in India and parts of South America. They might consider expanding into these markets directly rather than delivering their content from Japan or from the U.S.

So when Content Providers apply the model, they look at the current latency, packet loss, perhaps jitter and other metrics between customers and their servers via a transit provider from an adjacent but centrally located Internet Region. The next best alternative might be to expand into a new Internet Region and serve up the traffic locally. If they build in, how much of their traffic can be peered away for free, how much might Internet Transit cost there, and how much will it cost to build in? They will apply the business case for peering but instead of looking at the peering break-even point, these network savvy content providers primarily consider the number of milliseconds from the round trip time to display their web pages.

Generally content providers and content-heavy ISPs and CDNs will adopt an open peering policy globally. In some cases, the content provider or CDN may even require their hosting company in the region to pay for their bandwidth and hosting expenses.

Content companies and CDNs may require that the hosting company’s competitors be allowed to interconnect with the content company or CDN directly, something the hosting company may be disinclined to allow otherwise. No “walled gardens are allowed” has been Google’s approach as they deploy caches into emerging Internet Regions to help a local Internet Peering Ecosystem grow.

In South Africa, such was not the case when Akamai deployed a node into a hosting company data center that turned around and required competitors to buy paid peering fee to access this cached Akamai content. In fairness, the hosting company apparently paid for Akamai’s bandwidth expenses, so they were trying to recover their transit costs for seeding that cached content. The point is that the content owner’s have some cards that can be played here.

In my opinion, Google did it right, and Akamai did it wrong. It is in the content provider’s best interests to provide their content as topologically close to the end-users as possible, and do so openly so as many eyeballs as possible get great performance. The question is how to best balance these various forces in an emerging Internet Region when often the current distribute-from-afar solution works well enough.

Another set of issues that the content providers look at are around the business context in the foreign Internet Region. If they build into Europe for example, they may have to honor the strict European privacy standards. If they build into China they may need to honor the governmental censorship orders and deal with a strong-armed and invasive government controlled Internet Region. Some countries have large import duties and some require majority local ownership before a company is granted a license to offer telecommunications services there.

These are the sorts of discussions that come up during the peering workshops, and the framework serves as the skeleton for dissecting and discussing some of these types of topics.

Summary

These articles provided a simple model for exploring international peering strategies that have come up in almost every Internet Peering Workshop I have run. These group discussions surrounding the model exercise the definitions and helps us sketch out broad stroke strategies about where we want the “ISP A” in question to be in 5-10 years as the ecosystems develop. The simplifying assumptions have proven necessary to avoid getting caught up in the complexity and unique characteristics of each market.

This model has served me well as a 1st pass for exploring an international peering deployment. Refinements are often made to consider different price points, different peering policies, and different market penetration strategies for each region.

In this analysis we used for an example deploying across Internet Regions in Africa but the same logic applies to any international deployments. In the next few years much of the African Internet will emerge as a set of first-class independent Internet Regions. It may take some time, and there is risk here, but in a world where so much of the ecosystem is hardened, Africa looks to be a fertile ground for the next generation Internet consisting of a billion users consuming new local Internet content.

DrPeering@DrPeering.net