The Peering Advice Column

Peering Gravitational Pull on Transit Prices

Fri, 27 Jan 2012 16:25:40 -0800

Q: DrPeering -
How does having a strong regional Internet Exchange Point improve the regional Internet?

What is the effect of peering on the transit prices in nearby countries?

As an incumbent in a developing Internet Peering Ecosystem, we enjoy strong pricing power. But we now have to project the decline of our transit pricing, they say, due to the dropping price of peering and transit in far away countries !

What are the dynamics at play here? Why does peering, a completely different interconnection system, in a distant country, have any affect on the price of transit in my home market ?

Sue Ho-Jornt

A:
Sue -
This is a really interesting question to explore here. I can illustrate the effect with an example that came up...
I spoke with an ISP network engineer building from Romania who was building into Frankfurt. He said that they were paying about $6000/month for a 10G circuit into the DE-CIX. Why were they paying for a circuit into Frankfurt? Because the price of transit in Romania was very high, and it was cheaper to pay for the transport costs and buy transit (and peer) in Frankfurt.
So the question I pondered is:
Do strong regional IXPs have a “Gravitational Pull” on the price of Transit in the region?
I believe the answer is, yes, that peering greases the skids of dropping transit prices in the region. I often see the Internet Peering Ecosystems go through the following phases:
1) An risk-taking ISP feels that it is paying too much in transit fees and looks to buy transit (and perhaps peer) in a better transit market.
This Romanian ISP might pay $10,000/month ( transport, equipment, colo, 10G port, etc.) to build into Frankfurt to buy $2/Mbps transit.
If they can buy 5Gbps at $2/Mbps then have a total cost of $20,000/mont ($10,000/month for transport + $10,000/month for 5G of transit in that remote market) which works out to $4/Mbps for transit. If transit in the home market is more than $4/Mbps, then an ISP will do this. As the ISP can peer traffic away for free at that location, the price will drop even further.
Let's assume the price of transit is $10/Mbps in Romania. The sequence of events continues as follows.
2) This Romanian ISP buys transit for $4/Mbps, and sells transit in their home market for $8/Mbps and still make a healthy margin. Other ISPs see what this ISP is doing and follows suit, building into Frankfurt to get that cheap transit to resell. Over time, these ISPs all peer more of their traffic away, continuing to drop their total cost of accessing the Internet.
1) The incumbent ISP in Romania sees its larger customers buy transit in the more attractive market and drops its price in response.
Over time, the market price of transit in Romania drops towards that $4/Mbps price point. It drops toward the average cost of traffic exchange, demonstrating that the cost of peering and transit in a remote location can directly affect the price of transit somewhere else.
Peering Dynamics At Play Here
1) Transport getting cheaper and enabling the business case for peering.
Discussions with European ISPs led to the following approximations on transport pricing: ISPs utilizing a local 10G circuit into a local exchange point might see a price of $2000/mo, while coming in from an adjacent country might cost $4000/mo and coming in from far away (but still within Europe) might result in a cost of $6000/mo. These are rough numbers but they seem to be about right, so I show them in the diagram below as t, 2t, and 3t.

The cost of transport (t) is variable here. The other costs of peering (colo, IX membership fees, peering ports) are generally fixed for all participants at any particular IXP in Europe - they do not discriminate.
Therefore, a key driver for dropping Internet costs in your home transit market, is the cost of transport into and out of the country.
2) Peering Ports are getting cheaper and enable the business case for peering.
The European IXes are generally not-for-profit associations and therefore continually drop their peering port pricing to avoid making a profit and losing their not-for-profit status. A 10G peering port that used to cost $10,000 per month just a few years ago now costs $1400-$2300 across Europe. So in addition to buying cheap transit, an ISP from Romania can also peer away some of its traffic very inexpensively.
The lower the total cost of peering remotely, the lower the price of transit in that remote market, the lower the price of transit that this Romanian ISP can offer in its home country.
Transit Prices have always declined in competitive markets.

1) Other Home Market Drivers
Other factors include the amount of local content, the percentage of traffic served only by the incumbents, the number of Tier 1 ISPs, the number of Tier 2 ISPs and their home market strength, inclination to peer and for a peering community to come together, etc.
Summary
The Gravitational Pull of Peering on regional transit prices is real. It is enabled by the dropping prices of international transport and it is powered by the dropping price of transit along with the continually dropping cost of peering in other markets.
Therefore, as the incumbent, you can approximate that in a rational market, you can hold a transit price as high as
[ RemoteMarketTransitPrice (T) +CostOfRemotePeering (t) ] / CapacityOfTransport
Since the costs of transit have always dropped, I agree that you can expect that they will continue to drop. If this Romanian ISPs can buy transit in Frankfurt for $2/Mbps and it costs $10,000/mo to offload 5Gbps on a 10G circuit, you might see transit in your Romanian market drop down towards $4/Mbps or so in the next year or so.
This would be the logic I would use to make your estimation. Thanks for the question Sue!

DrPeering@DrPeering.net
PS - I am using round numbers for clarity, your actual numbers will of course vary.

CDN Pricing Trend

Tue, 3 Jan 2012 14:23:32 -0800

Q: DrPeering -
I am currently doing some research on the Internet market structure. The
historical and projection of IP transit prices (on page 47 of your book)
are very helpful. I am wondering if you have similar figures for CDN
prices. They don't have to be very accurate and official, just want to
get some idea/feeling about the CDN prices.

Mal Reynolds

A:
Mal -
The short answer is that I haven’t been tracking the price of CDN services.
However, my recollection is that CDN distribution of web objects was initially about 30% or so higher during the early days of CDNs (2000-2001 era). And why not - it was a better service than transit since it provided performance acceleration by locating the objects closer to the eyeballs.
Customers started comparing the costs of transit and CDN services, but there was a challenge: CDNs charged on GB downloaded - $0.12/GB downloaded for example back in the day. But transit was charged on a Mbps basis - $10/Mbps for example, and comparing volume downloaded vs. meter at 95th percentile was difficult. So CDNs started offering 95th percentile billing.
Competition creeped in, and emerging CDNs started undercutting Akamai, charging only maybe a 20% premium above the cost of transit, and then 10% above transit.
Around 2008 I seem to remember the price of transit and the price of CDN converged to became about the same and have remained so since.
You will find some more aggressive CDNs offering lower prices than the market price for transit but they are pretty close. If the price is the same, CDNs provide a very compelling value proposition as compared with transit.
Q: DrPeering -
Our Internet traffic volume has grown 4000% in the last year. We are trying to build out a network team now and look into peering. What resources are available?
Astrid Levenson
A:
Astrid -
I would suggest that you peruse the DrPeering.net web site and perhaps buy the Internet Peering Playbook. I say that not just because I wrote it, but because it provides a convenience form factor for all of the material you will need right now.
Some suggestions...
As you think through the budget for your team, make sure to include a healthy travel budget and send a designated peering coordinator to the Internet operations meetings in the regions where you intend to build out.
Have conversations with the leading colocation vendors in the markets you are considering - see how much free help you can get from them. They have an incentive to prove the value of their colocation space to you, so they should be willing to help you make the business case for peering there.
If you haven’t already done so, start the traffic analysis now. Identify the top traffic destinations and sources. This will help you identify the locations for peering and the selection of transit providers as you build out a hybrid peering and transit solution.
If you need some interim help as you build out your permanent team, send me a note. I know a few networking guys that can help you spin up and do so on a consulting basis until you have your staff in place.
This should help get you started.

DrPeering@DrPeering.net

When does peering make sense?

Wed, 21 Dec 2011 10:03:23 -0800

Q: DrPeering -
When does it make sense to peer?

Internet transit is so inexpensive, I don't have the networking expertise in-house, and I am already understaffing my wildlife and wilderness content creation activities. Our traffic is growing towards 1 Gb per second (800Mbps actually), and we're looking at adding a second gigabit connection to the Internet. We are paying $10 per megabit per second, and I am confused about why your website says the prices close to three dollars per megabit per second. I look forward to hearing your advice.

Backwoods Ned.

A:
Ned - Thanks for giving me a holler. The “when does peering make sense?” question is the #1 question I get.

The price of transit varies widely in any particular Internet peering ecosystem. For example, in some parts of rural Texas, I am still hearing about transit prices closer to $60 per megabit per second. In these cases, the prices are high because there is little competition, and it is just expensive to get out into the rural communities. Historically, we have seen the prices drop significantly once competition enters a particular part of the ecosystem. For example, as soon as Level 3 expanded across Europe, the Internet service providers in those areas immediately dropped their prices below the point where Level 3 could recoup its investment. This is of course a very aggressive competitive situation, but it illustrates the potentially powerful effect on pricing when there is competition in the Internet peering ecosystem.

Given your situation I will also highlight a couple of things that make me NOT engage in peering. While the price of transit might appear to be high, the fact that you are only doing 800 Mb per second worth of traffic suggests to me that it may be more expensive to expand your network into an exchange point and try and peer your traffic away. Here is the math:

800 Mb per second times $10 per megabit per second equals $8000 per month.

This means, for peering to make sense, that you need to spend less than $8000 per month for a transport circuits, rack space, a gigabit peering port, and the router. If it costs $1000/mo for a gigE circuit, $1000/mo for a half rack, $1000/mo for the port and $1000/mo for the equipment, You can change the numbers of course, but in this circumstance, you will need $4000/mo in cost savings to cover the $4000/mo in the monthly costs of peering by the cost savings by peering for a while.

Shown another way, if you take that $4000/month in peering costs and peer a lot of traffic over that infrastructure, you see that the cost of peering goes down as shown in the table below. At 400Mbps you should be indifferent from a purely financial perspective between peering and buying that $10/Mbps transit.

The Cost of Peering Backwoods Ned’s Traffic

You have to be able to peer 400Mbps of your traffic away (50% of your traffic) using this infrastructure. I think you'll unlikely be able to do this anytime soon.

Given that networking isn’t your core competence, nor is it strategic to your content delivery, peering is probably not a good fit here.

I would, however, should suggest that you take a look at the market, and see if you can get that second Gb on a different Internet service provider. In this way you can multi-home your network and perhaps manage your traffic better across these two ISPs. Do this before getting into the next stage of complexity with peering.

Once you are comfortable with multi-homing and your traffic volume grows, you might consider buying your transit at a network dense colocation facility. The prices could be as much as 30% less and it would be the stepping stone to peering at a later date. We can do that math in a future column.

Best of luck Ned and Happy Holidays !

I hope this helps.

DrPeering@DrPeering.net

Expanding Peering Overseas

Tue, 13 Dec 2011 10:55:11 -0800

Q: DrPeering -
We need to expand our network into different parts of the world now. We are getting complaints of packet loss, high latency, and are not serving our customers in these regions very well.

How do I decide which regions to build into, and what help is available in making these decisions and scheduling that deployment?

Captain Steve Rogers
A:
Captain Rogers -

Thanks for that question. This is been coming up a lot lately, and I'll tell you what I have been advising folks.

First, the exchange point operators in the region are going to be very helpful in identifying which transport providers can get you into the exchange point. The better sites include a list of all the fiber providers and carriers that offer services into their Exchange Point, along with the e-mail addresses and phone numbers of the person you can contact to find out how much it will cost to build in and when they can deliver capacity.

So I would start with contacts at the exchange points in the regions of interest. While you're at it, ask them about colocation fees, peering port fees, time to install, the methods used for deployment, power and maybe even latency to various destinations. Leverage these free resources - the good exchange points are more than happy to help. The exchange points that don’t help much during pre-sales, will likely not help much more after you sign their contract.

As to the decision of which region to expand into, and which Internet Exchange Point to choose, the most common selection criteria are listed here.

I would probably concentrate first on the goals of the deployment. You said you are looking to decrease latency, and control routing to minimize packet loss. To this end, I would suggest that you take measurements of the latency from various parts of the world where you are (and locations that you might want to be) to the destination that you are trying to serve. You are trying to get a reasonable match between the cost of deployment options, and the latency improvement that you get by building out. For example, if you go to a place like Japan in order to service a market in India you will find it a fairly expensive deployment and latency will be about the same as if you deployed to lower cost Hong Kong location in order to serve traffic to India.

If you have time, attend some regional peering forums and IX member meetings if they allow you to attend as a member prospect. You can get really good ground truth from the people deployed in the area.

The other avenues. I would also have discussions with Internet service providers that serve the target market well already, and consider sending traffic through them to get to that destination. I'm not saying do not deploy into the new market, but it is definitely worthwhile having a conversation about the cost and the reach.

Remote Peering. Consider also the option of expanding your peering over a long transport circuit. There are pro’s and con’s here of course, but some people have had luck with this method.

This isn’t everything but I would say that these are a few of the high order bits surrounding this topic.

I hope this helps.

DrPeering@DrPeering.net

The Academic Operator Divide

Sat, 3 Dec 2011 09:58:29 -0800

Q: DrPeering -
In the heart of Gotham City, I attended a conference put on by leading academics discussing the economics of the Internet, focusing on the dynamics of peering. There were a few network operators (ISP, CDN, Content Provider, Search, etc.) and a handful of PhD candidates.

It was like they were talking about different Internets.

Gadzooks but these were completely different views of the world! And clashes. Pow! Bamm! Zop!

Which brings me to the riddle -

Why are there such clashes between these non-competitors?

Adam West
A:
Golly Gee Willigers you are right Adam!

There has long been this underlying inter-group conflict for as long as I have been in the industry. The operators sometimes say

“the IETF standards-making body is openly hostile to the network operators that attend.”

On the flip side, the researchers and hardware manufacturers continually have their motivations questioned when they submit talks at the operator group meetings like NANOG.

With inter-group conflicts you often see a phenomenon called “perceptual distortion.” Each group ascribes motivations and characteristics onto the other group that reinforces their own world view, and they unite with the others in their group behind that “common enemy.”

A way to bridge the gap is to pull the groups together with a strong facilitator and talk openly about these distortions, maybe laugh about these extreme and untrue views, validate some and invalidate others, and find areas of common interests and alignment. Working on specific projects together where there is alignment helps breakdown these perceived boundaries.

Let’s talk about these two groups and the circles they travel in.

The Internet Engineering Task Force (IETF) meets in different locations around the world several times a year. To participate you are advised to participate actively in the working group mailing lists, to read the RFC drafts being discussed/debated along with the supporting groups of RFCs. In some cases participants are asked not to participate in the live discussions at the meetings unless they have kept up with all of this institutional knowledge. While these meetings are “open” in the sense that anyone can pay and attend, the amount of work required to keep up and participate under these informal rules, and the cost of international travel required to attend sequential IETF meetings makes it very difficult for network operators without a significant research budget to attend. As a result, only hardware manufacturers, researchers, and in some cases academics with travel budgets drive the IETF.

When operators participate at IETF, their numbers are small, and their voices are drowned out by the vendors that espouse second-hand (i.e. “a lot of customers have asked for this”) points of view. As a result, the operators view IETF as “not worth attending”, promoting their agenda at IETF as “swimming upstream”, and

“The IETF is openly hostile to network operators.”

The network operators instead tend to participate at network operators group (*NOG) meetings instead, where their numbers are strong (~40% of the attendees are ISPs), the meetings are local (relatively speaking) where they can compare operational issues like what works and what doesn’t. To do this they need a group meeting with a social environment conducive to identifying and comparing notes with their peers from other companies.

The conference submissions from network vendors are severely scrutinized and viewed with great suspicion. “What are they trying to promote here...what is their agenda?” is the common perspective in the operator community here and presentations are often dismissed based on the perception of spin rather than on the chance that someone might learn something from the content. As a result, the vendors have learned that the way to get on the agenda at operator meetings is to have a customer network operator co-submit or submit their experience with the technology.

Researchers are not viewed with suspicion because no one in the operator community really cares about “researchy” topics. So, they put the researchers into the “research track” opposite of the peering track, thus guaranteeing that the researchers don’t get any input from operators that they came to get input from!

Each group has their own private invitation-only conferences as well. Both groups focus on getting the right and manageable number of attendees in the room. Sometimes this is done with a requirement that the attendee be a customer of one of the sponsors, or a government-sponsored lead researcher. These private meetings have the side effect of reinforcing the boundaries between these groups. After all, if these “others” would be allowed to attend they would no doubt ruin the event in some way.

These are manifestations of this perceptual distortion, reflective of the underlying inter-group conflict.

Academics vs. Operators

Let’s look at the divide between the academics and operators. (These views are my own and no doubt reflect my own personal perceptual distortion.)

1) Models vs. Practice.

The academics look at the Internet as a massively important infrastructure that could spin off tons of data that could be processed using their favorite advanced mathematical and modeling techniques. They want the data so they can do their work, identify something interesting that has been overlooked, and publish the results. The more counter-intuitive, the more important, the more interesting the results, the more quoted and read they become. Academic rock stars are made this way.

However, the data they seek is not typically been made publicly available, so in the absence of publicly available data they make simplifying assumptions about how the Internet works, and make models based on those assumptions.

The operators on the other hand are not looking for simplified models or interesting anomalies as much as trying to keep their network working, customers happy, and their business profitable or at least afloat. Fundamentally they are focused on how the flawed equipment that they put into their production network actually works when connected to the real Internet.

The operators are surprised and dismayed at the gap between their understanding and the academic understanding. It is enough to dismiss a speaker with great haste and there is not much tact displayed when doing so.

1) Math vs. Money
Academics are not rich. They are not making a lot of money from their research in general and do not have the budget to build test beds of the scale of the Internet. Therefore, they count on their R&E networks for data, or more commonly, on simulations that are based on their hypothetical models of how the Internet works.

Without the raw data, they are forced into the world of theory. I have rarely seen an academic presentation based on truly relevant and interesting data culled from the public Internet. They are more typically simulation models, greek letters and graphs to prove importance and correctness.

The ISPs however keep their data proprietary and have since the NSFNET days. The data could (and will) be used by competitors to prove to prospects why they are better than the other’s own pronouncements. Sharing the actual data may materially harm the stakeholders, so the data is not made readily available. Even anonymized data that mentioned winners and losers made one researcher very unpopular with those that contributed anonymous data. So the risk is high and the payoff is low for sharing data, and it is all about the money.

1) Theoretical optimization vs. stepwise refinement
Academics appear to be looking to apply math to identify things that are interesting and ideally lead to something akin to a mathematical proof of optimality.

Operators on the other hand appear to be engaged in the stepwise refinement of applied heuristics that make incremental improvements in the current operational context.

Different motivations.

1) Cultural differences.
The operators are a bit more “rough and tumble” with their discussion styles than the academics. They may interrupt others more often, listen less well, perhaps speak louder and more often than researchers. This may be the group culture stemming from the years of working in a crowded team environment filled with uncertainty and the need to identify those few things that are solid truths. Those that assert most loudly, and rightly, take increasing responsibility and rise through the engineering organization. They tend to be a bit aggressive.

1) Network Operators don’t need academics, but academics need network operators.
A number of academic papers published that describe models and make assertions that are pretty far removed from reality according to some in the network operations community. This reflects the desire for academics to publish without hard data and the cooperation of network operators, competing with the interests of the network operators to protect their information. As a result, the research done by academics is thought to be “irrelevant” and generally not needed by the industry. From this perspective, the academics need the network operators more than the network operators need the academics.

These cultural clashes have gone on for years and the bridge needs to be built. There are a lot of interesting projects that one group can do that would benefit the other if they can get over these differences.

DrPeering@DrPeering.net

What is an Internet Peering Ecosystem?

Tue, 22 Nov 2011 18:56:43 -0800

Q: DrPeering -
I read in your book that you model the Internet in each country as an autonomous “Internet Peering Ecosystem.” I am not so sure that is an apt analogy. In ecosystems you have predators and a prey that interact as a system, a cycle of life. I don’t see such a thing in the Internet Peering ecosystem.

Ahogo Dahadat
A:
Thanks for reading the book! And yes, I model each Internet Region (usually bounded by country boundaries) as an independent ecosystem, each with its own critters operating its own habitat. Each habitat is a little bit different, with different:
- Tier 1 ISPs
- Tier 2 ISPs
- Internet Exchange Points
- transit prices
- transport prices
- etc.
Each ecosystem has its own peering community, with their own quirks and preferences and obstacles to peering. In every case, the cost of local loops is different leading to a different break-even analysis. So systematically, they are each independent of one another.

It is also true that these players do not exactly eat each other, but similar players do hunt (compete) for resources, get consumed (acquired), go extinct (bankrupt), and excrete old resources (network gear and systems) <fill in your own joke here.>, etc. The key is the mutual interdependence in the ecosystem, the interactions and positional power as analogous to the activities and place of the participants in the traditional ecology notion of ecosystems.

Q: DrPeering -
Do ISPs ever buy transit from others in one ecosystem and peer with those same parties in another ecosystem?

Mr. Bun E. Carlos
A:
Thanks for the question. I documented this in The Internet Peering Playbook as Tactic 4. Bundle Internet Transit with Peering - some ISP peering coordinators considered it a cheap trick and more trouble than it was worth.

Tactic 4. Bundle Internet Transit with Peering

The beauty of this approach is that the internal advocate handles all internal hurdles to peering, the peering can be executed immediately, and using separate interface cards makes it easy to implement.

The more common way I see this is when an ISP splits their network into a couple of independent Autonomous Systems (ASes) and they peer them independently - an AS for Asia, one for the US and one for Europe for example. Then the peering decisions are independent and non-transitive, so peering in Asia isn’t giving away your network routes in the U.S. for example.

These two questions are related in a way - they bring up a possible emerging trend. Some of my consulting with clients is highlighting that Asia itself may becoming a singular “Internet Region” - with its own “Asia Tier 1 club.” If true then this is new and interesting, because no single government can apply pressure to this group to get them to peer more openly, or document their peering practices, etc. as governments have when all Tier 1 ISPs interconnect within the confines of a single country.

This is an emerging trend that we should keep an eye on!

DrPeering@DrPeering.net

A Clever Transit Tactic

Tue, 18 Oct 2011 16:37:18 -0700

Q: DrPeering -
I have a new transit tactic for your Internet Peering Playbook. It is similar to “Tactic 5 - Play the Market” and “Tactic 4 Renegotiate multi-year contracts.”
First, understand that we have 20-30Gbps of content to ship off to transit providers.
We could of course lock in a good price from a couple of transit providers, but that would lock us in to the metered price for the duration of the contract. Instead, we purchase transit from a field of transit providers, committing only to 1Gbps each on a 10Gbps circuit.
To see how this works, assume that the market price for transit is $2.50/Mbps and that each ISP has a price for traffic that bursts over the commit (i.e. it is not capped).
At the beginning of the month, we ask each ISP for their best burst price so we could decide who would receive the revenue for the remaining 13 Gbps. The answers come back as shown below between $2.00/Mbps and $2.70/Mbps.

For the rest of the month, care is taken to meet the 1 Gbps commitments while sending the bulk of the remaining traffic only to those with low burst prices as shown below.

At the end of the month, the ISPs that did not receive any burst traffic are given the chance to lower their burst price and receive more traffic and revenue next month.

In this way, we are able to keep the (burst) prices of transit dropping for the duration of the contract.

The other thing that we do is insist on only 1-year or 18-month terms. We have been very effective in this market at turning the thumb screws on the transit providers in the market. Keeping the duration short gives us more opportunities to negotiate the 1Gbps commit price.

This maneuver has worked well for us and would complement the other tactics in the book. Thanks for your consideration -

-- @aDbCc

A:
Thanks for this. I am adding it to the 2nd edition of the playbook. If you are willing to be cited as the contributor, please let me know.

DrPeering@DrPeering.net

A Peering Success Story

Fri, 16 Sep 2011 10:10:02 -0700

Q: DrPeering -
Is there any international precedence for Tier 1 ISPs peering with others? Either forcefully, through regulatory pressure, or otherwise?
Epony Radstein
A:
Generally speaking, there is little incentive for a Tier 1 ISP to peer with anyone else. But I have another success story about an emerging leader ISP that was able to obtain peering with the largest Tier 1 ISP in the Internet Region.

For those of you new to this blog, a “Teir 1 ISP” is an ISP that access to the entire Internet Region (typically bound by a country boundray) solely through their free and reciprocal peering relationships.

By definition, the Tier 1 ISPs already hear all routes in the country for free. I hear them say “We have all the peering that we need” and they are correct.

So on to the story.

The best way to describe this is to review Tactic 21. Overseas Power Play from The Internet Peering Playbook.

In this maneuver, ISP A wants to pressure the incumbent Tier 1 ISP B to peer by forcing traffic along a more circuitous and expensive international path. The Tier 1 ISP refuses to peer, so ISP A turns off the in-country transit connection, and builds into a well-populated foreign IXP so that it can send its traffic through an international upstream to reach the incumbent Tier 1 ISP.

ISP A has to pay for the international circuit. The incumbent Tier 1 ISP also has to pay for its international circuit into the overseas IXP where it peers. The customers of both ISP A and the Tier 1 incumbent ISP both experience poor performance. The hope is that the Tier 1 ISP will come to the table and say, “We should peer. It is costing us both a lot of money to peer in this other region.”

MWEB has taken this approach in South Africa:
“We have made a conscious decision that starting next month [November] we will not pay anybody for transit traffic anymore. So if you don’t want to peer with us, that is it! We will not pay you one single cent anymore.”
– MWEB CEO Rudi Jansen

Just like the Traffic Manipulation tactic, this play counts on the pricing and perhaps the performance problems to pressure the target to peer locally to improve performance and decrease its international transit costs. In this example not only will Telekom South Africa customers receive their MWEB video traffic from London at great expense, but the latency-sensitive gaming traffic will also experience really poor performance (Figure 11-36).

Figure 11-36 Peer overseas.

Several ISPs have used this tactic to get cheap transit, to force traffic along a circuitous path to pressure the local incumbent to peer, and to decrease the overall cost of traffic by taking care of its own international transit.

Notes from the field.
Pressure in South Africa
In 2010, MWEB offered a unique uncapped Internet Transit service to the South African market. The market responded enthusiastically, and by 2011 had acquired massive market share. Even with this tactic, MWEB has not obtained peering with the incumbent as of this writing, but it has experienced massive growth and better performance from the overseas transit and peering deployment for in- and out-of-country traffic. The amount of traffic between MWEB and the incumbent is now a relatively small percentage of MWEB’s total traffic.

The Peering Success Story.... As of September 2011, MWEB is peering with Telekom South Africa. This is according to some private sources (second hand, rumors, bar gossip) and public sources (including http://mybroadband.co.za/news/broadband/33234-telkom-and-mweb-peering-at-last.html), but as usual, the specifics of the arrangement are officially protected behind an NDA.

Congratulations MWEB - you are making history.

DrPeering@DrPeering.net

Access Power Peering

Tue, 6 Sep 2011 14:47:33 -0700

Q: DrPeering -
How has Internet Peering changed over the past couple of years and what are the implications - is there a trend us investors should be aware of?

Alice Nelson
Bunch of ASes
A:
Thank you for that question, because this is one of the most interesting peering interplays the Internet Peering Ecosystem has seen. It is one of my favorite discussions as it brings out and exercises the industry terminology, it demonstrates the power positions in the ecosystem, and allows us to see the gamesmanship of the players involved in the open market for Internet services.

This story also highlights a fundamental shift in power going on right now, something I document in The Internet Peering Playbook. I describe a handful of evolutions that all have led up to the current shift in the power dynamics of the Internet Peering Ecosystem. I will skip all of the history and answer your question by saying that the Access Networks are emerging as the new power players. I call this emerging new maneuver “Access Power Peering”.

<The following is clipped from chapter 10 of The Internet Peering Playbook: Connecting to the Core of the Internet. See the whole chapter here.>

Evolution #6 – Access Power Peering

The Internet Peering Ecosystem in the U.S. appears to be morphing again, this time positioning the access networks into a much stronger power position.

(Editor’s note: We are naming Comcast explicitly when illustrating this evolution. There are other examples of this maneuver, but there is far more data available about the Comcast-Level 3 dispute. By walking through this story we can more succinctly and concretely demonstrate Access Power Peering.)

Video Dominates the Internet. In 2010, at least 40-50% of all Internet traffic today is video, according to conversations with ISPs. Separately, the movie industry people emphasized that video is unlike other applications. They said that disruptions (pixilation, freeze frames, audio garbling, etc. called “artifacts” in industry jargon) in a movie video stream cause a break in the “suspension of disbelief.” This effectively destroys a video watching experience for the end user. Video traffic dominating the Internet and the requirement for flawless video, taken together, highlight the importance of an optimized Internet Peering Ecosystem. Content distributers need to be able to deliver increasing volumes of video traffic with very high quality.

At the same time, significant change occurred in the U.S. Internet Peering Ecosystem.

Comcast Peering Policy

Comcast put in place a Peering Policy stating that they require (among other things) that traffic volumes exchanged be roughly balanced. Traffic in the inbound direction (to Comcast customers) had to be (roughly) balanced with the outbound direction (from Comcast customers). Some in the industry have said that the specific not-to-exceed-ratio is around 2:1. This peering ratio prerequisite and the ratio value is somewhat common in the industry.

However, consider that

1. Internet video is or soon will be the dominant Internet application, and

1. Internet video tends to be massively asymmetric (as high as 30:1), and

1. Comcast customers consist of tens of millions of eyeballs primarily pulling down content from the Internet.

Very few peering candidates have or will ever have a balanced ratio to offer to access network. Video content delivery companies such as Content Delivery Networks (CDNs) and content-heavy ISPs certainly won’t have balanced peering ratios with access networks. These companies are in the business of getting video objects to the eyeballs and have very little traffic to pull from Comcast. Since a large portion of Internet traffic is destined to be video traffic asymmetrically delivered to the access networks, this Peering Policy effectively ensures that this traffic can not be freely peered.

Comcast Paid Peering

Comcast started offering a Paid Peering service for those that did not meet their peering prerequisites. They priced the service at about the market price for Internet Transit. Companies with content to send to the tens of millions of Comcast customers could send their traffic through their upstream ISP, or they could send it directly to Comcast for about the same price. At this price point, it is a good value proposition and several CDNs have gone down this path and purchased Paid Peering from Comcast. Let’s look at the affect on the dynamics in the ecosystem.

There is only one way to get to the Comcast customers – all traffic must directly or indirectly make its way through the Comcast network to reach the Comcast customers. Some in the industry call these customers “captive” customers since there is no alternative path to reach them. This captive customer base is the source of potentially great positional power in the Internet Peering Ecosystem as shown in Figure 10-7. With this Access Power Peering, all traffic destined for access network customers (and all corresponding Paid Peering revenue) flows directly to the access networks.

Figure10-7. The Access Power Peering paradigm.

We will see some more Access Power Peering dynamics with the Comcast-Level 3-Netflix dispute discussed next.

Case Study: Comcast-Akamai-Limelight and Paid Peering

In late 2010, Comcast was purchasing Internet Transit from several ISPs as shown in Figure 10-8.

Akamai was purchasing Paid Peering from Comcast and enjoying low-latency high-capacity access to Comcast customers. Limelight Networks, a competitor to Akamai had a choice to make. Should it continue to send its traffic through its upstream ISP to reach Comcast customers? By doing so, Limelight traffic will suffer higher latency and potentially greater packet loss than its competitor. Philosophically, Limelight feels that it shouldn’t have to pay Comcast to deliver the content that Comcast customers requested!

Figure 10-8. Alternative to peering is a more circuitous path.

Limelight’s hand was forced however when one of Comcast‘s upstream networks experienced chronic congestion. When this occurred, Limelight content traffic destined to Comcast customers experienced performance problems (packet loss). Since Limelight is being paid by the content providers to distribute the content to the edge for a better quality end-user experience, its customers began to complain. The cause of the problem seemed to be the links between Comcast and Comcast’s upstream transit provider. Limelight’s upstream transit provider is powerless to fix this problem. As a result of this context, Limelight has to purchase Paid Peering from Comcast (Figure 10-9).

Figure 10-9. The more direct Paid Peering path to the eyeballs.

From a practical perspective, if you are paid to deliver video to Comcast customers, there is no choice but to purchase Paid Peering from Comcast. This is a potentially huge shift in power in the ecosystem. This power is demonstrated in the Netflix, Comcast and Level 3 story next.

The Netflix , Comcast and Level 3 Story

Setting the stage. Continuing this example, the relationship between Level 3 and Comcast is a bit more complex than simple transit. According to the ISPs familiar with the situation, Comcast entered into a broad business relationship that included as a condition of the agreement “free on-net” access. This is a well-known tactic discussed further in the ISP Peering Playbook chapter. With this “free on-net” or peering relationship, Comcast got free access to Level 3 customers (which happened to include most of the other cable companies) and Level 3 in return got free access to Comcast customers. This was shown in previous Figures 10-8 and 10-9 as “T+P.” The point is, this broader business relationship enabled Comcast to have settlement-free peering with Level 3, a Tier 1 ISP in the U.S. region.

Act I. Netflix, a large scale video distribution company had gained considerable traction with its video delivery over the Internet and used CDN services from both Akamai and Limelight. Netflix was experiencing a near exponential growth in traffic. As shown in Figure 10-9, most of the ‘$’ symbols are on Comcast‘s side, so as Netflix traffic grew, so did the Paid Peering fees paid to Comcast.

Act II. Also in late 2010, Level 3 bid for and won the Netflix video delivery business, undercutting Akamai. As a result, Netflix traffic moves from Akamai to the lower-priced CDN that Level 3 offers (Figure 10-10).

Figure 10-10. The shift of video traffic from a paid peer to a settlement-free peer.

Act III. Level 3, since they have a “peering” relationship with Comcast, informs Comcast that there will need to be some additional interconnect capacity. This is a common thing that peers do (Figure 10-11).

Figure 10-11. Additional ports requested from Access Network.

At this point, who wins and who loses? Netflix wins a lower cost for video distribution. Akamai obviously loses the Netflix business, so the revenue from Netflix to Akamai decreases. Comcast also loses a corresponding amount of Paid Peering traffic and revenue. Level 3 wins the business and the additional revenue from Netflix. But Comcast loses again, because not only is it is giving up the Paid Peering revenue from Akamai, it is also being asked to spend money on additional peering ports to handle that same Netflix traffic over their free peering relationship with Level 3.

Act IV. In response to Level 3‘s request, Comcast refuses to add interconnect capacity and points out that Level 3 is “out of ratio” and needs to pay Comcast for Paid Peering. They argue that it wouldn’t be fair not to charge them since they charge the other CDN partners for Paid Peering. It would be an unfair competitive advantage if Level 3 (which offers a competing CDN service) didn’t pay as well.

This dispute was very public and much like my airplane story in the beginning of the book, there are opinions across the spectrum as to who is good and who is evil.

To complete the story, Level 3 acquiesces and pays Comcast for Paid Peering as shown in Figure 10-12.

Figure 10-12. Level 3 acquiesces and becomes a Paid Peering customer.

This result further strengthens Comcast‘s power peering position, since now even a Tier 1 ISP is paying for Paid Peering with Comcast to access its eyeballs. As stated in the beginning of this section, this Comcast story is merely a good demonstration of Access Power Peering evolution, and by no means the only example.

The Problem with Access Power Peering

[ I will step up onto the soapbox for a moment here. ]

In this chapter and the previous chapters, we have seen that the U.S. Internet Peering Ecosystem has morphed. The players and the relationships, the power positions and behaviors evolved with the context.

First, the Tier 1 ISPs were in charge and received a piece of the traffic that directly (from a customer) or indirectly (from a customer of a downstream customer) traversed the Internet region. In this case, the Tier 2 ISPs could peer their traffic between themselves to bypass their transit provider. The Tier 2 ISP could also select a different upstream ISP if they thought they could get a better price. There were choices, and both peering and buying transit in an open market led to cost and performance efficiencies. The Internet Peering Ecosystem evolved into a more efficient fat middle.

The Access Power Peering scenario however is problematic in that it leverages the lack of alternative paths to the eyeballs.

Since 80% of the Internet traffic is destined to be video, a large and massively asymmetric stream, the peering ratios clauses will prevent settlement-free peering from happening for 80% of the Internet traffic. Anyone with video content to send to the eyeballs will not qualify for free peering.

So you might say that there are alternatives to free peering: Internet Transit and Paid Peering. However, since 80% of the Internet traffic will be video, video delivery companies must get the content close to the eyeballs. The Internet Transit path is inferior and the video delivery competitors all pay for peering to get direct access to the eyeballs. If you are delivering video, from a practical perspective, there is no choice but to buy paid peering from the access network.

So then you might ask “What is the problem? Eyeballs have a choice - if they don’t like the performance of cable they can move to DSL and vice versa.” Performance would have to be very bad for all access customers to switch. During this time, the content companies will get complaints from the access customers, and the CDNs will hear those complaints as well. It is easier for the few content companies to purchase paid peering or purchase from a CDN that purchases paid peering than for the access customers to all pick up and move to a competitor. The context is such that the video delivery companies will give in first and pay the access networks. The problem is that all traffic and revenue in this model leads to the access networks, solely because they have the power position based on a captive customer base.

In the earlier example, Comcast charged market transit prices for Paid Peering. But what prevents them from charging a higher price? What is the alternative for CDNs whose business is to get the web objects as topologically close to the eyeballs as possible?

Is the next Internet Peering Ecosystem model really that everyone pays the access networks to deliver the 80% video Internet traffic?

From this we can recognize its position in the Internet Peering Ecosystem, identify its motivations and even predict its behavior. At this point you should have a good idea of the players, the positions, the corresponding motivations and behaviors observed within the Internet Region.

In any case, I find this development fascinating. Comcast has effectively leveraged their position and made a transit provider pay for access to their eyeballs. Since Comcast is a leader in the cable company space, I wonder if we will see the other access networks exercising their power position in a similar fashion. As always, the peering ecosystem continues to morph and adjust to the stimuli applied, much like a living organism.

DrPeering@DrPeering.net

The Value of an IXP

Thu, 18 Aug 2011 13:44:28 -0700

Q: DrPeering -
How can one estimate the financial value of a colocation center or an Internet Exchange?

Jamal Horga'hn
A:

The value of the IXP (to the population) is proportional to the number of participants (the how many), the routes available at the IXP (the who there), and the volume of traffic peered (the how much), as shown mathematically as follows:

Value of the IXP = ValueDerived - CostToParticipate

Value of the IXP = f(p, r, v, m) - c

where on the value side:
p = the number of participants at the IXP—the "How many?"
r = the number and uniqueness of routes available at the IXP—the "Who there?"
v = the volume of traffic that is peered at the IXP—the "How much?"
m = the match of the service to the market needs and stickiness—the “Why are they there?”

and on the cost side:

c = the cost of participation at the IXP (colocation, switch ports, equipment, cross-connect, membership fees, etc.)

All of the IXP Playbook tactics in The Internet Peering Playbook: Connecting to the Core of the Internet manipulate one or more of these parameters.

Those facilities that have a positive value are past the critical mass point and generally have a provable value proposition to offer to their clients. Beyond this point they have the ability to justify increased pricing.

The calculus and various tactics are a collection of the researched manipulations applied by IXPs around the world to build, grow and maintain critical mass.

In later articles I will go through some of the manipulations to build critical mass at an IXP:

Tactic 1. Bluff the Size of the Population

Tactic 2. Build a Network Umbilical for Later IXP Migration

Tactic 3. The Group Buy-In

Tactic 4. Buy-In Key Players

Tactic 5. Divide and Conquer

Tactic 6. Beachhead Verticals and Niche Markets

Tactic 7. Extend the Dominant IXP.

Tactic 8. Prevent Rogue IXPs.

Tactic 9. Swim with the Fishes

Tactic 10. Bundling

Tactic 11. Build and Maintain Population Stickiness

Tactic 12. Strengthen Peering Population

Tactic 13. Demonstrate Leadership

Tactic 14. Purchase Data Center During Downturns

Tactic 15. Drop Peering Cost

Tactic 16. Price on Value

Each of these gets its power by manipulating one of the variables in this equation.

DrPeering@DrPeering.net

Digging Peering in Africa

Tue, 16 Aug 2011 15:48:11 -0700

Q: DrPeering -
Fiber is being deployed across some countries in Africa very rapidly. Is Africa an emerging market for peering? What are the obstacles and how long with it take for Africa to catch up to the West?
A:
The African continent is a new frontier for Internet Peering. Here is why.

Emerging Peering Forum. Karen Rose (ISOC) and her team (Michuki Mwangi, Chris Morris, and many others) organized their second terrific peering forum last week called the African Peering and Interconnection Forum (AfPIF-2) in Accra, Ghana. As many of you know, it takes a few peering forums before they become a must-attend event, and these are well on their way to becoming a continental peering forum. And this continent has huge potential.

Massive underdeveloped Internet Region. Consider the graphic below (thanks Karen Rose and informationisbeutiful.net) showing that the continent of Africa is absolutely massive. It is also underdeveloped from an Internet penetration perspective. Of the 1 billion people on this continent, less than 20% have Internet access. Hence the word potential.

Inexpensive Peering Infrastructure. Many of us are used to hearing about the bleeding edge of technology, but there are some good aspects to being on the trailing edge. Many countries are having some early success in peering reusing free hand-me-down or inexpensive second hand switches. The deployed technology is sound - their peering switches are very well tested in battle hardened environments. All of this leads to lower peering costs and speedier arrival critical mass.

Risk-Seeking Technology Agnostic Population. Kenya has leap frogged the west. With ePesa, people buy and sell goods on the street using their cell phone. A merchant selling a bag of potatoes can set the price for the passerby who sends the money using his cell phone. Her phone alerts her of the money transfer and she hands over the potatoes. The same cell phone is then used for paying her bills. Pretty cool, and driven by the poor state of the physical banking infrastructure in the country (long lines, short hours, charing for deposits and withdrawals, etc.). Cell phone commerce is relatively frictionless.

So the population is not technophobic and used to leveraging mobile technology.

Internet Transit is Expensive. The price of Internet Transit across Africa is so high (~$350/Mbps) that the business case for peering is pretty easy to make. However, the amount of locally in-country peerable traffic is pretty low. Most of the content that the eyeballs want is originated in the U.S. or Europe. As a result, the amount of traffic peered across an African IX is in the Mbps or down in South Africa in the Gbps. It will be some time before they wrestle with the switching trunk issues of the European IXes that peer in the Tbps.

Needs Local Content. The emerging peering community is using any means necessary to bring down content providers to locally host their content in Africa. In many cases, this can be accomplished by luring the CDNs down, as they deliver so much of the content desired by this population. This is taking longer than it should, as the content providers generally will do anything they can to make sure that all eyeballs have a positive experience. They in turn can apply pressure on their CDNs to build into and throughout Africa.

Needs Knowledge Workers. One thing that is missing is education and peering evangelism to bring these communities together. Early switch deployments across Africa countries was not sufficient to build a peering population. There needs to be leadership to pull the various ISPs and content providers together into a peering point, and then evangelize that peering point to others. That is why these peering forums are so valuable and why I contributed my time and some content to the forum including the Peering Simulation Game and various stories and excerpts from The Internet Peering Playbook.

Friendly Group of Folks. I will admit that I rather enjoyed the book signing line (shown below). I had never written a book so had never done one before and it was very gratifying to get the content into the hands of people who really wanted the material. I found the populations in the peering communities to be social, friendly, and passionate about peering. These are of course necessary for emerging peering populations to advance the continental peering initiatives.

Government interest and support for peering. I know, governmental involvement or interest is a negative to many of you. And to me as well, but there are many millions earmarked for encouraging more peering across the continent. The people I met were as passionate about learning about peering, and seem to want to encourage the right things. Money may provide the lubrication needed to motivate peering among some of the larger providers across the continent as done during the NSFNET transition period in the U.S.

One African Peering Challenge: Digging up Fiber

At this conference we spoke a lot about peering being one of the answers to the problem of poor Internet access across the continent. Others issues Africa face are the lack of cheap high capacity fiber and power infrastructure.

Quite a bit of fiber has been laid across some of the countries in Africa.

Some of the fiber is the victim of sabotage from competitors. But that is a small percentage of the total fiber ripped out of the ground.

Some of this fiber is ripped out of the ground in Kenya by the poor that are used to digging up the copper wire from the ground and making jewelry to sell by the side of the road. There are many stories of pick axes turned into molten metal after hitting the live power lines. The fiber companies then put up signs indicating that this is a fiber run, not a copper run, and they will get no value from digging up the fiber. But many of these people can not read, so the signs do no good.

In some areas the fiber companies lay the fiber with a Kevlar sheath to protect the fiber. These bundles are cut and attached to the tow bar of trucks and yanked out of the ground for the Kevlar, which the poor weave into black market bullet proof vests.

In response, the fiber providers pour cement over the trenched fiber in high loss areas. A very time-intensive and expensive process.

In some parts of Africa, there are sections where the “Area Boys” harass the fiber layers until they get paid off. It was portrayed as a kind of young man’s mafia.

The other issues raised at the conference are common across other parts of the world.

Top 10 Challenges for the African Internet (as heard at the AfPIF-2 conference)

1) Dominance by incumbents - lots of anti-competitive behaviors
2) limited IXPs - many assumed the field of dreams approach
3) absence of tariff/price controls to essential facilities - too expensive
4) lack of market awareness - need education
5) low investment in transport & energy infrastructure - stable power and facilities
6) difficulty in acquiring rights of way - challenges outlined above
7) low economies of scale - volume makes peering make sense
8) country conflicts - wars, legacy bad blood delay deployments
9) challenges with cross border circuits - a perennial favorite here
10) limited local content - most content comes from Europe and U.S.

Top 10 Opportunities
(as heard at the AfPIF conference)

1) Nolliwood - believe it or not, Nigeria actually produces a ton of really good video content. The Soap Operas in particular are quite addicting.
2) SECOM has available massive fiber between countries via the ocean.
3) GDP growth - leads the world here.
4) 1 Billion people (2B eyeballs).
5) More middle income people than India.
6) Cellphone Internet
7) It has 4 times larger population than developed economies
8) Growth potential - 21% penetration vs. 75% in the west.
9) MWEB and others uncapping and starting disruptive and enabling pricing.
10) RTT can go from 800ms down to 200 ms migrating from satellite to terrestrial and transoceanic.

To me, the best argument for peering is that there are over a billion people in Africa - lots of eyeballs that are experiencing sub-standard Internet access. In South Africa companies like MWEB are pushing uncapped Internet access and people are using it. There appears to be a latent market demand here. An emerging rich African infrastructure will enable high-speed access to a new large population using primarily mobile Internet technologies. This will no doubt take some years, but can you think of a more exciting new frontier?

DrPeering@DrPeering.net
[Thanks to Karen Rose for the invitation to speak and release my book at the AfPIF-2 meeting, and thanks to Scott Landman for edits to this article. ]

Peer During Time of Crisis

Tue, 2 Aug 2011 21:09:17 -0700

Q: DrPeering -
Does it make sense to help out competitors in a time of crisis?
Brian Jones
A:

During the fiber outage in the Taiwan Straits a few years back. several large ISPs lost their primary connectivity and saturated their alternative paths to the rest of the Internet. The result was seriously poor performance for their customers. We are talking really bad performance. Customers could not check out of hotels because their credit cards weren’t going through because of the congestion.

During this crisis, ISPs that did not meet the peering prerequisites stepped up and offered temporary free peering to help out during the crisis. And sure enough, after the crisis was over, the peering remained. Helping out a competitor in the time of crisis actually led to sustained peering. And this story was repeated several times, enough so that it will be added as a tactic in the next edition of the Internet Peering Playbook.

So the answer is that it is in your best interest to offer assistance to a competitor in a time of crisis, particularly one that you would like to peer with one day.

DrPeering@DrPeering.net

The Next Generation Internet Exchange Point

Wed, 20 Jul 2011 09:31:43 -0700

Q: DrPeering -
The dominant exchange points have a large population of peers. And if we be just a startup IX, the competitive situation would look mighty bleak. Is there anything that will displace the big guys?
Mal Reynolds
A:

Many of these IXPs have grown well past critical mass and therefore provide great value to their customers and offer great value to the marketplace. Startup IXPs will have a tough time convincing prospects that they, too, have value. However, some startup IXPs recognize that their strength is their agility. They will realize that they can frame a superior value proposition to customers, they can satisfy some niche, they will be more entrenched in the community, and they will be able to execute when the large organizations have meetings to discuss a response to the changing market. The large IXPs service so many countries and have so many staff that they will be slower to adjust, and the smaller startups will capture the opportunity to build the next-generation IXP.

The next-generation Internet Exchange Points will emerge in the next 5 to 10 years. These new facilities have three drivers. First, the current generation of interconnect facilities were built over a decade ago, with a power density (1.75KVA per rack) appropriate for the era. These facilities are simply unable to provide the power for the next generation of equipment that will handle the emerging video traffic. Internet traffic has traditionally grown at 35% per year or so, and the video traffic is expected to be 80% of all Internet traffic by 2013. This growth will require the ability to interconnect and the ability to stream video at terabit-per-second speeds. The equipment to provide these speeds will likely require closer to 20KVA per rack, and the older facilities are unlikely to be able to provide that environment for the massive volumes of video distributors and access networks at the core.

The next generation of facilities will likely follow the same trajectories as in the past.

The Tier 1 ISPs will not be able to keep up in the current environments; they will get together and put out a request for proposal to all of the major IXPs and colocation companies, and then they will select a center in each of the interconnection regions across the country. These facilities will be 20KVA facilities, near the existing colocation centers, and the Tier 1 ISPs will extend their network there. The CDNs and access networks will follow them there, and shortly thereafter the place for Tier 2 interconnection and content hosting will be at these new facilities.

I would put the chances of a next generation Internet exchange emerging at 80% likely within the next 3-5 years. It will of course take some time to emerge, but all signs indicate that the pain points will be there to drive it forward.The gravitational pull into these new facilities will be strong.

DrPeering@DrPeering.net
[ Editor’s note: This article was extracted from The Internet Peering Playbook: Connecting to the Core of the Internet. Reprinted with permission. ]

The Network Umbilical

Thu, 16 Jun 2011 16:00:15 -0700

Q: DrPeering -
The data center market is commoditized and I hear it characterized as trench warfare... Data centers are aggressively competing for racks in the $400-$500/rack range.
Gern Digawit Blanston
A:
The data center market is commoditized unless you are a colocation provider and your population is beyond critical mass, or has some other attraction. The market price for commodity data center space is about $500/rack instead of $1500/rack for premium space at a well populated colocation center. This has led to a tactic I call the network umbilical pitch.

Let's assume a target colocation customer interconnects with network service providers and hosts servers at the premium colocation center as shown in the top part of figure 13-9.

Figure 13-9 Network Umbilical Pitch
In this tactic, an upstart colocation provider builds critical mass by starting as a data center, but positions itself as a network umbilical (colocation extension) site. This data center company pitches its data center as a cheaper place to host the larger server deployments and edge routers. The customer keeps its core routers at the premium network dense colocation site, but deploys the rest to the discount network umbilical site as shown in the bottom half of the diagram. The fiber between the two data centers is paid for by the cost savings of hosting the large number of servers at the cheap site.
Once the network umbilical site has enough participants, it has a roster of impressive peers that can help it make the pitch as a peering point. While some companies may not be able to peer at the extension site, some will. The roster of participants is impressive in any case and helps build credibility when establishing a new IX.

These were excerpts from The Internet Peering Playbook.

DrPeering@DrPeering.net

The Internet Transit Playbook

Thu, 19 May 2011 11:05:48 -0700

Q: DrPeering -
Any tricks to purchasing transit?
Kikki Bridges
A:
Kikki - yes! I have about a dozen of these transit purchase tactics gleaned from the field documented in the upcoming (delayed) book. Here are three of the standout tactics.
Note Well: None of the tactics in the book or presented here are recommended tactics. They are shared because they exist in the field and you should understand them.

Transit Tactic 11 - Short Term Transit
In this approach the ISP will simply negotiate for a one-month term with little or no traffic volume commitments.

With this tactic the customer keeps ratcheting down the negotiated price every month, loosely tracking the dropping market prices for transit. In this way there is little chance that one will be stuck with a contracted transit price that is significantly greatly than the market price for Internet Transit.

Figure 11. Purchase with little of no commits for short duration contracts.
Some of the very large content companies shared that they are able to go beyond this. They have negotiated no-commit month-to-month pricing. They have the luxury of deciding where to send their traffic without the burden of watching their traffic volumes against their commits.

Transit Tactic 5 – Play the market
In this tactic, the ISP plays one transit provider off the other to get the lowest price, but taken to an unethical extreme.
One ISP we spoke with said that the verbal agreements and handshake deals would be the starting points for the process of playing each ISP off of each other. In some cases, parallel negotiations were in play all the way to the physical signing of the contract. The price points, even protected under NDA, might be used or bluffed during this process to negotiate alternative terms with other providers. In some cases, even after signing, the price negotiations continued as alternative offers from the parallel negotiations chimed in.

Figure 10. Playing the market participants against each other.
In this tactic, the buyer uses information asymmetry and bluffing to achieve the lowest possible transit price points.

Transit Tactic 8 – Build into Cheap Transit Internet Region
Some Internet regions have a relatively expensive Internet Transit market. For example, in 2010, the price of Internet Transit in South Africa and Kenya was close to $400 per Mbps, when Internet Transit could be purchased in London for $2-$4/Mbps. The question is, can you save enough money to cover the cost of building into London from these places?

Figure 13. Buy Internet Transit in a better Internet Region.
For an emerging set of ISPs from the newer ecosystems in Africa, Eastern Europe and the Middle East, the answer is "Yes!" especially when they have the goal to reduce the dependency on the transit providers in their home Internet Region.
A company called SEACOM for example has been lighting transoceanic fiber into each country around the perimeter of Africa. This replaces the high cost and complexity of acquiring cross-border circuits between neighboring countries, and also heads up to Europe. The reason I bring this up is that this oceanic fiber deployment enables this play for many ISPs in Africa.
Frank Orlowski (DE-CIX) Pointed out that this is a popular tactic coming from Eastern Europe and the Middle East as well. Cheap transit ($5-$10/Mbps) is a powerful lure for those who are paying 100’s of dollars per Mbps.

These were excerpts from The Internet Peering Playbook.

DrPeering@DrPeering.net

The Origins of 95/5

Fri, 29 Apr 2011 11:04:49 -0700

Q: DrPeering -
Where did this 95th percentile thing come from?
Unlucky Michael
A:
(excerpts from The Internet Peering Playbook)
The Origin of the 95th Percentile Measurement
The Internet Transit pricing model went through three distinct pricing paradigms.
Internet Transit sold by access capacity. In the early Internet days, Internet Transit was charged on a circuit capacity basis. You would purchase a T1 circuit (1.544 Mbps) to the Internet and pay as if you used the entire capacity of the circuit 24 hours a day. But if you didn’t use very much of this capacity, you were still paying as if you did. At the time, this made Internet Transit tough to sell to some low volume prospects.
Internet Transit sold by average usage. One of the early ISP pioneers started charging based on average use, but this measure ended up being skewed by the general “burstiness” associated with Internet traffic. Customers that used only a little transit were paying as if they used much more because the average was skewed upwards by these bursty outliers.
Internet Transit sold by 95/5 model. To address this, one ISP adopted the 95th percentile measure that was primarily introduced to not overly punish a customer for the occasional spikes in traffic volume, and still allow the ISP to bill based the load placed on its network. This approach seemed palatable to the market and sold well. The rest of the industry followed suit and within a year or so 95/5 was the dominant form of pricing for the Internet Transit service.

The 95th percentile measurement process is typically a three-step process as follows (and diagramed above).

1. Every five minutes the meter on the service is sampled.
2. The delta is calculated between the two adjacent samples and the result is stored. (It is in this step that corrections are made for wrapping counters, counter resets from reboots, anomalies in the measurement on the metering device, etc.)
3. At the end of the month, the five-minute-deltas are converted to Mbps and stacked lowest to highest. It is the 95th percentile value that is used to calculate the traffic volume for the month.

DrPeering@DrPeering.net

The Peering Simulation Game run over over Cisco Telepresence

Mon, 18 Apr 2011 14:07:48 -0700

Since most of the 1800 or so of you reading this RSS feed probably already know what the peering simulation game is, I thought you might find it interesting to hear that we ran the first ever peering simulation game over Cisco’s telepresence.
If you are not familiar with the peering simulation game, it is a teaching tool that I use in the peering workshops. In the game, four players are recruited to play the role of peering coordinator. Each player rolls the dice to grow their network of customers. Each square represents customers and a chunk of traffic. The network is grown by the number of squares as represented on the die. If the peering coordinator builds into an Internet Exchange Point they can negotiate peering with the other players that are also in the exchange. There is a cost of peering of course, so the game turns on the value derived from peering and we observe the corresponding negotiation power. I have led about 35 peering simulation games around the world, and there are others running the game across Africa and in New Zealand -- it is a good lead in to a social.
For those of you who are not familiar with the Cisco telepresence gear, it more than a high quality video conferencing facility where you do forget about "technology" while interacting with others. The telepresence room we used had three HD screens of remote participants sitting in other parts of the U.S. Our session was two days long ( 8 hours a day) and all remote participants were flawlessly present for the whole two days.

How well did the peering game work over the Cisco telepresence environment?
At first, we were aware that the people on the screens were somewhere else in the world. The high resolution screens showed the minute details of the participants, the audio worked well, but in the beginning you still knew you were talking at a HD TV. The three screens were angled inwards with a table in front giving the illusion that the participants are all around a table together. But you still knew it was a screen.
Before an hour was up though, the technology faded into the background and I forgot these folks were not in the room. The body language and subtle things like raising an eyebrow, all came through. It is really pretty amazing.
Not onto the peering simulation game....

Obviously, the remote participants could not roll the dice so they had to depend on us.
There was somewhat of an “us vs them” mentality between the remote participants and those in the room. The third class citizens were those that could dial in for audio only or those that used webex. It wasn’t anything on purpose, it was an artifact that those in the room or in telepresence could just get to speak more quickly because you could see the body language of the people who wanted to speak and there was no delay. The non-telepresence people couldn’t or didn’t seem to be as actively involved.
There was no “video conference fatigue”, the phenomenon that occurs when low rez video conferences go on past an hour. Low resolution video does not give the feeling of real time feedback which apparently leads to subtle effect that tires the brain. The telepresence folks shard this explanation with me.
The negotiation of peering over the telepresence worked well. The remote participants would mute the audio as they “internally” discussed the proposed peering arrangement in the game. When they came back un-muted the negotiation carried forward exactly as it has occurred in any other peering simulation run.
We missed one important aspect of the game using telepresence - the social afterwards. Normally, the participants get to mingle with the audience members. Discussions tend to focus on why this move or that move was made in the game, or why a particular maneuver wasn’t tried. Absent that, the peering game went off just as the game is run in person.
Remote participants were able to go home after the workshop since they didn’t have to travel to the workshop, which brings me to the next point.
The potential client cost savings of telepresence.
I run these two-day peering workshops around the world now, so I will use my price points for the example. The cost of the workshop is the time I am dedicated to the client and for the materials used. The materials used are the presentations (made freely available) and the travel expenses.
Travel Expenses. Business class to South Africa was about $10,000 and involved over 30 hours in transit. It was a long exhausting flight consisting of two 13 hour legs. I made use of the virgin lounge and got a lot of work done for the client in flight, but it was a lot of travel to get there. I needed the remainder of the first day there to recover.
My billing rate is $3000/day for the time that I am dedicated to the client, which includes travel time. The two-day workshop required a total of six days when travel was taken into account.
Figure a couple thousand for hotel, meals, cabs, etc. and you have the $30,000 event.
(I am ignoring my additional out-of-pocket costs for extra nanny time and the burden of having a single parent handle the kids while I am off traveling for this, etc.)

The point is, the cost for this on-site consulting and two-day workshop on the opposite side of the earth cost the client $30,000. Let’s compare that to the cost of telepresence.
Telepresence is available for rental for $200-$250/hr, so the two 8 hour days is about $4000. In both cases I only charge the client for the days dedicated to their activities, so the client would pay for the two days : $6000. The total telepresence cost would be about $10,000. In these circumstances, the client would save 66%.
The assumptions here include that:
1. the client has telepresence gear (they are not paying anything for the telepresence use on their side)
2. the client has the bandwidth capacity (20Mbps sustained)
3. the client trusts that the engagement can take place effectively over the immersive environment
4. the consultant is able to and doesn’t mind giving the workshop on the midnight shift (10PM to 6AM).
It further assumes that there is some sort of directory that allows the client to find me and that we can connect our two conference facilities together. (There is apparently a walled garden approach with this service today, so only customers of the same video service provider can connect.)
Summary. Personally I really love the idea of not traveling so much. I could help more people, it would cost them less, I can do more engagements which lead to being kept on staff on a retainer basis. I would love to cut out the travel time and not deal with the jetlag. Those of you who know me know that I don’t handle I don’t handle the jetlag well. It takes me a week to get acclimated and a week when I return home. During these weeks I wake up at 3 in the morning and fall asleep after dinner. This is why I limit the workshops to one per month. Telepresence not only worked well for the peering game but I consider it a requirement for an effective and profitable international consultant ecosystem.
I presented here maybe the worst-case scenario here. I have an expensive consulting rate charged for the duration of the time away from home to deliver an in-person consulting / workshop on the opposite side of the earth. If a client wanted to pursue the lower cost telepresence alternative, I would certainly consider it now.

DrPeering@DrPeering.net

2011 Internet Transit Market

Tue, 12 Apr 2011 14:19:24 -0700

Q: DrPeering -
What is the price of transit out there now?

Nigel Holland
U.S. RightOn

A:
DrPeering has been tracking the market price for Internet Transit since 1998. Some recent discussions with peering coordinators highlight some resistance crossing the $1/Mbps price point.

A Price Floor at $1/Mbps
The majority of the Tier 1 ISPs in the U.S. are maintaining the transit price floor to $1/Mbps with at least a 10Gbps commit. There are some reported outliers in the $0.25-0.65 price points but unless your name is Google and/or you exchange enormous volumes of traffic, you probably won’t get these prices. Even some the larger content players are paying $1.25- $1.75/Mbps.
For zero commit the U.S. market seems to be still hovering closer to $3-$4/Mbps, but....
Zero commit may not exist anymore.
Some folks have been complaining about salespeople not willing to take phone calls for deals with revenue less than $2000/month. In effect, they say that there is a 500Mbps minimum at a price of $4/Mbps.
What an industry. When do we start pricing in Gbps? Let’s try that pricing model on for size:
Salesguy: “Yes, we sell access to the Internet. Best price in town: $2000/Gbps.”
Customer: (to self) ...shift the comma over one, two, three. $2/Mbps seems high.
Maybe we should instead adopt pricing in cents.
Customer: How much for a transit?
Salesguy: 70. . . . . . 70 cents per meg.
Customer: but ISP A sells it for 47
Salesguy: Buy from ISP A then and come back to me when you have 5000
Customer: Megs?
Salesguy: No, Gigs! The deal has to be worth more than the price of my suit.
So far I haven’t heard anyone anything but $ / Mbps quoted, but we will see.
Level 3 & GLBX
Several people have expressed concerned about the Level 3 acquisition of Global Crossing. Apparently these two players were often played off one another to get better pricing.
As the field of scaled providers narrows, it may be more difficult to get market prices sub-$1000 per Gbps.

DrPeering@DrPeering.net

TheCoreoftheInter.net

Tue, 29 Mar 2011 06:19:21 -0700

Q: DrPeering -
How does one build an International Internet Exchange Point and make it the core of the Internet in the region?

Gern Blanston

A:
Gern -
Many of my discussions with governments and quasi-governmental organizations over the past few years have centered around making their country the core of the Internet in the region.

Some have dropped switches into a colocation center and adopted the "Field of Dreams" strategy - build it and they will come.

In many cases, they do not.

They want the most populated Internet Exchange Point in their country and to lure the strongest international ISPs in to peer at their location. Any why not? The largest Internet Exchange Points in the world (DE-CIX, AMSIX, LINX, etc) have the largest peering populations and the greatest traffic volume, and as a result, some of the lowest Internet Transit prices in the world. Further, once critical mass has been reached, it becomes easier for ISPs to prefer peering in those countries than in neighboring countries. Success breeds success in the IX world.

The problem is that most governments understand this point and want the same thing. Several countries in the middle east for example are each vying to be the core of the middle east Internet. Eastern Europe is catching up fast to western Europe in terms of populations and traffic volumes. And African nations all each seeking to build a strong and less expensive Internet region.

So how should they proceed?

First, read the IX Playbook - here I documented the strategies Internet Exchange Points have used to build, grow, maintain critical mass for their start up Internet Exchange Points. This is a good place to start as it covers why it is so difficult to build critical mass along with advice from some the of the smartest IX operators in the world.

Second, start running in-country peering forums. You need to build a critical mass of Tier 2 ISPs, content providers (and maybe even some Tier 1 ISPs) to discuss the goals of building an IX population. You need build enough of a population at the peering forum to build an IX worth building into.

Second, the government may help but proceed cautiously here. Governments involvement in IXes can often spook the ISPs. In the case of India for example, there was a movement for the government to require Indian ISps to show up and peer at an IX they saw as strategic for the country. This dampened the competing grass roots IX movements, spooked the ISPs about what the government might then require, and put various projects on pause.

Finally, identify the international ISPs that are strategic to building the regional IX and have an interest in being there.

DrPeering@DrPeering.net

Cross Connect Price Increases

Mon, 21 Mar 2011 13:26:42 -0700

Q: DrPeering -
I am calling foul. Cross connects have increased in price from $200 to $300 per month. Ridiculous, given the cost to run a cross connect is a one-time expense of about $40.

How much could a cross connect cost in the future ?

Enrico Pellini

A:
Enrico -
You are referring to the monthly recurring fees that U.S. colocation providers charge customers for (typically fiber) cross connects between two parties within their U.S. data centers. Within a cage, customers can of course run their own wires. It is when the wires cross the customer boundary that the colocation provider owns the interconnects and manages the interconnection process.For interconnects there is a monthly recurring fee in the U.S.
These cross connects are estimated to cost about $40 to run. This includes the cost of the fiber, the cost of the labor to run the fiber, and a small cost allocation for the indirect costs of a database to store the customer information in a database. Lots of assumptions here of course.
But the cost of the cross connect is immaterial. The important question is:
What is the value of the cross connect to the participant? How high could the price get?
Let’s compare free peering using a cross connect against the next best alternative, purchasing Internet Transit. If one could freely peer 5Gbps over a cross connect, and the alternative was to purchase 5Gbps of transit at $1/Mbps, colocation providers could charge as much as $5000/month for that cross connect!
Based on this analysis, the $300/month cross connect is a bargain.
There are a lot of assumptions in this analysis of course, but these are the broad strokes. In both cases (transit and peering) we are assuming peers and transit providers participation at the colocation center, no increase in equipment costs, and no incremental labor costs of peering, etc. The point is that the value derived from the cross connect is calculably higher than the price of the cross connect.
It is interesting to observe that in Europe, cross connects within data centers have always been either run by the tenants of the colocation center, or by the colocation providers for a one-time fee of about $200. So when the European ISPs came to the U.S., they were surprised that there were monthly recurring charges for a piece of fiber run between participants. They were also surprised that the colocation providers were also the operators of the exchange point fabrics, but that is another story.
Some of the European colocation centers I spoke with last week would like to see Europe adopt the U.S. monthly recurring cross connect model. The concern is that the customers would revolt. Some European colocation centers have already started mandating the use of their people to run cross connects and seem to be inching towards the U.S. cross connect model.
A couple interesting anecdotes to share. Since there is a monthly recurring cost for a cross connect in the U.S., ISPs are incented to “cheat” and bypass the system. How do they do this?
1) In one example, an ISP realized that their peer was directly below them on the next floor down. During a midnight installation, they drilled a hole through the concrete floor to run the cable to their peer.
2) In another example, an ISP paid the IX technician several hundred dollars under the table to run the fiber without reporting it to the IX operator.
3) In a final example, the ISPs ordered the cross connect and ordered the disconnect the following month. They realized that the fiber would be run between the two patch panels and then simply disconnected from the two patch panels, or maximally pulled up into the overhead wiring trays. The ISPs would both then fish down the fibers and reconnect them for free cross connect services.

DrPeering@DrPeering.net

The Optimal Colocation Data Center Facility

Fri, 18 Feb 2011 13:52:25 -0800

Q: DrPeering -
If you were operating a data center as a colocation facility and all available space would be leased, would you prefer

a) a single customer that bought all available space,
b) a few large customers that bought all available space, or
i) as many customers as possible, that each take up a little space but collectively booked all available space?

Or are you indifferent, since in all cases the building is fully booked?

Dinard Robinson

A:
Dinard -
The math proves that there is only one right answer.
A colocation facility is a purpose-built data center, typically constructed and operated by a carrier-neutral and ISP-neutral entity -- the operator of the building does not compete against those who colocate equipment within the colocation facility.
Now on to the question at hand.
A colocation operator *will* face the opportunity (a) to lease all available space to a single customer. This may seem great, since all space is sold out, the customer will presumably be credit worthy and pay their bills and provide a margin, but it is far from optimal for a colocation center.
The Exponential Colocation Interconnect Revenue Model
A colocation center makes margins on the interconnections between the colocated parties. A fiber cross connect for example might have a one-time cost of $40 to run, but generates maybe $200/month in recurring revenue! This is on top of the margin that is made on the rack, power, installation fees, etc. Even better is the fact that those who interconnect in a colocation facility, generally don't interconnect with only one other party; they interconnect with potentially hundreds of others.
Interconnection is just about 100% margin, but there is an even stronger motivation for a colocation provider to maximize the interconnect revenue. This is also where the stickiness is - a maximally interconnected colocation facility is one that customers simply will not move out of.
Mathematically,
If n is the number of colocated parties in a colocation facility, then n parties can interconnect with (n-1) others. So a fully meshed colocation facility would see
n*((n-1)/2) interconnects,
each generating $200/mo of pure margin.
So as a colocation facility, you would like to see your n as large as possible, so you can leverage that exponential n(n-1) revenue multiplier.
In choice (b) above, where several large customers lease all of the racks, you have a low value of n so you get very little interconnect revenue.
But choice (c) yields the theoretically maximum interconnect revenue while simultaneously resulting in the stickiest colocation facility.
The Colocation Stickiness Keeps Customers There
To customers, the value of a colocation facility is proportional to who else is in the building for you to interconnect with. A large building with lots of companies that you want to interconnect with is more valuable than one with fewer interesting parties to interconnect with. Therefore, you are less likely to leave a colocation facility filled with valuable interconnect partners. Why? It will be very difficult to reproduce those interconnections elsewhere. So you will stay until the value of participation is less than the cost of participation.
Summary
We see that choice (c) is the most valuable to the customers so they will stay put in the colocation facility, even as the colocation provider takes a premium. And choice (c) is the most valuable to the colocation provider, since they don’t have to offer a sweetheart deal to lease a ton of racks (everyone takes a single rack for example), and the colocation provider also gets the exponential interconnect revenue.
The graph below, which you have all seen before, shows the value of the colocation building to the population. We are simply exercising the graph to answer another common question abut colocation facilities.

DrPeering@DrPeering.net

Verizon.Terremark. neutrality?

Fri, 11 Feb 2011 09:56:51 -0800

Q: DrPeering -
What is the likely impact to the peering community of Verizon acquiring Terremark?

Tate Forcier
A:
Tate -
Teremark hosted the last NANOG meeting in Miami and Josh Snowhorn made a few statements during the opening and during the peering track that Terremark was to continue operating as business as usual. He suggested that there are many new Terremark IXes to be built out under with the same business model as before.
Will this work? Carrier-neutral colo has been purchased by ISPs and carriers before so there is some historical precedence to look at.
In 1999 AboveNet acquired the Palo Alto Internet Exchange (PAIX) and there was great concern raised by the community of the form:
“The PAIX is no longer neutral.”
The rumblings continued that ISPs would move out, that AboveNet would leverage the ownership of the PAIX to their advantage. People suggested that a new neutral exchange point would arise and take over as the peering hub for the bay area. Others suggested that the ISPs would certainly not expand their presence, and prefer instead to take a wait and see attitude.
However, none of these things actually happened.
The PAIX changed hands several times since then without significant disruptions. The PAIX continued to be operated as it always did, and continued to provide a quantifiable peering value to its customers.
All IXes go through the network externality “S” curve as shown below, and the PAIX in Palo Alto was already well past critical mass - the value of peering there greatly exceeded the cost of building in and operating peering infrastructure at the PAIX. The carrier neutrality “principle” was not a strong enough force to make people move out and give up that derived peering value (where peeringValue=valueDerived-costOfPeering).

So if history is any indication, Terremark's facility in Miami will probably continue to contain a strong ecosystem moving forward.
However, the new Terremark buildings present a different situation altogether. Even if the Terremark buildings are operated as a completely separate Verizon subsidiary, building in to these buildings ultimately supports the parent company, Verizon. Since the new buildings have to go through the exact same “start up hump” that all colocation and Internet Exchange Points have to go through (where the value of participation is so much lower than the cost of participation) they will have a harder time attracting the critical mass of carriers and ISPs.
The neutrality message will no longer ring true, so all of the heavy lifting needed to attract ISPs and build the value proposition will be much more difficult. After all, might Verizon get to pitch connectivity first to the new Terremark building prospects? Might Verizon get to peek through the tour and visitor logs to find prospects for bandwidth? Wouldn’t competitors prefer to see Verizon resources drain as a result of an over-built and unpopulated colocation facility? These are examples of concerns raised when carriers compared building into ISP-owned data centers against building into an open market carrier-neutral data center. For example:
“There is no way I will bring Qwest fiber into a Level 3 gateway. It doesn’t make any sense.”
History has a way of repeating itself.
So most likely, Terremark colocation facilities built from here on out will not be perceived as carrier or ISP neutral and will fail as colocation or Internet Exchange Point locations.
On the positive side, I do have faith that Verizon has a ton of cash to build some high quality data centers and that Terremark has the expertise to market and operate them. I just don’t see the success in Miami being duplicated elsewhere with this parent.

DrPeering@DrPeering.net

Next Generation Exchange Points?

Wed, 26 Jan 2011 11:02:01 -0800

Q: DrPeering -

Do you have a list of the top internet exchange points in the top five or six U.S. metros? Do you believe the top exchange points can be replicated by a new entrant or is the critical mass and resulting network impacts among customer ecosystems too strong? Will all the future internet traffic (video traffic) flow through the top existing exchange points or, over time, there is the possibility for new exchange points?

Thanks.

Kevin

A:
Kevin - thanks for the question.

I worked closely with the Tier 1 ISPs to learn that there are essentially 8 “interconnect regions” across the U.S.

The 8 U.S. Interconnect Regions
1. Seattle
2. San Francisco Bay Area
3. Los Angeles
4. Chicago
5. Dallas
6. New York City area
7. Washington DC area
8. Atlanta

The dominant Internet Exchange Points are the Equinix facilities for each region except for
Seattle (where the SIX pretty much rules the roost),
Los Angeles which is pretty much around 1 Wilshire and CoreSite (thanks Eric Bell for pointing this out),
Atlanta (where it is TelX who acquired Michael Lucking’s AtlantaIX at 56 Marietta Street), and
the New York City area where there is an oligopoly of competing IXes around 60 Hudson, 111 Eighth, 25 & 75 Broadway, in NJ Halsey street, etc. Check out this data center map.

The Washington DC area is pretty much Equinix Ashburn. There used to be Switch & Data competing in each of the 8 U.S. interconnection regions, and in practice leading the Bay Area market, but they are now absorbed into Equinix. The word in the field is that these are still operated as separate entities but are slowly consolidating.

In each Internet Peering Ecosystem that I studied, there is generally the same thing - there is usually a dominant IX and maybe a competitor or two in the metro market.

Can these top exchange points be replicated?

It is possible to displace an incumbent IX operator, but very difficult.

I documented the early Equinix vs. MAE-East struggles in “How Equinix beat MAE-East : IX Playbook tactics 8-11” and more broadly in “the Art of Peering : The IX Playbook” . From a decade of research I documented the many tactics used to beat the large incumbent IXes -- these clever tricks is one of the more popular and fun discussions in the peering workshops.

The challenge can be summed up with the graph below where we plot the value of the Internet Exchange point (or colocation facility) over the number of participants.

Every IX and colo starts out with few or no customers. The value of a colocation facility is proportional to who else is there. So how does a challenger attract those first few customers when there is noone to interconnect with? This is a challenge that seasoned IX operators and colocation providers are well versed in. The good news is that there are in fact tricks of the trade to build the critical mass.

The IX has to do whatever is necessary to get to critical mass so the value or participating is greater than the cost of getting in. This period is called the “startup hump.”

Once critical mass is reached, the IX can experience an exponential growth in value. Why exponential? The reason for the exponential growth in value is that n participants can interact with n-1 other participants. The aggregate value of the IX then rises to n(n-1)/2 and the customer can financially justify building in.

The problem with competing against the big colos and IXes is that the value is so high there that it will be very difficult to lure away the new players (who want to peer with the established guard), and even more difficult to lure the established guard away from a place where they derive much value. There are ways of doing both of these things, but there is a bigger problem.

In my experience, the companies that could execute maneuvers against the dominant IXes need to migrate from building racks and selling power to building a population and facilitating the interconnection. Beyond education and an inclination, there is an subtle art to building and maintaining a critical mass or participants, and once obtained it is enormously valuable and protected. So the challenger will require a culture shift, experienced personnel, patience, lots of travel and investment, and a skilled strategic sales team. The payback won’t be immediate, I promise you. But it tends to be long lasting if this is the strategic focus and a long term sustainable differentiator. Those engaged in commodity colo trench warfare need this to command a premium and a profit.

Your last question is about trends and shifts and the impact on the the future of IXes.

History has taught us whenever there are major changes in the ecosystem, there is an opportunity to exploit. The requirements for video traffic and the domination of this traffic type opens up the possibility for a digital media Internet Exchange that may or may not be colocated at the dominant IXes in the U.S. It is possible that an agile new player could leverage this opportunity and make the 8 interconnect regions insignificant in comparison in the next wave of evolution of the Internet Peering Ecosystem.

Another possible future comes out of a couple questions:

“How many IXes are needed in each interconnect region?”

and

“At what point will Switch and Data and Equinix be considered one IX?”

It is possible, but far from certain that the peering community says that they need a second IX
• for redundancy,
• one run by a different operator,
• with different security systems,
• different operating procedures and
• different peering hardware,
• with different ...

At that point, the market emerges for an agile startup IX to become number 2 in the Equinix dominated markets. See :How many IXes per metro for a discussion of the study that showed that ISPs wanted only 1 or 2 IXes per metro and the 9 selection croteria for an IX.

DrPeering@DrPeering.net

Why is Internet Exchange Point neutrality important?

Thu, 20 Jan 2011 10:30:39 -0800

DrPeering -
Q: Why does your site assume a data center has to be operated by a non-ISP and non-carrier?
Bud Fox
A: A few years back I learned first hand from an Internet Service Provider in Turkey why neutrality is an important characteristic of Internet Data Centers marketed as colocation centers.  He shared with me the following story.  There were really only three Internet Service Providers in Turkey, and the Internet traffic exchanged between them traversed the Atlantic Ocean (twice!) before reaching each others customers! This was a perfect place to apply Internet Peering, a direct business relationship between competitors whereby they each freely exchange access to each others customers.  But there was no Internet Exchange Point in Turkey to host an ethernet switch, and no separate commercial or association-based entity to oversee the Internet Exchange Point.   Well, one of the ISPs volunteered to buy the switch and host it at his data center, and even would provide free rack space in his data center. I will call him ISP A. The other two ISPs agreed and they built into his data center and the three ISPs peered with each other.  The traffic volume grew as a side effect of lower latency and lower packet loss. The importance of the Turkish Internet Exchange, owned and operated by ISP A, was recognized as the key to the successful information economy in Turkey and everyone celebrated.  Shortly after that celebration, ISP A started touting his ISP as being “the center of the Turkish Internet”, pointing to the amount of traffic traversing his data center, and the prestige and recognition it had received from operating the Internet Exchange Point. As a side effect, it garnered increased market share, which reinforced its market positioning that it was the center of the Turkish Internet.  The other two ISPs did not like this at all but could not pull out from the Internet Exchange and go back to having their traffic traversing the ocean again.  This is a cautionary tale that illustrates why around the world an Internet Exchange Point is generally operated by a neutral third party, one that does not compete against the participants.These are the motivations that have made the non-neutral IXes all but extinct. One thing that we have learned is that the long road from real estate to an Internet Data Center to a Colocation center and onto the destination of being a successful commercial Internet Exchange Point requires business model neutrality among other things.

For those of you building data centers, trying to make colocation centers, or investing in real estate data centers, I talk a little bit more about the tricks of the trade from IX operators in the IX Playbook, The European vs U.S. Internet Exchange Points, and Modeling the Value of an Internet Exchange. This is part of the base level of knowledge for IX operators, and the basis of a one-day workshop that was developed for IX operators. Colocation centers and Internet Exchange Points are not just lucky pieces of real estate.

DrPeering@DrPeering.net

The 9 Decision Drivers for Colocation Provider Selection

Tue, 18 Jan 2011 16:56:44 -0800

DrPeering -
Q: When firms enter the colocation or Internet Data Center market, what are the key selection criteria?

Daphne Blake
A: During the past ten years I spent time with literally hundreds of firms and asked this very basic question. In this article I highlight the top nine. This is what I learned and documented in the Internet Service Providers and Peering white paper.

We have discussed this before, but each firm tends to have different weighting for each of these issues.

1) Telecommunications Access Issues.

This group of issues has to do with getting telecommunications services into the exchange. How fast can circuits be brought into the colocation center? How many carriers compete for my business for circuits back to my local Point of Presence (POP)? For facilities-based ISPs, what is the cost of building into the exchange (how far away and what obstacles present themselves)?

2) Deployment Issues
This category of issues has to do with getting equipment into the exchange.
How do I get my equipment into the exchange (assuming it supports collocation)? Do I ship equipment in or do I have to bring it with me as I fly in? Will someone act as remote hands and eyes to get the equipment into the racks or do I do the installation myself? Comparing exchange environments in this context, what are the costs associated with deployment (travel, staff time, etc.) into this exchange? Does the exchange have sufficient space, power, air conditioning, etc.

3) ISP Current Presences Issues
This issue is based on the following observation by the peering coordinators: The most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange.

4) Operations Issues
This category of issues focuses on the ongoing operations activities allowed within the exchange after initial installation. Does the colocation facility allow private network interconnections? Are there requirements to connect to a central switch? How is access and security handled at the facility?

5) Business Issues
This category of issues focuses on the business implications of using this IX for peering.
Will using this exchange directly or indirectly support a competitor (contribute to their net income, their credibility, their positioning)? For example, UUNet, a large ISP at the time, also hosted MAE-East, a large IX on the east cost of the U.S. Building into MAE-East supported UUNet’s parent and required purchasing of circuits from its parent as well. A neutrally operated exchange (defined as one that is not owned or aligned with any carrier, fiber provider, or ISP) provides an open distortion-free marketplace for carrier and ISP services.

6) Cost Issues
This broad issue crosses all other issues. What is the cost and value of this Internet Exchange Point? What are the rack fees, cross connect fees, port fees, installation fees? What are the future operating fees going to be? What are the motivations and parameters surrounding these fees? Cost issues shadow most of the other issues listed in this paper.

7) Credibility Issue
Credibility goes to the financial support of the exchange. Does the exchange exist today and will it exist tomorrow? During the early stages of the exchange, ISPs are asked to make a leap of faith when committing, and therefore prefer an exchange with strong backing and the credibility to survive.

8) Exchange Population Issues
These issues focus on the side benefits to using this exchange for this peering. Are there other ISPs at this exchange that are good peering candidates? More importantly, are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)? ISPs will prefer an established and well-populated exchange, particularly one with potential customers that can generate revenue.

9) Existing Exchange vs. New Exchange?
Simply put, ISPs generally prefer an existing exchange to an emerging one. There are dozens of operational exchange points in each region of the U.S. There are also emerging (soon to exist) exchanges that may be soon be peering points at critical mass. However, given the pace of ISP expansion, it is unlikely that emerging exchange offerings are differentiated or compelling enough to be preferred over existing exchanges. Chronic traffic congestion can influence the decision to peer in an existing malfunctioning exchange or wait until a better exchange opens. Customers with heavy flows of regional traffic can also influence the decision. Long- term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one.

DrPeering@DrPeering.net

Internet Peering, Paid Peering and Internet Transit

Fri, 14 Jan 2011 09:43:22 -0800

DrPeering -
Q: There seems to be a lot of discussion surrounding Level 3 and Comcast and their peering, transit, paid peering, etc. Are there standard definitions used here?
Chris “Finchy” Finch
A: Absolutely - I spent a lot of time with the peering community to document the lexicon uses by this community. Below are the definitions that seem to resonate the most:

Internet Transit is defined as a business relationship where an Internet Service Provider provides (usually sells) access to the global Internet. This is typically a metered service at the core of the Internet, although retail customers like you and I typically purchase transit on an unmetered all-you-can-eat basis at our homes.

In the core of the Internet, Internet Transit is sold on a $/Mbps basis; the more traffic sent or received, the more it costs. This leads to the incentive to bypass the metered transit service with some Internet Peering. Internet Peering is the business relationship whereby two companies reciprocally provide access to each other’s customers.

Internet Peering is typically settlement-free, meaning that neither party pays each other for access to each other’s customers. This is reflective of the underlying notion that peering is a relationship of approximately equal value to each party. Since both parties benefit about the same from the relationship there is no need to encumber the relationship with measurement and settlement.

There is a derivative service available from companies like Comcast called paid peering.

A Paid Peering relationship is a peering relationship but with an exchange of compensation from one party to the other.

The compensation could take the form of $/Mbps as is the case with the more public offerings. In other cases it could be an asymmetric allocation of costs or the contribution of services of value.

Example: Paid Peering for Cash: Comcast offers a paid peering service for those who do not meet their settlement-free peering policy. The metered rate is rumored to be in the $2-$4/Mbps price range, in the same ballpark as the market price of transit. Why would one pay the market transit price when they only hear the Comcast routes? Peering generally provides a more direct and better performing path to the eyeballs, and if it costs the same and performs better, why wouldn't you prefer to pay for paid peering?

In any case, these are the three dominant ways to connect to the core of the Internet. My hope is that this article will provide a lexicon and some context for discussions surrounding the asymmetry that is sometimes in place in peering relationships, and as a precursor to discussions on the issues of net neutrality and the future of the core of the Internet.

Here is an article that has these definitions with some graphics that might help understand these concepts.

DrPeering@DrPeering.net

How can Equinix command a premium in this commodity IDC market?

Thu, 13 Jan 2011 18:52:51 -0800

DrPeering -
Q: How does Equinix command a premium in an otherwise commodity Internet Data Center market?
Norville Rogers
A: The short answer is population and stickiness.
For ten years I spent about 90% of my time on the road working with the peering community. I went to just about every Internet Operations conference I could find, from NANOGs in the U.S. to the RIPE meetings in Europe, to APRICOTs in Asia and to speak at IDC (Internet Data Center) conferences in Japan. I gave talks at and participated at over 200 events around the world. I was seeking
1. to deeply understand Data Center market segment,
2. to really know why customer would use an Internet Data Center such as what Equinix was putting in place,
3. to hear what customers were saying about Equinix and its perhaps more importantly about its competitors.
Here are three things I learned.  1) Customers expect that the lights and power will stay on. “What else do you have?” is a common question, and the answer would determine whether you were a commodity IDC or a colocation center capable of commanding a premium such as Equinix. At this point in the conversation a colocation center would invariably talk the peering play, about the value of a colocation center as being proportional to who else is there capable of receiving an interconnect.
1) Customers want to understand the financial value they will receive from participation. When anyone tours a modern IDC they are be overwhelmed with all of the technology, the noise, the industrial look and feel, and there is rarely any context map that tells them what an important IDC they are standing in. This is the where I found my Business Case for Peering white papers became so valuable - this and other white papers provide the “financial proof” of the value derived from participation. All of the information I documented in these white papers was collected from hundreds of conversations in the field, and they all remain publicly and freely available on the DrPeering.net web site as a community service.  3) Customers want to deal with people they like, and in particular, people that they believe will actually help them after the sale is made. A lot of my role at Equinix was to introduce customers to other customers, and to facilitate peering between parties. This was done to help my fellow man -- yes, I am the helpful sort -- but it was also in the best interest of my employer and our stockholders because:  IDC Stickiness is proportional to the degree of dependence on the IDC interconnections.  Nothing is better than a full mesh of interdependence at an Internet Exchange Point – there is so much value derived from participation there, no one would leave! It is this aligned mutual self-interest that leads to value creation and the ability to command a premium in an otherwise commodity IDC market.

DrPeering@DrPeering.net

How do ISPs cost their network?

Thu, 13 Jan 2011 16:13:25 -0800

DrPeering -
Q: Just curious here - How do ISPs cost their networks? With the price of transit dropping as shown in your graphs in “Internet Transit Prices: Historical and Projected”, it is a mystery to me how ISPs know if they are covering their backbone costs. Any insights from your work in the field?
Velma Dinkley
A: There is an excellent white paper here because there is a general sense that prices are so low today that ISPs could very well be losing money on every customer.

Transit Pricing Death Roll
Some ISPs that I spoke with said that the market was pricing transit in a “death roll” because of lack of information on how much it costs to move traffic. It would be good to have a white paper that would do an analysis of the base business model for an ISP service so we can identify the cost floor. Since I don’t have the time (or funding) to do such a study, I’ll share what I learned from a handful of ad hoc conversations in the field. Maybe someone else can take a stab at writing this white paper.

The first few conversations highlighted that there was no single way to cost a network. A couple categories of costing models came up though:

1) No Costing Model assumes that the costs will be covered by market charging market prices.
2) Simple Bit-MIle Costing uses a gross bit mile POP-to-POP calculation
3) Aggregate Backbone Cost Model applies that customer load percentage to the backbone cost.

The No Costing model might apply the cost of the customer port but generally sees the network as a cloud with a kind of “keep it big enough” expense.

The Simple Bit-Mile Costing model tries to establish a network component price per bit mile. This calculation might take all route segments and calculate the average cost per mile. The end result would be a $/bit mile cost number. Then gross traffic analysis would be applied to see what a customer actually uses of the network bit mile resources.

For example, if a customer has 55Mbps of traffic from SFO to Seattle (say 1200 miles) multiplied by the $/bit mile yields $X to carry that traffic. The customer also has about 95Mbps from SFO to LA (say 400 miles) multiplied by the $/bit mile yields $Y to carry that traffic. When taken in aggregate, one can get sense of the order of magnitude of the cost of this customer.

The Aggregate Backbone Cost Model uses the cost of the backbone divided by the amount of traffic a customer sends or receives. If the customer represents 10% of your traffic load, they cost 10% of your backbone cost.

Each of these basic models has pro’s and con’s of course and the conversations always lead to a better and more complicated calculation.

One could for example cost each link and each node in the network and try and estimate the per-customer load on each. But these models get hopelessly complex very quickly and tend to include hundreds of assumptions.
Here are some comments from M. Noufal for example:

“I'm a technical guy but i went through this exercise with marketing several times and every time we come up with a new costing model!
important factors to be taken in to consideration : 1- users contention ratio (this is the most important factor in the costing model). 2- IP cost  3- back haul cost 4- network equipment and administration cost ( this will be added to the costing model as a fixed cost)  5- last mile access ( in case of renting the CPE's for the customers )  also possible factors to be added : 1- consulting cost if any. 2- if you are offering a toll free number for your customer care ( add the cost per customer) 3- advertising cost if its a new product.”
If you have the staff to dedicate to this modeling please contact me - it would be great to have a community ISP Costing Guide white paper to document the best approaches in more detail. In the meantime, what I have shared above is just a shotgun sample of the costing used by a handful of ISPs that I happened to have lunch with over the last few months.

DrPeering@DrPeering.net

Netflix is the answer to Piracy

Wed, 5 Jan 2011 10:33:03 -0800

I love Netflix.
First - the people. I loved working with the Netflix networking folks when they generously contributed their time and shared their insights for the “Video Internet: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem” white paper. The people I worked with believed in my mission of documenting and evangelizing the Internet Operations distribution activities associated with the distribution of content across the Internet using peering, transit, CDN and peer-to-peer. My experience is that these are good people attracted to a company with a good culture.
Second, as a customer, I love the service. My kids love the service - it is just about the most popular app in our house on the iPad. Netflix has done an absolutely masterful job of delivering quality video to this device, and to the Tivo’d and Roku’d TVs in our house. Which brings me to the point:
I believe to a large extent, Netflix is the answer to Piracy.
When I researched “The Emerging Video Internet Ecosystem” I focused some of the time on understanding piracy, mostly because peer-to-peer was such a large consumer of bandwidth. Also, the distribution of the video content was predominantly peer-to-peer. A couple years back I spoke with some of the folks who captured the TV shows on the east coast, ripped out the commercials, and propagated the re-edited versions of the show to the ‘net. Some of these guys did movies as well. I asked them why they did it since there was a bit of work here, a potential penalty if caught, and they received no financial compensation for this work.
Here are the top 8 reasons they gave for piracy
1. They are fans and want to promote their show
2. Tit for tat - They can download more when they upload more
3. Commercials had gotten way out of hand – 44 minutes of content, 16 minutes of commercials. They want the digital format (w/o commercials) for themselves, so no more pain to share this cut with others.
4. Get their name (handle) out there – pride of craftsmanship. For example, the name of a torrent might be "Where.The.Wild.Things.Are.DVDSCR.XviD-NEPTUNE" and these people build their (nick) name and reputation by the quality of their encodings and uploadings. Great care is taken editing out the commercials for TV shows for example.
5. Movie studios put them on the other side of the fence – they create content for you and then sue you
6. The movie is not available in their country within a reasonable timeframe. Australian folks interviewed mentioned this the most often, citing TV shows not arriving in their area for six months to a few years after airing. When they did air, the TV stations edited out a few minutes more of the content to make room for even more commercials!
7. The EFF challenges intellectual property and its place as an exploitation by the powerful (the U.S.) over the weaker. The Pirate Bay makes a similar challenge.
8. The peer-to-peer folks said they have no problem paying for video content if the price was right ($4, not $24), and it was the right product (a movie they wanted to see – marketed, promoted, quality, without commercials, fast-forward enabled, etc.) in the right form (digital, portable, formatted for their many viewing devices) at the right time (not delivered six months after the season ended and their friends had already talked about it for months).

Netflix should be seen as the entertainment industry’s best friend. They provide a really good quality stream of Hollywood entertainment, and the streams are always available. I might even go as far as to say that I see little reason to need a physical media (DVD) or even a local cache on an in-home media server. Netflix streaming is so good that it could potentially remove the need for the local cache that illegal downloads provide.
(BTW - I say almost because my kids have already destroyed a few DVDs. One Scooby Doo DVD was so thin, all it took was pulling it out of the case for it to crack. That is why Netflix prefers to burn their own thick DVDs for their service. So when the DVD broke I wished I had ripped a backup like making a tape of an album and playing the tape instead of the original. I am dating myself here, aren’t I?)
While it takes some time to trust that the content is there in the cloud, and that the streaming quality will always be there, if you believe in the model and the growth of the service, Netflix removes maybe 60% of the reason people illegally download.
So, what about the remaining 40%? Stated another way, If these movies are available on demand, and the streaming quality is high, what other rationale is there for piracy?
1) Netflix content is not portable - for those times when you are on plane or in another country visiting, unless you enjoy watching The Simpsons in Japanese you might wish you had that cache of movies. Carrying on board a couple Netflix DVDs already feels so retro, like bringing a VHS tape or laser disk with you so you have something to watch at your parent’s house.
2) Netflix content is not the latest releases, and any Netflix TV content is certainly older than the East Coast guy who rips out the commercials and redistributes the content to the West Coast. (A valid criticism to be sure, but I would point out that there are a heck of a lot more really good movies on Netflix than new releases this year that I can’t wait to see.)
3) Netflix is only in the U.S. and now Canada
One could view these as weaknesses in the Netflix service - I view them in vulnerabilities to the Movie Studios who stand to lose potential revenue until these issues are addressed.

DrPeering@DrPeering.net

Internet Transit Prices Overseas

Tue, 14 Dec 2010 19:58:11 -0800

DrPeering -
Q: I like the Transit Prices Historical and Perspective page but it is only for the U.S.
What is the price of transit overseas?

Fred C. Dobbs
A: I spent some time speaking at a gig in Montpellier France speaking with Jesse Cheng (City Telecom (HK) Limited) about transit in Hong Kong a few weeks back so this is very recent data.
She said that Hong Kong FTTH is taking off, and as a side effect, lit services to the home are very inexpensive now:

$16/month will get you 100Mbps ethernet Internet service to your home
$27/mo will get you 1G ethernet Internet service.

Pennies per Mbps. Amazing, and she claims that you can actually use a large portion of that - performance has not been an issue.

The fiber is apparently run back to the HKIX where much of that Hong Kong traffic is freely peered. It was interesting to note that (according to Cheng) about 50% of the traffic is destined to and coming from China. It was my understanding from past conversations that China routes (China Telecom, China Netcom) were not freely peered in Hong Kong and was in fact fairly expensive to reach. Maybe someone in the know can help explain what is going on here?

In any case, I will proactively admit:
Yes, Hong Kong is a small place with multi-tenant buildings that are comparatively easy to light up. You build in once and sell to many people inside, and the distances are low so it is relatively easy to offer these services and prices in Hong Kong. If there were mountains, plains, restrictive rights of way, etc. as in the U.S. it would certainly cost more to reach those millions of subscribers.

Still, Hong Kong is enough to make one green with envy, as my AT&T Uverse Internet service costs me $65 for ~16Mbps down, 1.5Mbps up.

DrPeering@DrPeering.net

Paid Peering

Tue, 14 Dec 2010 10:32:37 -0800

DrPeering -
Q: I have a question about how paid peering happens in practice. From the
following scenarios, which would you consider the most common?

1) A network X offers paid peering, where it is willing to peer with
another network, but expects to get paid. X sets a certain price for
the paid peering (how does X set this price? something less than X's
transit price?) and won't peer unless the other network pays that
price.

2) A network X is willing to participate in paid peering, where it can
both pay another network, or get paid. The outcome (whether X pays or
gets paid) and the paid peering price is subject to negotiations. In
this case, X could be paying for certain paid peering links, and
getting paid for others.

3) None of the above :) (in which case, what actually happens?)

Thanks much,
Amogh
A: Paid Peering is like paying for a date to the prom; neither party has an incentive to talk about it.

The buyer often wants others in the field to see it as having similar attractiveness as the peer (that it pays), and the seller of paid peering puts the paid relationship at risk if it discloses that it is actually being paid. At the end of the day, there is money exchanged but very few talk about it as this would defeat the purpose. [This surprisingly apt analogy came up at our Peering Workshop in Johannesburg last month.]

As for your three scenarios....

A few years back, AOL did scenario A, but the price was a little higher than the cost of transit. They argued it reflected the actual cost of delivery. They also correctly argued that it provided a higher performance path to the customers. The market balked at the price and very few purchased it.

Comcast today offers paid peering but price it competitively, and as mentioned in a previous article, those who purchased it say it performs well.

AT&T has offered paid peering as a stepping stone to peering approach for a long time as well.

It is worth pointing out that in both the AOL and Comcast paid peering tactic, free peering was available if some prerequisites were met (traffic volume, geographical distribution, consistent announcements, etc.)

The only time scenario #2 has come up has been in the telephony world (buying and selling minutes and settling one way or the other at the end of the month). I have never heard of this settlement system applied to Internet Peering. It would be like, to torture the analogy, you pay for a date to the prom, but at the end of the night, the two of you settle based on which one of you was more attractive to the party attendees. You would never get a truly attractive date to do this deal, and then how would you quantify value and could that be gamed? Doesn’t tend to happen.

And, for your none of the above scenario....The fallback to peering or paid peering has always been to reach the destinations via transit providers.

The Level 3 - Comcast Dispute
Since many folks have asked for opinions on the Level 3 - Comcast dispute, a related topic I will comment briefly here. I believe both sides have perfectly valid positions. The parties have locked horns with the very human belief that their view is the “right” one and the other view is completely distorting reality.

The Level 3 - Comcast dispute highlights a fundamental flaw in the Internet architecture:

there is no mechanism for dollars to flow to where the bottlenecks are.

DrPeering@DrPeering.net

Content Providers and Open Peering?

Sat, 13 Nov 2010 10:51:08 -0800

DrPeering -
Q: Why don’t they answer my emails? Are they ignoring me? I thought they had an open peering policy?
Emo Philips
iCrushed
A: During the European Peering Forum, Kurt Erik Lindqvist gave a talk about his recent experiences with public peering. (http://www.peering-forum.eu/assets/NewFolder/K-Lindqvist-rootserver-peering.pdf) He runs the I-Root Server which is deployed around the world. Kurtis pointed out that of the 250 emails he sent out, he was able to establish 85 new sessions. That many sessions setup over public peering in a short amount of time is impressive. What struck me was slide 16. Kurtis presented to the group that “Most: No reply at all”. Knowing that we were connected to the same exchanges where I-Root Servers are located, I went back and checked my email to see if we received the email. After checking, I noticed we indeed receive the email. I thought I would provide some feedback to what happened.
The first thing I noticed on the email, was that while it was sent to the correct people, it was sent as a BCC. Most people use email filters which might key on items like who the email was sent to. For me, anything sent to “peering@” gets sent to a specific folder where I can review and respond later. The best option is to check peeringdb.com and see where they want peering emails sent.
Even when you do everything right, you still may not get an answer. This is due to many factors. While you may be connected to one, two or three peering points, some providers are connected to 10, 20 or 30 + different peering points around the world. This means that there are a lot of different emails coming in all the time requesting public peering. While the goal is to respond to all emails, it does not always possible. You should remember, that while peering is a focus, there are many times where peering is just one of the many items they do. Even large companies which from the outside might seem to have unlimited resources, are still constrained by headcount budgets. While it would be nice to have enough people to be available to answer all emails, it is not always the case.
“But they state they have an open peering policy and they still ignore me.” Once again, please never take it personal. It might just come down to resource availability.
If you fall into this situation, what can you do?
The first thing to do is to look at peering with the route-server at the exchange. Router-servers are a great way to get a good number of routes with just one peering session. At some larger exchanges like LINX and AMS-IX, once you connect to the route-server, you are able to instantly get a high number of routes, sometimes exceeding 25,000! This can be the easiest way to pick up peering. What if the exchange you are connected to doesn’t have a route-server? Push the exchange, especially if you are paying them, to install one.
If the route-server doesn’t work, the next option is when you email them, provide all the needed information to bring up the connection. If you do this, it makes it much easier to bring up the session. Also, feel free to provide your phone number and ask them to call to ring you up. Or provide a way for them to contact you, IM, IRC, whatever works for you. It is much easier to bring up a connection when the other person is on the other end of the phone to confirm everything.
Lastly, if all else fails, be persistent. While you don’t want to overdo it, an email to the company every 2 – 4 weeks is not out of line.
Christian Nielsen
Christian Nielsen has worked on peering for 15 years. He started a regional peering exchange back in 1995 in Salt Lake City and has worked on peering for major content providers and networks.

Transit Pricing Leveling Off? Next Gen IX Threat?

Sat, 6 Nov 2010 15:48:35 -0700

DrPeering -
Q: What is happening in the world of Peering? Any future predictions?
Fred Jones
Mystery Machine Investment Services
A: Drpeering had a couple of interesting discussions recently at some peering events.
Transit Prices Leveling Out?
One ISP said that he sees transit pricing leveling off towards $1.50/Mbps this year.
He believes this price as about as close to the cost of delivery of a quality transit service as you can get, and when you factor in the amount of “usable” quality bandwidth, this is about the price you get. What is usable bandwidth?
A low priced provider like Hurricane Electric or Cogent might offer a flat gigE service for $900/month to the market, and people tend to divide the $900 per month by the capacity:

when in reality, that full 1Gbps pipe is not usable. He goes on to explain that when you take into account the layer 2 framing overhead, the bursty nature of the offered load, and the performance characteristics of the upstreams to handle the traffic, that you are lucky to get 70% of the capacity at the 95th percentile. Plugging in that 70% usable capacity figure we see:

I did an ad hoc survey of customers of Hurricane Electric and Cogent expecting to find complaints and poor service stories but I heard only good things. One R&E customer said “It has always worked for me.”
It would be surprising if transit pricing is actually leveling out. Every year since at least 1998, the price of transit has dropped and every year the peering community said “It can’t drop any further” and “no one is making any money at these prices.” And yet the pricing drops every year.
Equinix Risk Factors?
In my consulting practice I get a lot of IDC, Colo, and Internet Exchange Point questions, and Equinix questions on occasion. For example, “I just got back from the Equinix investor event and everything was roses and sunshine. What are some risk factors?” One risk might be that Equinix is so big of an amalgamation, and has never done a stellar job of integrating its acquisitions, so they run the risk of being out maneuvered by more agile competitors.
For example, in the IX Playbook we documented how difficult it is for a new IX to compete with an IX already at critical mass. However, it is also documented how this has been done in the past. Let’s create a scenario. Chinks in the Equinix armor might include the high variability of services across the acquired companies, and perhaps there will be some service failures in the future. Pricing pressures increase as well, providing an opportunity for an agile colo next door to pull some customers away.The momentum of critical mass migrating away from Equinix might happen with a clever service offering including the support for peering with folks that don’t move as people transition over. As history tends to repeat itself, one could envision a group of the larger peers and eyeball networks get together (with some encouragement) to create a new cross connect mesh at a nearby building instead of Equinix. The point is, one could envision an Equinix so focused on dealing with its issues of scale that it is out maneuvered in the market, and as a result, loses that which allows it to command its premium: its critical mass.
Anyway, that sounds like a risk factor to consider.

DrPeering@DrPeering.net

Peer my Content

Fri, 22 Oct 2010 11:39:47 -0700

DrPeering -
Q: We are expanding our CDN into Singapore. What are the arguments people use to get peering with the eyeball networks there?

Cia Dean
A: I asked that question at NANOG 50 in Atlanta of some peering folks that graciously shared their experiences and insights. There appear to be a handful of arguments to try.
Here is what you can say:
1) Mr. Eyeball Network, the cost for you to get to the U.S. is very high - maybe $200/Mbps across the transoceanic cable systems. So, today you purchase transit to reach our network at maybe $30/Mbps. As an alternative, we should consider that this traffic may be cheaper for us to peer directly in Singapore. In fact, the cost savings will most likely cover all of your costs of building into an exchange point. Here is the math:

Here in Singapore, Internet Transit can be had for about $30 per Mbps. If you peer with me, I estimate that we would be able to freely exchange about 667Mbps. We would both save about $20,010 per month!

Of course, this is assuming that you would build into an IX - let’s choose the Equinix facility there for example. As long as the total cost to peer at the IX doesn’t exceed $20,010 per month, we can prove that it makes financial sense to build in and peer there. Whatever remains is gravy!

2) If you build into the IX to peer with us, you reap the benefits of not only peering with us for free. Once you build into the Equinix Singapore IX for example, you can freely reach Microsoft, Google, Yahoo!, and CDNs like Akamai, Limelight, EdgeCast, etc. and whoever else builds in into the future.

3) And of course, there is no chance of us purchasing transit, we are definitely not a sales prospect.

4) And also of course, there is no way that we will compete with you. We only want to provide your customers with great performance.

5) The value of our content can not be understated (tell your content story). And the value will increase over time - we have these initiatives coming up (tell your future story here).

Anyway - these are some of the arguments that might help the discussions. You may need to adjust the numbers, but the general gist is the same.

Dr Peering
DrPeering@DrPeering.net

Origins of Internet Peering

Fri, 22 Oct 2010 06:54:00 -0700

DrPeering -
Q: What were the origins of Internet Peering? When was the first peering agreement?
George Costanza
A: I came across a very nice white paper from Lyman Chapin the Interisle Consulting Group that described the story far more eloquently than I could so I will simply quote the relevant two paragraphs and encourage you all to read the rest of his Interconnection and Peering among Internet Service Providers - A Historical Perspective paper:
The relevant section:
“2.3.2 CSNet and NSFnet
The USENET, BITNET, and commercial X.25 networks could not be connected to
the ARPAnet (or to the other federal networks that were interconnected at the
FIXes) because of the government policy limiting ARPAnet to government
agencies and their contractors. The turning point that eventually brought them
all together was the CSNET project, which was created in 1981 under a grant
from the National Science Foundation . The purpose of CSNET was to link all of
the computer science departments and industry labs engaged in computing
research. It provided TCP/IP interfaces with USENET, BITNET, and the X.25
networks, and established nameserver databases to enable any computing
researcher to locate any other.
The development of CSNet highlighted the disconnect between the “haves” and
the “have nots” in the computing research community—between those who
could find a government agency or contractor to sponsor their connection to the
ARPAnet, and those who could not (connecting instead to CSNet). In modern
terms, we would say that the customers of one ISP (ARPAnet) could not
communicate with the customers of another ISP (CSNet), because no mechanism
existed to reconcile the different Acceptable Use Policies of the two networks.
This disconnect persisted as both sides assumed that any agreement to exchange
traffic would necessarily involve the settlement of administrative, financial,
contractual, and a host of other issues, the bureaucratic complexity of which
daunted even the most fervent advocates of interconnection—until the CSNet
managers came up with the idea that we now call “peering,” or interconnection
without explicit accounting or settlement. A landmark agreement between NSF
and ARPA allowed NSF grantees and affiliated industry research labs access to
ARPAnet, as long as no commercial traffic flowed through ARPAnet. This
agreement was the turning point at which the evolution of commercial network
interconnection began.”

You learn something new every day. I always wondered why peering was traditionally “free”, without mention of the value of what is provided reciprocally. It turns out, there is power in not mentioning.

Dr Peering
DrPeering@DrPeering.net

coloValue != ft²

Thu, 23 Sep 2010 09:05:35 -0700

DrPeering -
Q: What are the key differentiators between colocation providers and real estate companies with conditioned space?
Grey Harry
---------------
A: Here is the typical chalk board model talk I give to my investor clients.
You can express the value creation in the Internet Data Center space as a set of concentric value enhancing rings built around real estate.
At the core you have the real estate, which is, as the expression goes, all about location. For real estate that is destined to be a colocation center, proximity to fiber is critical, as this is what the tenants will ride in on. If the colocation center is far from fiber, then the cost of building in is higher. The most valuable colocation centers are located at fiber nexus points. Real estate and proximity to fiber is so important that we are starting to believe that the value equation in Modeling the Value of an Internet Exchange needs to take into account the cost of building into the IX. So, an IX that is expensive to get to will be inherently less valuable to the peering population than a similar IX that is relatively inexpensive to get into. Why? Because the cost of peering will be less there and it will be easier to attract more peers in. But we digress - back to the real estate core of the model.
The developer at some point will install a fire suppression system in the real estate to make protect it and whatever goes inside. This becomes a more valuable piece of real estate as a result.
When the developer builds out the real estate to include reliable power, the real estate becomes more valuable. Now it isn’t just square feet with fire suppression and location benefits; it also has conditioned power and can provide power during brown and black outs. Build around that a Heating, Venting and Air Conditioning (HVAC) System and you have a facility that not only has the location characteristics and conditioned power, but is also fully conditioned space suitable for reliably hosting telecom equipment. This is more valuable still. For some customers in fact, the availability and price of power are the key selection criteria between colocation companies.
A security ring of services around this conditioned space makes it a secure conditioned space with reliable power in a location with certain characteristics. If this is a single tenant building then perhaps the security ring is a simple single secured exterior and a sufficiently secured employee access point. For multi-tenant data centers however, as all colocation spaces are by definition, the security ring needs things like customer employee authentication and escorts, multi-tenant logging systems with biometrics, camera systems archived for months, etc.
Without Network, a Data Center is an island.
Armed with conditioned space, the colocation space needs carriers - those that can provide transport in to the colocation space for the tenants. These are the folks that bring in layer 1 and layer 2 connectivity - circuits for the tenants to use to get bits into and out of this point of presence. Some data centers are owned by the carriers themselves and rarely have alternative providers selling transport to their customers. Carrier neutral data centers however support multiple carriers selling transport to the customer base, making the well-connected colocation center inherently more valuable than one with zero or only one carrier selling services.
Around the carrier ring is the ISP ring. The ISPs come into the building to establish a point of present, to peer with one another, or to participate (buy/sell) in the marketplace for transit services. Here again, an ISP-owned data center is unlikely to have competitors in their building selling transit to their customers, but a colocation center will tend to have a variety of ISPs forming the open market place for transit services. Some in the community estimate prices for transit to be 30% less in well populated colocation centers than outside the colocation center. The more ISPs the better as far as the colocation provider is concerned as the open and robust marketplace for transit and peering are the key value propositions that will bring additional customers in.
FInally, hosting companies, portals, CDNs, content providers, etc. tend to prefer a richly networked ISP colocation center with an open marketplace of ISP services. This provides the required flexibility, robustness and ever decreasing transit prices that are reasons that large scale content providers build into colocation centers in the first place.
What differentiates colocation centers from the real estate players?
The key differentiator between providers of real estate and providers of colocation space is the managed ecosystem. Colocation is more than a real estate play - successful colocation providers understand their customers ecosystem, and build the right mix of players based on how customers interact with and derive value from each other.

Dr Peering
DrPeering@DrPeering.net

Peering Personals

Tue, 7 Sep 2010 08:23:55 -0700

DrPeering -

I am new to peering and it seems kind of cliquey. How does one break in?

Reginald Barclay
NCC-14232

------------
Reginald - thanks for the question. We will answer from the DrPeering library here.

First off, it tends to take three sequential meetings to be seen as a regular at any smallish (<500 attendee) Internet conference. The first time, you know no one, and no one knows you, and maybe you meet a couple people. You may notice a couple people who seem to know everyone. The second time, you see the people who seem to know everyone again and maybe a person or two you met at the first one. The third one, maybe you meet one of those people who seem to know everyone, and maybe they recognize you from the previous meetings. It is face recognition and company affiliation thing, at least this is what people seem to key on, as in “Oh, this is Reg from the engineering group at NG Telekom.”

Companies that send different people to each peering fora, usually out of an internal sense of fairness, do themselves a disservice since no one on their staff is ever recognized as a regular attendee.

Is the Peering Community Cliquey?

From Merriam-Webster - The Free Dictionary:

cli·quey/ˈklēkē/Adjective
1. (of a group or place) Tending to form or hold exclusive groups.
2. (of music or art) Appealing only to a small group or minority.

It is not as exclusive as it may appear from the outside. I would guess that maybe 60% of the attendees already know others at a peering forum. The regular attendees with the larger social networks seem to have little catchup reunions when they first see each other at the meetings, so it may seem like you are attending a strangers party. But there are some very cool people that like to help.

There are about five people I would refer to as “Social Centers” in the Peering Coordinator Community who do seem to know everyone. They are also very nice about doing introductions to peering people that you want to meet. It is important that you have your top 50 target AS list handy and know who is attending the event from these target organizations. It is also helpful to have some background information on your targets, like their peering inclinations, who they already peer with that is like your company, etc.

And don’t go after the largest or the most selective or restrictive targets first. You might think that this would make you appear to be a more attractive peering candidate or an important player, but it will most likely come off as ignorance. Everyone that is familiar with the industry sector knows that no amount of cozying up to Sprint (for example) will result in peering, and that Sprint knows who their peering candidates are already. Bringing them up first will backfire; the larger peering targets, if it is possible to per with them, will require some skill and strategy. You do not want a newbie to pursue this.

When we researched the peering process we identified the ways that seasoned peering coordinators identify who they needed to speak with about peering (see the top ten list below).

Peering Personals Helps

Many peering fora have adopted the Peering Personals that Ted Hardie and I started back at the Equinix Gigabit Peering Forums in 2004. We evolved it a bit with icons, a bit of social engineering, and even a Windows-based program to execute it, but fundamentally Peering Personals has remained the same - a broadcast method for peering people to meet each other at a peering forum. After all, that is one of the big reasons that they attended the peering forum in the first place.

Here is how Peering Personals works:

We put the peer’s important information on the screen behind the peering coordinator. Specifically, the screen highlights the peer’s ASN, company name, amount of traffic they exchanged in total, what they looked for in a peer and why people in the audience should want to peer with them, contact information, etc. This way, while they speak, the audience can jot down the information, and not have to count on hearing the speaker say the words. People sometimes get nervous and speed through this information. Sometimes there is a language issue. Sometimes people forget to say important things. The MC has a role here to make sure that the interesting, relevant, important information is communicated to the group.

Frank Orlowski and the DE-CIX team put on a top-rate peering forum last week in Frankfurt that included a Peering Personals section. The MC role is important as it facilitates the audience discussion; if done correctly, the Peering Personals feels like a dialogue between the audience members and the peers and helps speed the peering discussions that follow.

This Peering Personals method has proven to be an effective method up to about 20 peering personals; after 15-20 peering personals people seemed to forget what the first person said and what they looked like.

Dr Peering
DrPeering@DrPeering.net

Peering into Africa

Wed, 18 Aug 2010 08:35:17 -0700

DrPeering –

The Kenya Internet Exchange Point (KIXP) is launching a second IX in Mombasa, with the Nairobi one carrying only about 100Mbps while the European IXes are carrying towards 1 Tbps. What is the value in peering at the KIXP?

P.T.
--------------------
P.T. - thanks for the question !

Here is an application of "Modeling the value of an Internet Exchange Point" for the Kenya Internet Exchange : http://drpeering.net/IXPs/KIXP.php . Where did we get the data?

DrPeering was an invited speaker at the 1st African Interconnection and Peering Conference in Nairobi, and I will tell you that these guys ran a top notch peering forum! They attracted about 150 people from over 20 countries - mostly ISPs, with some regulators and IXP Operators in attendance as well. Some of the comments I heard included

“There was more content packed in these two days than most peering events have in three. “

You would never guess that this was the first peering forum -- the speakers were solid, sharing real-world experiences, discussing openly the challenges, contextual nuances, and what has worked. The participant discussions were dynamic and insightful. It was as effective as any peering conference I have attended.

After the event, a bunch of us stopped over at the Kenya Internet Exchange Point ( KIXP ) for a tour and we collected some data surrounding your question. Here is what we learned.

Data Collection

Our guide ( Nick, shown here) estimated that 100Mbps of traffic is peered at the KIXP, and according to Fiona Asonga (CEO,Telecommunications Service Providers Association of Kenya/ Kenya Internet Exchange Point) the going rate for Internet transit is about $400/Mbps in Kenya. From this we can calculate the value that the peering population derives from the IX : the participants’ transit fees are decreased by 100Mbps * $400/Mbps = $40,000 per month.

But to calculated the value of the KIXP to the peering population we need to take into account the cost of peering there. We applied the “Modeling the Value of an IX” and documented the exercise in the page below:

The Utility and Pain of Internet Peering

Mon, 16 Aug 2010 06:23:30 -0700

The Utility and Pain of Internet Peering

From the Desk of Dr. Savageau

In the early 1990s TWICS, a commercial bulletin board service provider in Tokyo, jumped on the Internet. Access was very poor based on modern Internet speeds, however at the time 128kbps over frame relay (provided by Sprint international) was unique, and in fact represented the first truly commercial Internet access point in Japan.
The good old boys of the Japanese academic community were appalled, and did everything in their power to intimidate TWICS into disconnecting their connection, to the point of sending envelopes filled with razor blades to TWICS staff and the late Roger Boisvert (*), who through Intercon International KK acted as their project manager. The traditional academic community did not believe anybody outside of the academic community should ever have the right to access the Internet, and were determined to never let that happen in Japan.
Since the beginning, the Internet has been a dichotomy of those who wish to control or profit from the Internet, and those who envision potential and future of the Internet. Internet “peering” originally came about when academic networks needed to interconnect their own “Internets” to allow interchange of traffic and information between separately operated and managed networks. In the Internet academic “stone age” of the NSFNet, peering was a normal and required method of participating in the community. But,… if you were planning to send any level of public or commercial traffic through the network you would violate the NSFNET’s “acceptable use policy/AUP” preventing use of publically-funded networks for non-academic or government use.
Commercial internet Exchange Points such as the CIX, and eventually the NSF supported network access points/NAPs popped up to accommodate the growing interest in public access and commercial Internet. Face it, if you went through university or the military with access to the Internet or Milnet, and then jumped into the commercial world, it would be pretty difficult to give up the obvious power of interconnected networks bringing you close to nearly every point on the globe.
The Tier 1 Subsidy
To help privatize the untenable growth of the NSFNet (due to “utility” academic network access), the US Government helped pump up American telecom carriers such as Sprint, AT&T, and MCI by handing out contracts to take over control and management of the world’s largest Internet networks, which included the NSFNet and the NSF’s international Connection Managers bringing the international community into the NSFNet backbone.
This allowed Sprint, AT&T, and MCI to gain visibility into the entire Internet community of the day, as well as take advantage of their own national fiber/transmission networks to continue building up the NSFNet community on long term contracts. With that infrastructure in place, those networks were clear leaders in the development of large commercial internet networks. The Tier 1 Internet provider community is born.
Interconnection and Peering in the Rest of the World
In the Internet world Tier1 networks are required (today…), as they “see” and connect with all other available routes to individual networks and content providers scattered around the world. Millions and millions of them. The Tier 1 networks are also generally facility-based network providers (they own and operate metro and long distance fiber optic infrastructure) which in addition to offering a global directory for users and content to find each other, but also allows traffic to transit their network on a global or continental scale.
Thus a web hosting company based in San Diego can eventually provide content to a user located in Jakarta, with a larger network maintaining the Internet “directory” and long distance transmission capacity to make the connection either directly or with another interconnected network located in the “distant end” country.
Of course, if you are a content provider, local internet access provider, regional network, or global second tier network, this makes you somewhat dependant on one or more “Tier 1s” to make the connection. That, as in all supply/demand relationships, may get expensive depending on the nature of your business relationship with the “transit” network provider.
Thus, content providers and smaller networks (something less than a Tier 1 network) try to find places to interconnect that will allow them to “peer” with other networks and content providers, and wherever possible avoid the expense of relying on a larger network to make the connection. Internet “Peering.”
Peering Defined (Wikipedia)
Peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the customers of each network. The pure definition of peering is settlement-free or “sender keeps all,” meaning that neither party pays the other for the exchanged traffic; instead, each derives revenue from its own customers. Marketing and commercial pressures have led to the word peering routinely being used when there is some settlement involved, even though that is not the accurate technical use of the word. The phrase “settlement-free peering” is sometimes used to reflect this reality and unambiguously describe the pure cost-free peering situation.
That is a very “friendly” definition of peering. In reality, peering has become a very complicated process, with a constant struggle between the need to increase efficiency and performance on networks, to gaining business advantage over competition.
Bill Norton, long time Internet personality and evangelist has a new web site called “DR Peering,” which is dedicated to helping Internet engineers and managers sift through the maze of relationships and complications surrounding Internet peering. Not only the business of peering, but also in many cases the psychology of peering.
Peering Realities
In a perfect world peering allows networks to interconnect, reducing the number of transit “hops” along the route from points “A” to “B,” where either side may represent users, networks, applications, content, telephony, or anything else that can be chopped up into packets, 1s and 0s, and sent over a network, giving those end points the best possible performance.
Dr Peering provides an “Intro to Peering 101~204,” reference materials, blogs, and even advice columns on the topic of peering. Bill helps “newbies” understand the best ways to peer, the finances and business of peering, and the difficulties newbies will encounter on the route to a better environment for their customers.
And once you have navigated the peering scene, you realize we are back to the world of who wants to control, and who wants to provide vision. While on one level peering is determined by which vendor provides the best booze and most exciting party at a NANOG “Beer and Gear” or after party, there is another level you have to deal with as the Tier 1s, Tier 1 “wanna-be networks,” and global content providers jockey for dominance in their defined environment.
At that point it becomes a game, where personalities often take precedence over business requirements, and the ultimate loser will be the end user.
Another reality. Large networks would like to eliminate smaller networks wherever possible, as well as control content within their networks. Understandable, it is a natural business objective to gain advantage in your market and increase profits by rubbing out your competition. In the Internet world that means a small access network, or content provider, will budget their cost of global “eyeball or content” access based on the availability of peering within their community.
The greater the peering opportunity, the greater the potential of reducing operational expenses. Less peering, more power to the larger Tier 1 or regional networks, and eventually the law of supply and demand will result in the big networks increasing their pricing, diluting the supply of peers, and increasing operational expenses. Today transit pricing for small networks and content providers is on a downswing, but only because competition is fierce in the network and peering community supported by exchanges such as PAIX, LINX, AMS-IX, Equinix, DE-CIX, and Any2.
At the most basic level, eyeballs (users) need content, and content has no value without users. As the Internet becomes an essential component of everybody on the planet’s life, and in fact becomes (as the US Government has stated) a “basic right of every citizen,” then the existing struggle for internet control and dominance among individual players becomes a hindrance or roadblock in the development of network access and compute/storage capacity as a utility.
The large networks want to act as a value-added service, rather than a basic utility, forcing network-enabled content into a tiered, premium, or controlled commodity. Thus the network neutrality debates and controversy surrounding freedom of access to applications and content.
This Does Not Help the Right to Broadband and Content
There are analogies provided for just about everything. Carr builds a great analogy between cloud computing and the electrical grid in his book the “Big Switch.” The Internet itself is often referred to as the “Information Highway.” The marriage of cloud computing and broadband access can be referred to as the “4th Utility.”
Internet protocols and technologies have become, and will continue to be reinforced as a part of the future every person on our planet will engage over the next generations. This is the time we should be laying serious infrastructure pipe, and not worrying about whose content should be preferred, settlements between networks, and who gives the best beer head at a NANOG party.
At this point in the global development of Internet infrastructure, much of the debate surrounding peering – paid or unpaid, amounts to noise. It is simply retarding the development of global Internet infrastructure, and may eventually prevent the velocity of innovation in all things Internet the world craves to bring us into a new generation of many-to-many and individual communications.
The Road Ahead
All is not lost. There are visionaries such as Hunter Newby aggressively pushing development of infrastructure to “address America’s need to eliminate obstacles for broadband access, wireless backhaul and lower latency through new, next generation long haul dark fiber construction with sound principles and an open access philosophy.”
Oddly, as a lifelong “anti-establishment” evangelist, I tend to think we need better controls by government over the future of Internet and Internet vision. Not by the extreme right wing nuts who want to ensure the Internet is monitored, regulated, and restricted to those who meet their niche religions or political cults, but rather on the level of pushing an agenda to build infrastructure as a utility with sufficient capacity to meet all future needs.
The government should subsidize research and development, and push deployment of infrastructure much as the Interstate Highway System and electrical and water utilities. You will have to pay for the utility, but you will – as a user – not be held hostage to the utility. And have competition on utility access.
In the Internet world, we will only meet our objectives if peering is made a necessary requirement, and is a planned utility at each potential geographic or logical interconnection point. In some countries such as Mongolia, an ISP must connect to the Mongolia Internet Exchange as a requirement of receiving an ISP license. Why? Mongolia needs both high performance access to the global Internet – as well as high performance access to national resources. It makes a lot of sense. Why give an American, Chinese, or Singaporean money to send an email from one Mongolian user to another Mongolian user (while in the same country)? Peering is an essential component of a healthy Internet.
The same applies to Los Angeles, Chicago, Omaha, or any other location where there is proximity between the content and user, or user and user. And peering as close to the end users as technically possible supports all the performance and economic benefits needed to support a schoolhouse in Baudette (Minn), without placing an undue financial burden on the local access provider based on predatory network or peering policies mandated by regional or Tier 1 networks.
We’ve come a long way, but are still taking baby steps in the evolution of the Internet. Let’s move ahead with a passion and vision.
(*) Roger Boisvert was a friend for many years, both during my tensure as US Air Force officer and telecom manager with Sprint based in Tokyo (I met him while he was still with McKinsey and a leader in the Tokyo PC User’s Group), and afterwards through different companies, groups, functions, and conferences in Japan and the US. Roger was murdered in Los Angeles nine years ago, and is a true loss to the internet community, not only in Japan but throughout the world.
Comments Welcome -

John Savageau <savageau@pacific-tier.com>

A 100G Peering Break Even Point Analysis

Wed, 4 Aug 2010 09:47:36 -0700

DrPeering -

What is the best possible unit cost of traffic exchange I can realize with 100G Peering?

Julie Tinklenberg

----------------------------
The question was also asked at NANOG 49 a few months back and here is a back-of-the-envelope analysis we did at the table to determine the best possible cost for peering exchange.

The Business Case for Peering at 100G (port only)

I asked a couple of IX operators at NANOG what 100G port pricing might be for their planned 100G service.

One IXP said maybe $25K/mo, but maybe as low as $15K/mo. Another IXP said they probably couldn't sell it at $15K and maybe $10K/mo was the likely pricing. The latter IXP said that the pricing of peering ports tends to be 4x the cost for 10x the bandwidth. So with $2500/mo being the market price for 10G ports today, one could expect the price of 100G ports to settle down to about $10K. We will consider each of these prices and continue with simplifying assumptions to facilitate discussions.

Assumptions
1) Let’s assume that the 100G transport into an IX colocation center is already in place for either transit or for peering. (Remember - we were asked for the best case scenario.)
2) Let’s assume the 100G IX switch is available and there is plenty of bandwidth.
3) Let’s further assume that the next best alternative to peering at this time is paying a
metered transit rate of $0.63/Mbps (Just for illustration but perhaps not totally unreasonable based on the Transit Pricing Historical and Projected estimates and NANOG discussions.)
4) We will also assume there is enough peering traffic to fill the 100G port.

The model we are working with is shown in Figure 1 below.

Figure 1 - 100G Peering Costing Model

So let’s consider the three price points:
A) 100G peering port is priced to $25,000 per month,
B) 100G peering port is priced to $15,000 per month, and
I) 100G peering port is priced to $10,000 per month.

(We are ignoring the cost of colo, equipment, transport to the IX,
etc. for now - add whatever you like for your modeling.)

With these assumptions we see the (relatively) narrow windows where 100G peering makes sense financially. There are those who are quick to point out the other benefits to peering (better performance, more control over end user experience, marketing benefits, operations impact, etc.), and they are correct.

Figure 2 - 100G Peering Analysis Table

In these case, the Peering Break Even Point is where as ISP is financially indifferent
between sending traffic to its upstream ISP(s) at $0.63/Mbps or
sending that traffic to a peering port with at a fixed aggregate
monthly fee of $25,000, or $15,000, or $10,000 per month.

The Effective Peering Range is the range of peering traffic volume where peering provably makes financial sense. This is a useful metric since, based on the table above, we can say:

“If you can peer 39.7Gbps, I can prove to you that peering at 100G makes sense when the cost of peering is $25,000 per month. Then you can additional traffic, up to 70Gbps, on that peering port and save money doing so.”

“If you can peer 23.8 Gbps, I can prove to you that peering at 100G makes sense when the cost of peering is $15,000 per month. Then you can additional traffic, up to 70Gbps, on that peering port and save money doing so.”

“If you can peer 15.9 Gbps, I can prove to you that peering at 100G makes sense when the cost of peering is $10,000 per month. Then you can additional traffic, up to 70Gbps, on that peering port and save money doing so.”

What is the best cost of peering I can hope for if I fully utilize a
100G peering port?

To answer this question, the best case scenario is that 100G peering costs $10,000 per month, and let's say you can push 70Gbps at the 95 percentile.

So the lowest unit cost is $10,000 per month / 70,000 Mbps = $0.14/Mbps

On the DrPeering Google Groups message board and private discussions, one reviewer said :
“If the best I could do with 100G peering was costing me 25% of the cost of transit, then I would not go the public peering route. I need to save a lot more given the cost and the risk. You should do the private peering analysis as well.”
Yes - if this is really your criteria than you should add to the cost of peering some of the other costs. If you are attending the DECIX Anniversary meeting in Frankfurt August 31, you would like the “100G Peering - Public vs. Private Peering” analysis that we will share.
Another Reviewer said:
“I don't think any analysis without a hardware estimate depreciated over time for their port cost is valid.”
I agree it would be a good analysis to do. We need the find the cost of for 100G Peering kit, and maybe while we are at it, the cost of 100G transport into an IX.
We did this type of analysis for several previous white papers including Business Case for Peering in 2004, The Video Internet: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem and The Great Public vs Private Peering Debate. You know what we found? The amortized cost of the equipment was a pretty small cost component compared to the cost of everything else (transport, transit fees, colocation space, staff, etc.) but we need to do this analysis next.

One final reviewer said :
“Asked a friend how much transit was and he said $0.63 is way below market, that $5 is a good price today, and that is with pretty large volumes - this ISP might be in the camp exploring 100G peering.”
This is a good example of how wide the transit prices range as we mentioned in the Transit Pricing Historical and Projected white paper. If the price of transit is higher, Internet Peering of course looks more attractive. The nice thing about models is that you can easily adjust the numbers to your situation and immediately see the results.

Dr Peering
DrPeering@DrPeering.net
PS -
Another study that might be worth doing would be to compare the case where the service provider in question is NOT already in an IX. One reviewer indicated that in this case the transit pricing of $0.63 is about right but would include the transport circuit to their facilities, while the peering option really needs to include the 100G transport into the IX.The results can be estimated by simply adding the cost of the 100G transport to the port pricing shown above.

Acknowledgements
Thanks to Dave Temkin, Jay Sjoholm, Lane Patterson, Job Witerman, Chris Malayter, Pete Cohen, Joel Hanicott, Pat Bennet-Ramsey, and Alan Hannan for their conversations and insights.

Par for Peering

Mon, 12 Jul 2010 18:13:46 -0700

DrPeering -
Peering is now our strategy... How much traffic can a content heavy network realistically peer away?
Cheryl Will
AS XXXXX

Cheryl -
Thanks for the question.
Diagnosis: Peering success will depend on a variety of factors such as:
1) traffic profile f(volume, dest, type, growth, IX presences, etc),
2) peering coordinator skill and approach,
3) timing and peering inclination trends in the target peer community, etc.
but the window of peering opportunity in the U.S. has been closing a bit.
I used to tell folks that they could expect maybe 40% of their traffic to be easily peerable, and the remainder gets exponentially more difficult to peer away. Recent discussions with large scale peering folks however reveal that it is harder to peer away that first 40% than it used to be. They point to the rising peering requirements of the access networks, and perhaps their predilection toward paid peering instead of free peering. They point to the Arbor Report which shows fewer sources and sinks are responsible for a larger and larger chunks of traffic. For whatever reason, peering with the access networks is harder and probably more important. Given that you are content heavy, it may be more difficult than it used to be to get to 40% free peering.

I would estimate peering par today to be closer to 25% - expect that a good peering coordinator should be able to easily peer away about 25% of your traffic at a couple IXes.

Prescription: I would advise that, if you need peering for performance or marketing strategy reasons, you should get in now before it gets harder. Peering tends to stay in place even as peering policies change. There will be some companies that de-peer because they adopted more stringent policies over time, but this is not the norm. Historically, legacy peering tends to stick around. Because of this stickiness, back in the boom years, companies were even valued (in part) on what peering they had! Some ISPs even purchased companies with the expectation that the Tier 1 peering sessions would survive the acquisition.
So,
1) there is value in peering,
2) the window of opportunity is closing,
3) peering sessions tend to stick around even as policies for new peer changes.
If peering is a strategic intent, you should get in now.
Comments: The other thing I can suggest to you is to send in your top 50 target peers. We have seen remarkable similarity in peering profiles across companies in the same sector. We can predict who will peer, who will not, how to approach, etc. from the list and provide you with some feedback/suggestions. This is often done to help new ISPs select the best Internet Exchanges for their profile. We have done hundreds of peering introductions over the years and would be glad to help you with this.
Always track the price of transit as well - you might find over time that your split of traffic rationally changes over time. Lots of tools like paid peering and inexpensive high quality CDNs are available now and should be in your tool bag.
Hope this helps -

Dr Peering
DrPeering@DrPeering.net

U.S. Internet Transit Pricing - Historical and Projected

Thu, 10 Jun 2010 10:36:00 -0700

DrPeering -
What is transit going for in the U.S. ? At what point do peering not make sense anymore?
Cat D’etat
-------
Cat - I will share some data from an early draft report: Internet Transit Pricing - Historical and Projected along with one prognosis for the peering industry.
First, here is are the DrPeering historical and projected transit prices for the next few years in the U.S.:
When does Internet Peering Make Sense?

What does this table show? Transit Prices are dropping, and the range where public peering makes sense is getting narrower over time.
If you assume the cost of public peering is $10,000/month ($2500 for a 10G public peering port and $7500 for colo, power, transport, equipment, etc.), you see that in 2010, peering makes sense financially if you can peer away at least 2Gbps (2000Mbps) for free to your peers. This gives you plenty of breathing room to peer traffic away for less than the price of transit, up to 7.6Gbps (7600Mbps), which is approximately the amount of traffic one can reasonably sustain over a 10G peering port. But this is for 2010 with an average transit price of $5/Mbps.
With these pricing estimates we see that in 2012, peering still makes sense, but with an average Internet Transit price of $2.34/Mbps, we see a peering break even point of 4274Mbps where peering begins to make sense. Beyond this point, peering makes sense financially up to 7.6Gbps, the capacity of the pipe.
This is not much of a range; getting peering traffic volumes to fit in this range is tricky, so peering folks will mostly likely not go forward with public peering (all things remaining equal).
As fewer and fewer ISPs publicly peer, the rationale for public peering to aggregate smaller peering sessions together over a public peering fabric diminishes. And the technology is not well suited today to handle massive peering sessions over public peering fabrics.
As a result, Private Peering will dominate the peering ecosystem in the U.S. by 2012. Point to point fiber is just too cost effective, reliable, and ease to deploy to compete with.
It is certainly possible that the pricing of IX ports will drop, but the U.S. has been slow to do that compared with the European IXes who have consistently led the charge in dropping peering port fees.
Take a look and I would love to hear your feedback on the early draft of the report:
http://peering.drpeering.net/white-papers/Internet-Transit-Prices-Historical-And-Projected.html
There are some other predictions in the report
1) The U.S. Peering Sector will refocus on Video Distribution Companies, High Performance Network Services Sector, while maintaining customers that simply stick around,
2) ISPs and CDNs will morph into higher value service providers or die,
3) Access Networks grow in power in the Peering Ecosystem, and
4) Cooperative inter-AS services may bridge the gap.
All of these predictions are of course 100% guaranteed to come true or your RSS feed is free!

Dr Peering
DrPeering@DrPeering.net
P.S. - Several have asked for a 100G Public Peering Port break even analysis. We will do this in the next article.

Pursue a Paid Peering Strategy?

Thu, 3 Jun 2010 10:04:50 -0700

DrPeering
We are a large access network in Asia. Is it possible to forge relationships with content providers so they pay to peer with us? What should be my strategy?

regards
Tan

-------
Tan - Yes.

At the heart of Paid Peering is the power relationship. Understand your power position.

In every Internet Peering Ecosystem we studied, Access Network customers are typically singly homed. The Access Network (you) connect many “captive” customers, customers who are reachable only through you. The Content Providers that wish to send their content to your customers have no alternative ; to get their content to these eyeballs they have to send it directly (by peering) or indirectly (through transit) through you. Since peering will be better performance, it is reasonable that some might pay extra for the privilege.
The Content Providers on the other hand are typically multi-homed and provide access to only one customer - themselves ! This is the key aspect to this power asymmetry; they need you more than you need them.
So the content provider question is:
“Is it worth money to get my content closer to the eyeballs? “
If you follow the lead of Comcast and price Paid Peering similar to Internet Transit pricing, you can enable the Content Provider better performance and extract a little revenue in the process.
There are a few issues to consider of course. Among other things, you will need to develop the internal capabilities to support these paid peers as customers. You will need to develop a Peering Policy that articulates your peering inclination; we have done several peering workshops with companies to do this, bringing their multi-disciplinary Peering Steering Committee up to speed. (You don’t need a workshop to do this by the way - just about all of the material we would use to bring your team up to speed is on the DrPeering.net web site. )
So yes, you do have the ability.
It has always been an interesting debate in the Peering Community about whether access networks should charge content providers to deliver content your customers requested.

Dr Peering
DrPeering@DrPeering.net

Top 9 Internet Exchange Selection Criteria

Tue, 18 May 2010 09:54:21 -0700

DrPeering -

We are building out our peering infrastructure....How do people select an IX?

Razi

---------
Razi - thanks for the note.
This was one of the first questions asked of the peering community and documented in “Internet Service Providers and Peering”:
How do you choose an IX?
What we learned from the hundreds of conversations was that there are essentially 9 Internet Exchange Point Selection Criteria as shown in the flow chart below.

Top 9 Internet Exchange Point Selection Criteria

Figure 1 - Exchange Environment Selection

We will talk briefly through each of the top 9 IX selection criteria.
1) Telecommunications Access Issues
These issues have to do with getting telecommunications services into the exchange. How fast can circuits be brought into the interconnection environment? How many carriers compete for business for circuits back to my local Point of Presence (POP)? For facilities-based ISPs, what is the cost of trenching into the exchange (how far away and what obstacles present themselves)? Are there nearby fiber providers that lease strands? These questions will help answer the most important question to ISPs: How fast can my peer and I get connectivity into the exchange? Multiple carriers lead to speed and cost efficiencies. Some ISPs have volume deals with certain carriers or otherwise prefer carriers and therefore prefer exchanges where these carriers can quickly provision circuits. These answers strongly impact the desirability of the exchange environment.
2) Deployment Issues
These issues have to do with getting equipment into the exchange. How do I get my equipment into the exchange (assuming it supports collocation)? Do I ship equipment in or do I have to bring it with me as I fly in? Will someone act as remote hands and eyes to get the equipment into the racks or do I do the installation myself? Comparing exchange environments in this context, what are the costs associated with deployment (travel, staff time, etc.) into this exchange? Does the exchange have sufficient space, power, air conditioning, etc. The answers to these questions impact the deployment schedule for the ISP(s) engineers and the costs of the interconnection method.
3) ISP Current Presences Issues
This issue is based on the following observation by the peering coordinators: The most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange. There is a hidden assumption here that there is sufficient capacity to interconnect at the exchange. Cross-connects or switching fabrics can easily establish peering within a few hours or at most days. ISPs will prefer to interact where one or both ISP already has a presence.
4) Operations Issues
These issues focus on the ongoing operations activities allowed within the exchange after initial installation. Does the exchange allow private network interconnections? Are there requirements to connect to a central switch? How is access and security handled at the facility ? Is there sufficient power, HVAC, capacity at the switch, space for additional racks, real time staff support ? Is it easy to upgrade my presence over time? Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges over those with restrictions over future operations.
5) Business Issues
“Bandwidth, strategic partner alliances, and corporate ties often override the technical justification .” – Lauren Nowlin, (at the time, Peering Coordinator for Onyx Networks.)
Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)? A neutrally operated exchange (defined as one that is not owned or aligned with any carrier, fiber provider, or ISP) provides an open distortion-free marketplace for carrier and ISP services.
Market distortions often result when an exchange is owned by one of its participants. This often manifests itself in requirements (required use of their carrier or ISP services) that constrain the market for services within the exchange . Since it is difficult and disruptive to move equipment out of an exchange, ISPs will prefer a neutrally operated exchange environment that will not suffer from market distortions and limitations due to business conflicts of interest.
6) Cost Issues
This broad issue crosses all other issues. What is the cost of using this exchange? What are the rack fees, cross connect fees, port fees, installation fees? What are the future operating fees going to be? What are the motivations and parameters surrounding these fees? Cost issues shadow most of the other issues listed in this paper. All else being equal, ISPs will seek to minimize the costs, particularly upfront costs, associated with the interconnection for peering.
7) Credibility Issue
The credibility issue is twofold.
First, credibility goes to the financial support of the exchange. Does the exchange exist today and will it exist tomorrow? During the early stages of the exchange, ISPs are asked to make a leap of faith when committing, and therefore prefer an exchange with strong backing and the credibility to survive.

Figure 2 - Value of exchange varies over population
Second, does the exchange operator have the backing and credibility to attract the more valuable peering candidates? Since the value of the exchange (shown in the graph below) is proportional to the number and type of participants. Does this exchange have the backing to attract my peers? Who is managing the exchange and what technology is in use? These answer signal the credibility and survivability of the exchange. ISPs will prefer an exchange with credibility – one that is financially and technically well backed and likely to attract the most desirably peering candidates.
8) Exchange Population Issues
These issues focus on the side benefits to using this exchange. Are there other ISPs at this exchange that are peering candidates? Are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)? ISPs will prefer an established and well-populated exchange, particularly one with potential customers that can generate revenue.
9) Existing Exchange vs. New Exchange?
There are many operational exchange points in each region of the U.S. There are also emerging (soon to exist) exchanges that may be considered as peering points. However, given the pace of ISP expansion, it is unlikely that emerging exchange offerings are differentiated or compelling enough to be preferred over existing exchanges. Chronic traffic congestion can influence the decision to plan to peer in an existing malfunctioning exchange or wait until a better exchange opens. Customers with heavy flows of regional traffic can also influence the decision. Long term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one.
One Final Note on Exchange Criteria: Weighting
The ISPs we spoke with shared with us varied weightings of the importance of each of these issues. To some, the most important issues were the business issues, and others weighted more heavily the operations issues. Each ISP places higher or lower importance on different issues and not surprisingly select their operations environment based on their specific criteria.
We’ve done a ton of work with folks going through this process of selecting IXes, and it is wonderful observing the skills of the more senior peering coordinators. Much of the work has to do with traffic flows today and projected into the future, but more than that, it is their mastery along the dimensions of technical, business, legal and social that sets the seniors peering coordinators above the junior peering coordinators. By the way, this is precisely what the Peering Workshops do.

Dr Peering
DrPeering@DrPeering.net

Top 10 Reasons Why Inter-Provider QoS Hasn’t Taken Off

Tue, 11 May 2010 10:28:01 -0700

DrPeering -
Why hasn’t inter-provider QoS been widely adopted?
Hugh Priorty
----------------------
Hugh -
We asked your question of a few dozen Internet Operators at a North American Network Operators Group (NANOG) meeting.
Top 10 Reasons why inter-provider QoS hasn’t taken off
First, an academic acknowledged that in the 90’s there were hundreds of QoS papers in academia, and the flow has trickled to nothing. QoS has been described as a “dead topic” The topic has been discussed for over a decade, yet aside from VPNs, we don’t see QoS deployed or discussed for the public Internet. Why is that?
#1: Bigger Pipes are easier, faster to implement, and less complex
When congestion is reached across a monitored ISP network, ISPs can identify the bottlenecks and simply increase the capacity at these points. No quality of service queuing disciplines need to be defined, negotiated, managed, debugged, etc.
Getting the NOC trained to handle calls associated with a more complex system has costs associated with it.
#2: Chicken and Egg Challenge
A negotiated QoS solution needs to include all possible paths that the traffic may take in order to provide assurances that a constant bit rate will be available between all end points. Given that there are many thousands of ISPs collectively operating the Internet today, it will take time and be challenging to get buy-in from enough ISPs to make the claim of cross AS guarantees.
#3: QoS doesn’t scale well
It is impossible to guarantee that spot events won’t overwhelm a QoS system. For example, imagine 1 million end points trying to access an unpredictable combination of 10K content sites. The end result will most likely be a locked up network satisfying a small fraction of the customers that might otherwise be served (albeit served unreliably). An adaptive application solution in this example would scale better.
Religious argument: “Core routers should shift bits, not make complicated (per-user:per-flow) QoS decisions.”
#4: I’m not going to hand over the keys to my network traffic engineering to my competitors.
Several concerns here were raised by ISPs:
1) “Bandwidth reservation may break my network.”
1) “Don’t want my competitors to know how my secret (traffic engineering) sauce works.”
2) “I don’t want it known that I over subscribe my network, and/or route traffic strategically (circuitously) to meet peering ratios.”
Generally, even transparency about treatment of packets in my network is an issue. “It exposes parts of my business practices. For political reasons I may not want the government or broader community to know that. Competitors may use this information against you. “
#5: Show me the business case that shows that $1 invested in QoS yields more than $1 in profit
Paul Mockepetris said that “QoS is irrelevant in the absence of price”, meaning a QoS business model needs to motivate someone to prioritize packets.
A friend from Stanford said that the Internet in the U.S. is at an economic impasse – “there is no incentive for dollars to flow to where the bottlenecks are.”
There needs to be data demonstrating how much more revenue would be generated by QoS based improvements.
When expressed by last-mile ISPs: “This argument sounds like ‘my last mile business model sucks, so Content Providers and customers: pay us more to make it not suck.’” said one tier-2 ISP.
The other concern expressed is that with QoS, there is the opportunity to charge both customers and content providers a little bit more. Once you charge a little, it becomes easier to charge a lot for access to this captive customer base. This is one issue at the heart of the net neutrality discussion...
#6: QoS is Packet Prioritization is Anti-Net Neutrality
There is a perception that prioritization of traffic in any way is anti-net neutrality. However, this is not necessarily the case.
Is it anti-net neutrality if best-effort and premium services are separated across different fibers? Across different VLANs? How about a soft prioritization that looks and feels like these two ? Perhaps the excess bandwidth added into the now larger system can be made available to the best-effort service when it is not in use by the high QoS services? Best effort can actually benefit from QoS services.
#7: Difficulty in agreeing on QoS specifics (QoS markings)
This is similar to the chicken and egg problem, but the focus is on the challenge getting all the ISPs in the system to agree on the markings and meanings, and have a framework in place to make sure it all works.
#8: Difficulty in developing trust models between competitors.
What prevents a participant from sending all of its traffic high priority?
What if clever engineers figure a way to manipulate the system:
“Show me a bi-directionally metered Internet peering service and I’ll show you a money machine that will make me money no matter what.”
#9: QoS is only relevant when congestion is encountered along the path.
Qwest used the bandwidth upgrade policy when traffic hit 40% peak usage. Level 3 claims to have never dropped a packet.
“Traffic between ISPs in the core are relatively uncongested.” Almost all people said that the issue, the bottleneck, is at the edge.
To them, inter-provider is not the problem. Markings carried across the whole Internet is not critical.
Does this imply that the only relevant parties for QoS are the last mile providers and the content companies (or the CDNs)?
#10: Paid Peering is working
Paid peering between the last mile and content guys is working today. And, a customer relationship with Comcast for example provides a greater attention to problems in the path.

Summary
So there you have it Hugh - the top ten reasons inter-provider QoS hasn’t taken off.

Dr Peering
DrPeering@DrPeering.net
Notes from the Field:
Definition: Negotiated inter-provider Quality of Service (QoS) is a relationship between two ISPs where they “tag” packets with markings to indicate a handling preferences; the other ISP in the relationship honors these markings by treating them in a pre-defined pre-negotiated manner. This might be accomplished by mechanisms described in RFC2474 (diffserve), “tagging” packets with bits indicating a queuing discipline across organizational boundaries.

Approximation for 95th percentile

Fri, 23 Apr 2010 09:32:41 -0700

DrPeering -
Is there a way to guesstimate 95th percentile from average utilization?
Andy Bernard
-----------------------------
Mr. Bernard -
Most wholesale transit (and some CDN) services are metered on the 95th percentile. The typical measurement method is to take 5-minute samples across a month, rank the deltas from lowest to highest, and pick the 95th percentile measurement to calculate the monthly transit fee. This method allows the customer to burst the top 5% without paying for it.
I have spoken with hundreds of network engineers over the decade about traffic management and one observation comes up time and again:
The 95th percentile rule of thumb: Internet traffic tends to follow a sinusoidal curve such that 95th percentile is about 2 to 2.5 times the average.
Caveat: Of course there are a variety of traffic patterns that can skew the 95th percentile # higher or lower. For example, one scrappy Tier 2 ISP described a scenario where she would burst for less than 5% of the month and the rest of the month she would simply send/receive traffic to her commit rate. That is one extreme, where the 95th percentile tends towards the lower end of the traffic utilization spectrum. The other extreme is when the pipe is always saturated, so 95%=Average.
For most folks, this 95th percentile rule of thumb won't be too far off.

Dr Peering
DrPeering@DrPeering.net

Emerging Markets for Peering?

Mon, 19 Apr 2010 18:20:40 -0700

Dr Peering -
I see that Hurricane Electric advertises $1.50/Mbps for Internet transit now. The price of transit is approaching $1/Mbps in some markets for significant commitments. The rumors are that Google, Microsoft, and others are paying towards 25 cents/Mbps with large commits, and that these guys are peering away towards 75% of their traffic for free.
My question is....At $1/Mbps or less, why bother peering? Where if anywhere does peering make sense?
Serge Valero
------------------
Serge -
The price of transit and cost of peering are key metrics that prudent network architects and planners watch. Why?
Most ISPs and large content providers purchase Internet Transit services. Then they look to peering to lower the aggregate price per Mbps by blending in some free peering for as much of the peerable traffic as possible. The more you can peer away for free, the lower the cost per bit transferred.
As you can see below, when the unit cost of peering (shown in yellow) is higher than the metered price of transit (shown in red) peering doesn’t make sense financially. Where the lines cross is where the cost of peering exactly equals the price of transit. This is called the “Peering Breakeven Point” ; it is the point where one is indifferent between peering and transit for offloading traffic.
Stated another way, for peering to make sense financially, one has to peer away enough traffic for free to offset the cost of peering. Beyond that is gravy.

Notification: The 111 8th Street Lesson

Mon, 12 Apr 2010 10:00:44 -0700

Hey DrPeering -
What Internet Exchange Point has the best redundancy and lowest downtime?
Sal Petersen
----------------
Sal -
All IXes are constructed with some degree of infrastructure and network redundancy.
DrPeering discussed IX redundancy needs with the ISP community and learned something interesting: that they wanted reliability certainly, but most of them also built their networks to withstand breakdowns in the peering fabrics. Many of them preferred a less expensive IX with less redundancy, preferring instead to provide the redundancy themselves ! What these folks wanted was not so much a robust multi-tiered redundant infrastructure, but a lightweight resilient IX with responsive Ops and most importantly, notification when things did go awry.
The 111 8th Street building outage illustrates the three rules of IX outages:
1) things break,
2) everyone recognizes rule #1,
3) how the IX keeps IX participants informed is critically important to the participants.
This 111 8th street story focuses on the extraordinary chain of events that led to a 24 hour+ outage. I share this story with the hope that we learn from it the lesson that notification and continual updates matter a lot to this community. George Santayana, who, in his Reason in Common Sense, The Life of Reason, Vol.1, wrote "Those who cannot remember the past are condemned to repeat it." And in fact the Internet operations community has repeated the failure of notification on a continual basis. Here are the broad strokes of the 111 8th street notification story shared with DrPeering anonymously ...
The 111 8th Street Outage
During the New York City blackout a few years back, the 111 8th street building (housing many major telecommunications company facilities) lost power. As designed, the UPS took over the building load and the generators on the roof kicked on.
On the roof of 111 8th street are a handful of generators fueled by a relatively small (500 gallon) diesel tank, refilled by a powerful below-ground fuel pump attached to a couple of 50,000 gallon diesel tanks safely stowed under the building. The system was designed so that when the automatic transfer switch kicked the building from city power over to the UPS and generator system, the fuel pump would also kick on to continually top off the roof top diesel tank. And here is where the first failure occurred.
When installed, the fuel pump was tested, and sure enough, it turned on and seemed to start pumping fuel. The problem was that the polarity was reversed on the pump, so instead of pushing fuel from the underground tanks to the roof tank, it was accidentally rigged to pull fuel from the roof down to the underground tanks! When the city power was cut, the underground fuel pump indeed powered on and made noise, but did exactly the opposite of what it was supposed to do.
During the blackout, all seemed fine at the Internet Exchange point (which we will call X from here on) until the first generator on the roof stopped running, and then the second, and then the third. The tenants of the building that didn’t have their own generators and fuel were told there was a problem with the building power system. Our IX “X” notified their participants at this point that there was a power problem and the building owner was checking it out.
Another generator kicked off. Only two generators were now left operating on the roof.
Internet Exchange Point Operator X sent a note to their customers suggesting that the building was down to its last generators and that customers should shutdown all unnecessary gear in the 111 8th street Internet Exchange Point Operator X facility.
So far, so good.
The building electricians figured out that the roof tank was empty and the underground fuel pump was indeed running but pulling fuel down from the roof. He tried the polarity reversal test and the roof tank started filling up.
The roof tank filled up. But the generators were not starting up. Why?
During the hours that all of this was taking place, the generator starter motors were continually trying to restart the generators. The starter motors burned out! It took many hours to get the replacement starters delivered through the partying streets of New York City.
Once the new starters were installed, the generators cranked but the generators would not start. Why?
At the bottom of the 500 gallon roof tank was diesel sludge that had clogged the fuel pumps that fed the generators. It took many more hours to get the replacement fuel pumps delivered through the partying streets of New York City.
Once the new fuel pumps were installed (and the sludge cleaned out of the fuel tank and fuel lines), the generators started up.
This all took many hours to unfold. And there were zero notifications sent to customers after the IX lost power - radio silence for over 24 hours of downtime.
During the outage, the building folks were probably scrambling to diagnose and fix the problem and perhaps not updating the tenants that depended on the building for power. Perhaps the building folks updated Internet Exchange Point Operator X periodically. We do not know. But what we do know is that Internet Exchange Point Operator X did not keep its customers updated during the outage, so the customers did not know if their equipment would be powering back up in a few minutes, hours, or days, and could not prepare. ISPs told DrPeering that they can understand that failures occur, but there were no updates for an outage that ultimately lasted over 24 hours.
Lessons we should learn
1) I anonymized Internet Exchange Point Operator X in this scenario because this could have happened to any data center operator where power is out of their control,. Indeed unanticipated failure scenarios happen to everyone in the data center space. What we should learn is that what seems to matter most to the participants is how the company responds to these events.
2) The part that was under Internet Exchange Point Operator X’s control was the notification and update, and in this scenario, they failed completely. The ISPs interviewed were very understanding about the outage itself, but were very upset about the notification failure.
3) The way the system failed demonstrates cascading failures. It was the sequence of power outage, followed by fuel pump polarity that led to the starter motor restart and the fuel pump clogging issues. If the fuel pump didn’t empty the roof tank, the other two failures (roof fuel tank sludge clogging the fuel pump, starter burning out) may not have occurred, at least not then. It was the sequence of failures that made this an extended physical plant outage.
4) An N+1 electrical and mechanical system redundancy did not help.
5) Testing of course was broken here.

Dr Peering
DrPeering@DrPeering.net

The Ideal Peering Forum (Part 2 of 2)

Wed, 17 Mar 2010 08:46:48 -0700

continuing the thread on the “Ideal Peering Forum” experience...
Goals of the Ideal Peering Forum
The primary goal of the ideal peering forum is to facilitate peering. The ideal peering Forum pulls together the “right” people into the room to establish, build, and maintain relationships conducive to increased peering between the attendee organizations.
There are also critical secondary goals. Chief among these goals is to increase the effectiveness of internet operations for this peering community. It is a goal that these issues are raised in this forum, and that we collectively highlight emerging trends and document best practices associated with the peering coordinator role. The forum assembles individuals aligned with these mutual interests and those willing to contribute their insights, and tries to pull them together into a cohesive community.
Another secondary goal is to ensure that the forum meets the needs of the attendees and their organization so they return to attend the next one.
It is for these Peering Forum goals, and from experiences at some of the best conferences all over the world that this paper tries to capture the best practices.
Attendee Walks Into the Ideal Peering Forum
When the attendee walks up to the Peering Forum Registration Desk he/she is greeted by name (a printed color peering yearbook with photo and name are taped to the table to help the registration staff), and are handed
• A very cool laptop computer bag, with logos of the sponsors of the event, with the name, location and dates of the forum stitched prominently on the bag. (Attendees get so many cheap bags that get stacked in a closet or given away, any viral marketing value of those bags is lost. This bag is intended to replace whatever bag they currently use, will become the bag of choice, and will show up at venues all over the world, making friends and family jealous and sure to try and get an invite to the next Peering Forum.) -- this idea based on Josh Snowhorn killer laptop bag given at a peering event
Inside the bag is
• A SIP phone, preconfigured with all attendees SIP phone #s so they can meet each other. This will facilitate the “I need to find Person X from ISP Y, but I have never met her.” Problem. This also is to demonstrate new technology in an open environment, and as a take home phone, can be configured for use at one’s home network. This phone during to forum is configured to allow the attendees to call home for free using the Internet and SIP gateways in each country that an attendee comes from. -- this idea from the APRICOT where Gaurab made a deal with a wifi phone mfg and preconfigured them with the SIP conference gear.
The attendee is welcomed and handed a Nametag with
1. The Name of the Person prominently displayed
2. the company and AS# prominently displayed
3. a clip-on and/or safety pin to hold the nametag to the shirt. The natural tendency is for people to position the nametag high on the chest, conducive to being read without detection by the attendee. (Note: lanyards are NOT handed out here; they hold the badge in a position below the dining table when seated, and force the eyes to look way down during casual meetings, leading the person to realize that you don’t remember their name.)
4. All nametag info is printed on both sides since lanyards (if an attendee insists on wearing them) often flip around leaving people to walk around with blank nametags around their necks! If no lanyards are used, the ideal forum can use the nametag holder to stow an agenda, meal tickets, etc.
More Giveaways
The laptop bag described earlier is packed with goodies including a logo’d largest possible memory USB sticks/drives. The USB memory stick contains, to the degree possible, the draft slides, associated white papers, the agenda and attendee lists in the form of a yearbook with photos, a couple of books on tape and eBooks for the flight home, etc. all in electronic form. In the evening social and the next morning, attendees can swap their USB pen for a new one that has all the material the original had but upgraded with an MP3/MP4 of the previous days presentations. That way, if there was buzz about a particular talk that was missed because discussions pulled an attendee out of the room, they can listen to the talk and review the slides on the plane ride home. When technology improves, the USB drives will have the previous days videos along with photos/video of the social events for perusal on the flight home, or for highlighting to the folks who approve the travel the value of the event. Since many people need to assemble a trip report, and/or presentation sharing the experience with the teams at home, having the materials handy helps them drag and drop the relevant materials onto an internal server or into a document with links to the sources on the server. The USB pen data will be erased eventually, but the logo on the USB drive will stay, providing the viral marketing benefit of the sponsor.
• Business Card carrying case preloaded with the business cards of the people organizing the event, perhaps with a personalized note to the speakers thanking them for agreeing to speak.
• Location Appropriate Goodies. For example, the NOTA 2005 forum handed out an additional beach bag with a large NOTA beach towel and hat. One could imagine since this NOTA Forum included a cruise to the Bahamas, that the bag might have also contained sun screen, sun glasses, aloe, some gambling chips from the cruise line to get the attendee started, etc. For the Equinix Ashburn tours in the winter in Virginia, a bus picked up the attendees, handing them a very nice black fleece logo’d jacket to keep them warm for the trip and while waiting outside for the bus to return. For Chicago in the summer, the giveaways included a Pizza Peering theme T-shirt, highlighting many types of peers as toppings on a pizza menu. These themes help reinforce the theme of the conference, and if done well, these giveaways represent a personal benefit to the attendee that decides (and successfully argues that they ought) to attend the forum.
• Quality Polo, Tshirt, long sleeve shirt as appropriate for the climate. Here again, quality is essential ($30 each in quantity). We like the NZNOG Polo shirt made of honeycombed mesh cotton as an example. It is lightweight, packs small, and doesn’t get wrinkled and is likely to get worn. -- Idea from Jonny Martin who did this for NZNOG
• Pad of 8.5 x 11 paper with a solid thick cardboard backing so that it can be taken out and written on while standing up. This can be logo’d with the event, dates, location. Waterman Pens engraved of course for writing, perhaps supplemented by LiveScribe smart pens and notebooks for electronic storage of meeting notes.
While perhaps cost prohibitive, the idea of including logo’d and engraved macbook airs in the laptop bag, preloaded with software, training material, video tutorials on various topics, welcome messages from the organizers, product spreadsheets, presentations, large video files, etc. is particularly appealing and will certainly increase the buzz for the following meeting. It is also a great way to incorporate social software to facilitate the meetings of folks interested in various things - kind of like an electronic bulletin board showing the ad hoc meetings, dinners, socials, etc. Having the software pre-loaded and configured removes some logistics hurdles.
Are these Bribes? Yes, there is an important word of caution here. Whenever the employee benefits (like giveaways at a forum) and there is an expense to the employer (flight to the forum, hotel, taxis, food, etc.) there may be a justifiable concern that all of this represents a bribe intended to curry favor with the attendees. A few people for example decided to attend the NOTA Forum during the day, but did not go on the sponsored cruise/social to the Bahamas. Some companies are more sensitive to this than others and may error on the side of caution to avoid the perception of impropriety. It is a difficult line to draw.
Giveaway Summary – all of these giveaways start the forum off to the right start. People will dig through their bags, discovering the goodies, talking about what is on the USB sticks, etc. Logistically bundling these things together in a bag speeds the registration process. The viral marketing accomplished by the bags, the USB sticks, etc. are proportional to the amount of time these things are seen by others. Others should become envious that they missed the forum and will try and curry favor with the organizers to make sure they get an invitation to the next Ideal Peering Forum. We want the attendees to recall in their discussions with other peers the Ideal Peering Forum event location, dates or at least year, and the best way to do this is with great usable giveaways. Peering Coordinators often recall the presentations and the event by the location or the social event, things memorable at the forum. This is something that can be socially engineered.
The Food Service at the Ideal Peering Forum
When you walk toward the room you notice a massive food and beverage selection on the table. The ideal Peering Forum has drinks available all day, not only at selected breaks. This forum is socially engineered to have the expected ad hoc breaks to have a discussion outside. We are trying to facilitate these interactions, and the break areas are the place to do this. In a variety of geographically dispersed break areas will be
• Coffee, variety of teas and chai, espresso drinks, all available in decaffeinated form.
• Fresh squeezed fruit juices
• Fresh vegetable drinks
• Bottled water and varieties of soda
• Healthy snacks as well as indulgent ones
Why focus on food and drink? At conferences, attendees are asked to wake up early (depending on timezone of origin) to drink up coffee and eat sugared sweets, and are then expected to sit still and listen actively to presentations. This works for a short while until the sugar crash occurs; the point is, these foods and drinks are universally delivered at conferences but are not conducive to the expected behavior. Therefore we need to provide and promote foods that help keep folks awake -- perhaps even educate the audience as to the benefits of longer tern sustainable energy foods and why the forum will make sure they are available.
This issue is particularly important in the afternoon. After lunch conference attendees suffer from ‘food-coma’, a sleep-like hypnotic effect that renders all but the most energized presentations incomprehensible. Heavy snacks for afternoon breaks such as brownies and ice cream have a similar negative effect when you want an attentive audience. Managing food is an easy way to indirectly improve the forum experience.
Also note that the first days of a conference have a more attentive audience, while later days the attention span drifts off. The number of attendees decreases over time as well, so a generally larger and more attentive audience is found in the early part of the first days.
Room Layout of the Ideal Peering Forum
The Ideal Peering Forum is held in a theatre-style venue with each row of seats descending down to a stage where the presenters present. George Washington University has such a theater and worked quite well for NANOG. Such accommodations are rarely available, so we will focus on the best practices assuming that the ideal peering forum is held in a hotel.
Classroom style with tables is used to facilitate note taking and use of laptop computers. There are two screens set as high up as practicable so all seats have an unobstructed view of the entire screen.
Each seat has access to power and hard wire Internet access via Ethernet plug for laptop users. Wireless may be substituted.
Video monitors on the floor in front of the speakers allowing them to always face forward, and to avoid them turning their heads to the side to view the slides on the screen which points their head away from the clip on microphones. The second monitor shows a countdown timer so the speaker knows how much time is left in their slot. The background color coding is Green until 5 minutes are left, Yellow until 2 minutes are left, and Red when 2 minutes or less are left. -- These best practices come from Next Generation Network conferences
There is a ‘green room’ where the speakers are wired such that the wiring is not easily viewable by the audience, nor is the microphone likely to be accidentally pulled out.
Each speaker has a bottle of water available on the table next to the podium should they need one during their talk. Each session should have a case of bottled water available so no one is ever looking around for bottles of water if a room monitor happens to be absent.
All (draft versions of the) talks are preloaded onto the presentation laptop, but presenters are allowed to update their preso up to the break before their presentation begins. In some case they may use their own laptop, but this is minimized to keep the speaker flow smooth and quick. The preferred method of updating slides is USB copying to the presentation machine with a quick walk through to make sure the transitions and animations etc. work as expected. This walkthrough detects the situation where a presenters newly linked file is absent from the shared laptop.
A transition title slide exists for each speaker showing a photo of the speaker and the title of their talk, with a URL linking to the presentation. This is especially helpful for the attendees viewing this cover slide as a thumbnail mode on their computers.
The laptops the audience uses have the ability to automatically follow along with the slides being presented by the speaker, with a notes page on the side for taking notes during the presentation. A forum wide instant message chat room is used for audience members to chat and ask questions during the presentation. A moderator will make sure this doesn’t get out of hand.
Agenda Selection
By far the most important preparation step of a successful Ideal Peering Forum is the agenda selection and its ability to pull together the right people into the room. To this end, select speakers who;
• Are social centers – they will bring/encourage others in the community to attend
• Are good speakers and have given talks in the past of interest to this crowd. Their talk title, their name, their contacts, their reputation will help attendees justify attending the conference. Think of the attendee asking their boss to attend: ‘What will you and the company get out of it?’
• Bring some topical material of interest to the crowd. For example, Todd Underwood from Renesys gave a last minute talk at NANOG about the recent level 3 – cogent de-peering. This is an area of interest to this crowd and was especially topical – it generated over 150 messages on the NANOG mailing list!
• Interactive Talks – find talks that have audience involvement. Examples: The Peering Simulation Game is a live simulation that brings audience members on stage to play the game. This successfully draws the audience in, and an on-line version would involve them even further. Another example is Peering Personals – a chance for the audience members to make a 2 minute ad hoc introduction of themselves. These are more dynamic, more unpredictable and more memorable than canned and approved presentations normally brought to these conferences.
Breaks and Socials
The breaks and socials are the most important times to accomplish the primary goals of the peering forum. The ideal peering Forum is careful to keep speakers on topic and on time, with the more controversial and those that inspire follow up conversations occurring just before the breaks and socials. Channeling these heated discussions and debates is a primary goal of the peering forum organizer.
Survey Forms
Surveys are on-line and hard copies are placed on each seat in the room at the breaks before the speakers for that session will speak. Each section of speakers has its own sections on the survey form so people who need to leave early are filling out the forms for the talks they heard. Some enticement is used to get survey forms filled out such as a raffle for an iPod, noise reduction headphones, PDA, etc. something that helps those who travel a lot.

Location, Access
The hotel is be reachable via a max of two plane segments from most origins, and be easy to get to from the airport. Directions and options for housing are listed on the event web site, complete with price points and pros and cons for each hotel based on personal experience. If possible, limos should be arranged from the airports to the venue for people traveling on the same plane together - start the social side early.
Social
Great care should be taken to select a social that accomplishes the following goals:
1. Encourages the interaction between the attendees
2. highlights an aspect of the location
3. is memorable – many times the event is remembered based on the social

We could go on - there are tons of execution ideas.
Consider these to be some ideas derived from some of the best practices of various operations forums around the world.

Dr Peering
DrPeering@DrPeering.net

The Ideal Peering Forum (Part 1 of 2)

Mon, 15 Mar 2010 11:14:49 -0700

Dear DrPeering -
Our Peering Ecosystem consists of several hundred Tier 2 ISPs and a few Tier 1 ISPs but we have never had a peering forum. We have a few exchange points but they don’t seem to be pulling together the ISP population. Do ISPs ever lead a Peering Forum?
Also - Will I see you at the Global Peering Forum cruise?
Greg Daniels
-------------------------------
Dear Greg -
Last question first... I am actively doing peering work for an ISP but, surprisingly, they seem reluctant to pay consulting rates for me to go on a cruise. Color me surprised. Not really.
The cruise venue has always been an issue with some companies - I guess it smells like a boondoggle, or a bribe, or worse (abuse of budget, collusion, pick your influence poison). Some folks can not attend for this reason, and this is unfortunate as it is simply a different venue.
I credit Josh Snowhorn (NOTA) with the first Peering Cruise. Josh made the discovery that hosting a peering event on a cruise was actually less expensive than hosting the event at a hotel (when you take into account catering, room nights, etc.) And the Peering Cruises for those who can attend have been very effective. I used to play down this “cruise” perception issue, until now, that I (sadly) can not attend.
Josh has also done a masterful job hosting, getting sponsors and creating great giveaways that people actually still use today, but that is another topic.
So back to your first question...
Peering Forums typically get started by an Internet Exchange Point (IX) that is looking to build a population of peers. By identifying a bunch of peers that can become members all at the same time, the IX can leverage their combined self-interests to build a critical mass of participants more quickly. In Europe, these seed ISPs might form a formal association and join en mass to bootstrap things and launch an IX. In either case, once enough participants are members of the IX, it is much easier to attract additional peers, and the peering forum helps achieve the ends of more peering for all. A real win-win-win.
But ISPs can start a peering forum, and might even be better positioned to do so. Why would an ISP start a Peering Forum?
1) Marketing Position. An ISP may seek to demonstrate leadership – in thought and in action. This messaging helps with customer acquisition, with regulators, and with popular opinion as the ecosystem grows and Internet prices drop. To be able to say “We have been taking a leadership role in evolving the peering landscape since 2008. Our goals are the same as the public goals – to evolve this ecosystem into a first class Internet ecosystem, with low prices, low upstream costs, and massive penetration of broadband Internet services” is powerful in emerging ecosystems like yours. Not only for goodwill; most people prefer to do business with a leader.
2) Regulatory. Since most governments prefer industry self-regulation, a peering leader has a bit more clout with the government, especially when guidance, information, or assistance / influence is desired. As issues such as network neutrality or de-peering disputes arise for example, key players may be consulted while followers and lurkers may not.
1) Market Research. Running a peering forum may actually be more effective and less expensive than paying for market research. The chart below shows the frequency and rough costs of running a peering forum twice a year, compared with the cost of purchasing a market research report every other year.
I would argue that one is more likely to hear timely information about happenings, and more likely to obtain the right answers to important marketing questions as a side effect of running the peering forums. Since you are in the industry you may be better positioned than an outside organizations to collect information directly from the field and running the forum gives you an excuse to talk with these folks in the field. You can also ask and evolve the queries to the right people at the right time, and change those questions over time. By running the peering forum, you stay in the loop by keeping communications channels open, collecting information before, during, and after the events that you run. Very valuable.
One final point that must be made - like IXes, Peering Forum demonstrate the characteristic that economists call “Network Externality” - that the value of the peering forum is proportional to the number and value of those attending. For IXes this effect is shown in the figure below.

Opportunity. Since you said your ecosystem doesn’t really have peering forums today, you have the opportunity build a peering forum to critical mass with no competitive peering forum in place.
Do it. It may be a challenge to start it up, but it is far more difficult to dislodge a competitive peering forum that is already at critical mass. Once you have critical mass, your peering forum can be very valuable to you and the population and be difficult to dislodge if you keep people happy. After all, there is a limited time and travel budget to attend meetings of this kind.
Next I will meditate on “The Ideal Peering Forum”...... Ummmm......

Dr Peering
DrPeering@DrPeering.net