The Peering Advice Column

Par for Peering

Mon, 12 Jul 2010 18:13:46 -0700

DrPeering -
Peering is now our strategy... How much traffic can a content heavy network realistically peer away?
Cheryl Will
AS XXXXX

Cheryl -
Thanks for the question.
Diagnosis: Peering success will depend on a variety of factors such as:
1) traffic profile f(volume, dest, type, growth, IX presences, etc),
2) peering coordinator skill and approach,
3) timing and peering inclination trends in the target peer community, etc.
but the window of peering opportunity in the U.S. has been closing a bit.
I used to tell folks that they could expect maybe 40% of their traffic to be easily peerable, and the remainder gets exponentially more difficult to peer away. Recent discussions with large scale peering folks however reveal that it is harder to peer away that first 40% than it used to be. They point to the rising peering requirements of the access networks, and perhaps their predilection toward paid peering instead of free peering. They point to the Arbor Report which shows fewer sources and sinks are responsible for a larger and larger chunks of traffic. For whatever reason, peering with the access networks is harder and probably more important. Given that you are content heavy, it may be more difficult than it used to be to get to 40% free peering.

I would estimate peering par today to be closer to 25% - expect that a good peering coordinator should be able to easily peer away about 25% of your traffic at a couple IXes.

Prescription: I would advise that, if you need peering for performance or marketing strategy reasons, you should get in now before it gets harder. Peering tends to stay in place even as peering policies change. There will be some companies that de-peer because they adopted more stringent policies over time, but this is not the norm. Historically, legacy peering tends to stick around. Because of this stickiness, back in the boom years, companies were even valued (in part) on what peering they had! Some ISPs even purchased companies with the expectation that the Tier 1 peering sessions would survive the acquisition.
So,
1) there is value in peering,
2) the window of opportunity is closing,
3) peering sessions tend to stick around even as policies for new peer changes.
If peering is a strategic intent, you should get in now.
Comments: The other thing I can suggest to you is to send in your top 50 target peers. We have seen remarkable similarity in peering profiles across companies in the same sector. We can predict who will peer, who will not, how to approach, etc. from the list and provide you with some feedback/suggestions. This is often done to help new ISPs select the best Internet Exchanges for their profile. We have done hundreds of peering introductions over the years and would be glad to help you with this.
Always track the price of transit as well - you might find over time that your split of traffic rationally changes over time. Lots of tools like paid peering and inexpensive high quality CDNs are available now and should be in your tool bag.
Hope this helps -

Dr Peering
DrPeering@DrPeering.net

U.S. Internet Transit Pricing - Historical and Projected

Thu, 10 Jun 2010 10:36:00 -0700

DrPeering -
What is transit going for in the U.S. ? At what point do peering not make sense anymore?
Cat D’etat
-------
Cat - I will share some data from an early draft report: Internet Transit Pricing - Historical and Projected along with one prognosis for the peering industry.
First, here is are the DrPeering historical and projected transit prices for the next few years in the U.S.:
When does Internet Peering Make Sense?

What does this table show? Transit Prices are dropping, and the range where public peering makes sense is getting narrower over time.
If you assume the cost of public peering is $10,000/month ($2500 for a 10G public peering port and $7500 for colo, power, transport, equipment, etc.), you see that in 2010, peering makes sense financially if you can peer away at least 2Gbps (2000Mbps) for free to your peers. This gives you plenty of breathing room to peer traffic away for less than the price of transit, up to 7.6Gbps (7600Mbps), which is approximately the amount of traffic one can reasonably sustain over a 10G peering port. But this is for 2010 with an average transit price of $5/Mbps.
With these pricing estimates we see that in 2012, peering still makes sense, but with an average Internet Transit price of $2.34/Mbps, we see a peering break even point of 4274Mbps where peering begins to make sense. Beyond this point, peering makes sense financially up to 7.6Gbps, the capacity of the pipe.
This is not much of a range; getting peering traffic volumes to fit in this range is tricky, so peering folks will mostly likely not go forward with public peering (all things remaining equal).
As fewer and fewer ISPs publicly peer, the rationale for public peering to aggregate smaller peering sessions together over a public peering fabric diminishes. And the technology is not well suited today to handle massive peering sessions over public peering fabrics.
As a result, Private Peering will dominate the peering ecosystem in the U.S. by 2012. Point to point fiber is just too cost effective, reliable, and ease to deploy to compete with.
It is certainly possible that the pricing of IX ports will drop, but the U.S. has been slow to do that compared with the European IXes who have consistently led the charge in dropping peering port fees.
Take a look and I would love to hear your feedback on the early draft of the report:
http://peering.drpeering.net/white-papers/Internet-Transit-Prices-Historical-And-Projected.html
There are some other predictions in the report
1) The U.S. Peering Sector will refocus on Video Distribution Companies, High Performance Network Services Sector, while maintaining customers that simply stick around,
2) ISPs and CDNs will morph into higher value service providers or die,
3) Access Networks grow in power in the Peering Ecosystem, and
4) Cooperative inter-AS services may bridge the gap.
All of these predictions are of course 100% guaranteed to come true or your RSS feed is free!

Dr Peering
DrPeering@DrPeering.net
P.S. - Several have asked for a 100G Public Peering Port break even analysis. We will do this in the next article.

Pursue a Paid Peering Strategy?

Thu, 3 Jun 2010 10:04:50 -0700

DrPeering
We are a large access network in Asia. Is it possible to forge relationships with content providers so they pay to peer with us? What should be my strategy?

regards
Tan

-------
Tan - Yes.

At the heart of Paid Peering is the power relationship. Understand your power position.

In every Internet Peering Ecosystem we studied, Access Network customers are typically singly homed. The Access Network (you) connect many “captive” customers, customers who are reachable only through you. The Content Providers that wish to send their content to your customers have no alternative ; to get their content to these eyeballs they have to send it directly (by peering) or indirectly (through transit) through you. Since peering will be better performance, it is reasonable that some might pay extra for the privilege.
The Content Providers on the other hand are typically multi-homed and provide access to only one customer - themselves ! This is the key aspect to this power asymmetry; they need you more than you need them.
So the content provider question is:
“Is it worth money to get my content closer to the eyeballs? “
If you follow the lead of Comcast and price Paid Peering similar to Internet Transit pricing, you can enable the Content Provider better performance and extract a little revenue in the process.
There are a few issues to consider of course. Among other things, you will need to develop the internal capabilities to support these paid peers as customers. You will need to develop a Peering Policy that articulates your peering inclination; we have done several peering workshops with companies to do this, bringing their multi-disciplinary Peering Steering Committee up to speed. (You don’t need a workshop to do this by the way - just about all of the material we would use to bring your team up to speed is on the DrPeering.net web site. )
So yes, you do have the ability.
It has always been an interesting debate in the Peering Community about whether access networks should charge content providers to deliver content your customers requested.

Dr Peering
DrPeering@DrPeering.net

Top 9 Internet Exchange Selection Criteria

Tue, 18 May 2010 09:54:21 -0700

DrPeering -

We are building out our peering infrastructure....How do people select an IX?

Razi

---------
Razi - thanks for the note.
This was one of the first questions asked of the peering community and documented in “Internet Service Providers and Peering”:
How do you choose an IX?
What we learned from the hundreds of conversations was that there are essentially 9 Internet Exchange Point Selection Criteria as shown in the flow chart below.

Top 9 Internet Exchange Point Selection Criteria

Figure 1 - Exchange Environment Selection

We will talk briefly through each of the top 9 IX selection criteria.
1) Telecommunications Access Issues
These issues have to do with getting telecommunications services into the exchange. How fast can circuits be brought into the interconnection environment? How many carriers compete for business for circuits back to my local Point of Presence (POP)? For facilities-based ISPs, what is the cost of trenching into the exchange (how far away and what obstacles present themselves)? Are there nearby fiber providers that lease strands? These questions will help answer the most important question to ISPs: How fast can my peer and I get connectivity into the exchange? Multiple carriers lead to speed and cost efficiencies. Some ISPs have volume deals with certain carriers or otherwise prefer carriers and therefore prefer exchanges where these carriers can quickly provision circuits. These answers strongly impact the desirability of the exchange environment.
2) Deployment Issues
These issues have to do with getting equipment into the exchange. How do I get my equipment into the exchange (assuming it supports collocation)? Do I ship equipment in or do I have to bring it with me as I fly in? Will someone act as remote hands and eyes to get the equipment into the racks or do I do the installation myself? Comparing exchange environments in this context, what are the costs associated with deployment (travel, staff time, etc.) into this exchange? Does the exchange have sufficient space, power, air conditioning, etc. The answers to these questions impact the deployment schedule for the ISP(s) engineers and the costs of the interconnection method.
3) ISP Current Presences Issues
This issue is based on the following observation by the peering coordinators: The most inexpensive and expedient peering arrangements are the ones made between ISPs that are already located in the same exchange. There is a hidden assumption here that there is sufficient capacity to interconnect at the exchange. Cross-connects or switching fabrics can easily establish peering within a few hours or at most days. ISPs will prefer to interact where one or both ISP already has a presence.
4) Operations Issues
These issues focus on the ongoing operations activities allowed within the exchange after initial installation. Does the exchange allow private network interconnections? Are there requirements to connect to a central switch? How is access and security handled at the facility ? Is there sufficient power, HVAC, capacity at the switch, space for additional racks, real time staff support ? Is it easy to upgrade my presence over time? Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges over those with restrictions over future operations.
5) Business Issues
“Bandwidth, strategic partner alliances, and corporate ties often override the technical justification .” – Lauren Nowlin, (at the time, Peering Coordinator for Onyx Networks.)
Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)? A neutrally operated exchange (defined as one that is not owned or aligned with any carrier, fiber provider, or ISP) provides an open distortion-free marketplace for carrier and ISP services.
Market distortions often result when an exchange is owned by one of its participants. This often manifests itself in requirements (required use of their carrier or ISP services) that constrain the market for services within the exchange . Since it is difficult and disruptive to move equipment out of an exchange, ISPs will prefer a neutrally operated exchange environment that will not suffer from market distortions and limitations due to business conflicts of interest.
6) Cost Issues
This broad issue crosses all other issues. What is the cost of using this exchange? What are the rack fees, cross connect fees, port fees, installation fees? What are the future operating fees going to be? What are the motivations and parameters surrounding these fees? Cost issues shadow most of the other issues listed in this paper. All else being equal, ISPs will seek to minimize the costs, particularly upfront costs, associated with the interconnection for peering.
7) Credibility Issue
The credibility issue is twofold.
First, credibility goes to the financial support of the exchange. Does the exchange exist today and will it exist tomorrow? During the early stages of the exchange, ISPs are asked to make a leap of faith when committing, and therefore prefer an exchange with strong backing and the credibility to survive.

Figure 2 - Value of exchange varies over population
Second, does the exchange operator have the backing and credibility to attract the more valuable peering candidates? Since the value of the exchange (shown in the graph below) is proportional to the number and type of participants. Does this exchange have the backing to attract my peers? Who is managing the exchange and what technology is in use? These answer signal the credibility and survivability of the exchange. ISPs will prefer an exchange with credibility – one that is financially and technically well backed and likely to attract the most desirably peering candidates.
8) Exchange Population Issues
These issues focus on the side benefits to using this exchange. Are there other ISPs at this exchange that are peering candidates? Are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)? ISPs will prefer an established and well-populated exchange, particularly one with potential customers that can generate revenue.
9) Existing Exchange vs. New Exchange?
There are many operational exchange points in each region of the U.S. There are also emerging (soon to exist) exchanges that may be considered as peering points. However, given the pace of ISP expansion, it is unlikely that emerging exchange offerings are differentiated or compelling enough to be preferred over existing exchanges. Chronic traffic congestion can influence the decision to plan to peer in an existing malfunctioning exchange or wait until a better exchange opens. Customers with heavy flows of regional traffic can also influence the decision. Long term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one.
One Final Note on Exchange Criteria: Weighting
The ISPs we spoke with shared with us varied weightings of the importance of each of these issues. To some, the most important issues were the business issues, and others weighted more heavily the operations issues. Each ISP places higher or lower importance on different issues and not surprisingly select their operations environment based on their specific criteria.
We’ve done a ton of work with folks going through this process of selecting IXes, and it is wonderful observing the skills of the more senior peering coordinators. Much of the work has to do with traffic flows today and projected into the future, but more than that, it is their mastery along the dimensions of technical, business, legal and social that sets the seniors peering coordinators above the junior peering coordinators. By the way, this is precisely what the Peering Workshops do.

Dr Peering
DrPeering@DrPeering.net

Top 10 Reasons Why Inter-Provider QoS Hasn’t Taken Off

Tue, 11 May 2010 10:28:01 -0700

DrPeering -
Why hasn’t inter-provider QoS been widely adopted?
Hugh Priorty
----------------------
Hugh -
We asked your question of a few dozen Internet Operators at a North American Network Operators Group (NANOG) meeting.
Top 10 Reasons why inter-provider QoS hasn’t taken off
First, an academic acknowledged that in the 90’s there were hundreds of QoS papers in academia, and the flow has trickled to nothing. QoS has been described as a “dead topic” The topic has been discussed for over a decade, yet aside from VPNs, we don’t see QoS deployed or discussed for the public Internet. Why is that?
#1: Bigger Pipes are easier, faster to implement, and less complex
When congestion is reached across a monitored ISP network, ISPs can identify the bottlenecks and simply increase the capacity at these points. No quality of service queuing disciplines need to be defined, negotiated, managed, debugged, etc.
Getting the NOC trained to handle calls associated with a more complex system has costs associated with it.
#2: Chicken and Egg Challenge
A negotiated QoS solution needs to include all possible paths that the traffic may take in order to provide assurances that a constant bit rate will be available between all end points. Given that there are many thousands of ISPs collectively operating the Internet today, it will take time and be challenging to get buy-in from enough ISPs to make the claim of cross AS guarantees.
#3: QoS doesn’t scale well
It is impossible to guarantee that spot events won’t overwhelm a QoS system. For example, imagine 1 million end points trying to access an unpredictable combination of 10K content sites. The end result will most likely be a locked up network satisfying a small fraction of the customers that might otherwise be served (albeit served unreliably). An adaptive application solution in this example would scale better.
Religious argument: “Core routers should shift bits, not make complicated (per-user:per-flow) QoS decisions.”
#4: I’m not going to hand over the keys to my network traffic engineering to my competitors.
Several concerns here were raised by ISPs:
1) “Bandwidth reservation may break my network.”
1) “Don’t want my competitors to know how my secret (traffic engineering) sauce works.”
2) “I don’t want it known that I over subscribe my network, and/or route traffic strategically (circuitously) to meet peering ratios.”
Generally, even transparency about treatment of packets in my network is an issue. “It exposes parts of my business practices. For political reasons I may not want the government or broader community to know that. Competitors may use this information against you. “
#5: Show me the business case that shows that $1 invested in QoS yields more than $1 in profit
Paul Mockepetris said that “QoS is irrelevant in the absence of price”, meaning a QoS business model needs to motivate someone to prioritize packets.
A friend from Stanford said that the Internet in the U.S. is at an economic impasse – “there is no incentive for dollars to flow to where the bottlenecks are.”
There needs to be data demonstrating how much more revenue would be generated by QoS based improvements.
When expressed by last-mile ISPs: “This argument sounds like ‘my last mile business model sucks, so Content Providers and customers: pay us more to make it not suck.’” said one tier-2 ISP.
The other concern expressed is that with QoS, there is the opportunity to charge both customers and content providers a little bit more. Once you charge a little, it becomes easier to charge a lot for access to this captive customer base. This is one issue at the heart of the net neutrality discussion...
#6: QoS is Packet Prioritization is Anti-Net Neutrality
There is a perception that prioritization of traffic in any way is anti-net neutrality. However, this is not necessarily the case.
Is it anti-net neutrality if best-effort and premium services are separated across different fibers? Across different VLANs? How about a soft prioritization that looks and feels like these two ? Perhaps the excess bandwidth added into the now larger system can be made available to the best-effort service when it is not in use by the high QoS services? Best effort can actually benefit from QoS services.
#7: Difficulty in agreeing on QoS specifics (QoS markings)
This is similar to the chicken and egg problem, but the focus is on the challenge getting all the ISPs in the system to agree on the markings and meanings, and have a framework in place to make sure it all works.
#8: Difficulty in developing trust models between competitors.
What prevents a participant from sending all of its traffic high priority?
What if clever engineers figure a way to manipulate the system:
“Show me a bi-directionally metered Internet peering service and I’ll show you a money machine that will make me money no matter what.”
#9: QoS is only relevant when congestion is encountered along the path.
Qwest used the bandwidth upgrade policy when traffic hit 40% peak usage. Level 3 claims to have never dropped a packet.
“Traffic between ISPs in the core are relatively uncongested.” Almost all people said that the issue, the bottleneck, is at the edge.
To them, inter-provider is not the problem. Markings carried across the whole Internet is not critical.
Does this imply that the only relevant parties for QoS are the last mile providers and the content companies (or the CDNs)?
#10: Paid Peering is working
Paid peering between the last mile and content guys is working today. And, a customer relationship with Comcast for example provides a greater attention to problems in the path.

Summary
So there you have it Hugh - the top ten reasons inter-provider QoS hasn’t taken off.

Dr Peering
DrPeering@DrPeering.net
Notes from the Field:
Definition: Negotiated inter-provider Quality of Service (QoS) is a relationship between two ISPs where they “tag” packets with markings to indicate a handling preferences; the other ISP in the relationship honors these markings by treating them in a pre-defined pre-negotiated manner. This might be accomplished by mechanisms described in RFC2474 (diffserve), “tagging” packets with bits indicating a queuing discipline across organizational boundaries.

Approximation for 95th percentile

Fri, 23 Apr 2010 09:32:41 -0700

DrPeering -
Is there a way to guesstimate 95th percentile from average utilization?
Andy Bernard
-----------------------------
Mr. Bernard -
Most wholesale transit (and some CDN) services are metered on the 95th percentile. The typical measurement method is to take 5-minute samples across a month, rank the deltas from lowest to highest, and pick the 95th percentile measurement to calculate the monthly transit fee. This method allows the customer to burst the top 5% without paying for it.
I have spoken with hundreds of network engineers over the decade about traffic management and one observation comes up time and again:
The 95th percentile rule of thumb: Internet traffic tends to follow a sinusoidal curve such that 95th percentile is about 2 to 2.5 times the average.
Caveat: Of course there are a variety of traffic patterns that can skew the 95th percentile # higher or lower. For example, one scrappy Tier 2 ISP described a scenario where she would burst for less than 5% of the month and the rest of the month she would simply send/receive traffic to her commit rate. That is one extreme, where the 95th percentile tends towards the lower end of the traffic utilization spectrum. The other extreme is when the pipe is always saturated, so 95%=Average.
For most folks, this 95th percentile rule of thumb won't be too far off.

Dr Peering
DrPeering@DrPeering.net

Emerging Markets for Peering?

Mon, 19 Apr 2010 18:20:40 -0700

Dr Peering -
I see that Hurricane Electric advertises $1.50/Mbps for Internet transit now. The price of transit is approaching $1/Mbps in some markets for significant commitments. The rumors are that Google, Microsoft, and others are paying towards 25 cents/Mbps with large commits, and that these guys are peering away towards 75% of their traffic for free.
My question is....At $1/Mbps or less, why bother peering? Where if anywhere does peering make sense?
Serge Valero
------------------
Serge -
The price of transit and cost of peering are key metrics that prudent network architects and planners watch. Why?
Most ISPs and large content providers purchase Internet Transit services. Then they look to peering to lower the aggregate price per Mbps by blending in some free peering for as much of the peerable traffic as possible. The more you can peer away for free, the lower the cost per bit transferred.
As you can see below, when the unit cost of peering (shown in yellow) is higher than the metered price of transit (shown in red) peering doesn’t make sense financially. Where the lines cross is where the cost of peering exactly equals the price of transit. This is called the “Peering Breakeven Point” ; it is the point where one is indifferent between peering and transit for offloading traffic.
Stated another way, for peering to make sense financially, one has to peer away enough traffic for free to offset the cost of peering. Beyond that is gravy.

Notification: The 111 8th Street Lesson

Mon, 12 Apr 2010 10:00:44 -0700

Hey DrPeering -
What Internet Exchange Point has the best redundancy and lowest downtime?
Sal Petersen
----------------
Sal -
All IXes are constructed with some degree of infrastructure and network redundancy.
DrPeering discussed IX redundancy needs with the ISP community and learned something interesting: that they wanted reliability certainly, but most of them also built their networks to withstand breakdowns in the peering fabrics. Many of them preferred a less expensive IX with less redundancy, preferring instead to provide the redundancy themselves ! What these folks wanted was not so much a robust multi-tiered redundant infrastructure, but a lightweight resilient IX with responsive Ops and most importantly, notification when things did go awry.
The 111 8th Street building outage illustrates the three rules of IX outages:
1) things break,
2) everyone recognizes rule #1,
3) how the IX keeps IX participants informed is critically important to the participants.
This 111 8th street story focuses on the extraordinary chain of events that led to a 24 hour+ outage. I share this story with the hope that we learn from it the lesson that notification and continual updates matter a lot to this community. George Santayana, who, in his Reason in Common Sense, The Life of Reason, Vol.1, wrote "Those who cannot remember the past are condemned to repeat it." And in fact the Internet operations community has repeated the failure of notification on a continual basis. Here are the broad strokes of the 111 8th street notification story shared with DrPeering anonymously ...
The 111 8th Street Outage
During the New York City blackout a few years back, the 111 8th street building (housing many major telecommunications company facilities) lost power. As designed, the UPS took over the building load and the generators on the roof kicked on.
On the roof of 111 8th street are a handful of generators fueled by a relatively small (500 gallon) diesel tank, refilled by a powerful below-ground fuel pump attached to a couple of 50,000 gallon diesel tanks safely stowed under the building. The system was designed so that when the automatic transfer switch kicked the building from city power over to the UPS and generator system, the fuel pump would also kick on to continually top off the roof top diesel tank. And here is where the first failure occurred.
When installed, the fuel pump was tested, and sure enough, it turned on and seemed to start pumping fuel. The problem was that the polarity was reversed on the pump, so instead of pushing fuel from the underground tanks to the roof tank, it was accidentally rigged to pull fuel from the roof down to the underground tanks! When the city power was cut, the underground fuel pump indeed powered on and made noise, but did exactly the opposite of what it was supposed to do.
During the blackout, all seemed fine at the Internet Exchange point (which we will call X from here on) until the first generator on the roof stopped running, and then the second, and then the third. The tenants of the building that didn’t have their own generators and fuel were told there was a problem with the building power system. Our IX “X” notified their participants at this point that there was a power problem and the building owner was checking it out.
Another generator kicked off. Only two generators were now left operating on the roof.
Internet Exchange Point Operator X sent a note to their customers suggesting that the building was down to its last generators and that customers should shutdown all unnecessary gear in the 111 8th street Internet Exchange Point Operator X facility.
So far, so good.
The building electricians figured out that the roof tank was empty and the underground fuel pump was indeed running but pulling fuel down from the roof. He tried the polarity reversal test and the roof tank started filling up.
The roof tank filled up. But the generators were not starting up. Why?
During the hours that all of this was taking place, the generator starter motors were continually trying to restart the generators. The starter motors burned out! It took many hours to get the replacement starters delivered through the partying streets of New York City.
Once the new starters were installed, the generators cranked but the generators would not start. Why?
At the bottom of the 500 gallon roof tank was diesel sludge that had clogged the fuel pumps that fed the generators. It took many more hours to get the replacement fuel pumps delivered through the partying streets of New York City.
Once the new fuel pumps were installed (and the sludge cleaned out of the fuel tank and fuel lines), the generators started up.
This all took many hours to unfold. And there were zero notifications sent to customers after the IX lost power - radio silence for over 24 hours of downtime.
During the outage, the building folks were probably scrambling to diagnose and fix the problem and perhaps not updating the tenants that depended on the building for power. Perhaps the building folks updated Internet Exchange Point Operator X periodically. We do not know. But what we do know is that Internet Exchange Point Operator X did not keep its customers updated during the outage, so the customers did not know if their equipment would be powering back up in a few minutes, hours, or days, and could not prepare. ISPs told DrPeering that they can understand that failures occur, but there were no updates for an outage that ultimately lasted over 24 hours.
Lessons we should learn
1) I anonymized Internet Exchange Point Operator X in this scenario because this could have happened to any data center operator where power is out of their control,. Indeed unanticipated failure scenarios happen to everyone in the data center space. What we should learn is that what seems to matter most to the participants is how the company responds to these events.
2) The part that was under Internet Exchange Point Operator X’s control was the notification and update, and in this scenario, they failed completely. The ISPs interviewed were very understanding about the outage itself, but were very upset about the notification failure.
3) The way the system failed demonstrates cascading failures. It was the sequence of power outage, followed by fuel pump polarity that led to the starter motor restart and the fuel pump clogging issues. If the fuel pump didn’t empty the roof tank, the other two failures (roof fuel tank sludge clogging the fuel pump, starter burning out) may not have occurred, at least not then. It was the sequence of failures that made this an extended physical plant outage.
4) An N+1 electrical and mechanical system redundancy did not help.
5) Testing of course was broken here.

Dr Peering
DrPeering@DrPeering.net

The Ideal Peering Forum (Part 2 of 2)

Wed, 17 Mar 2010 08:46:48 -0700

continuing the thread on the “Ideal Peering Forum” experience...
Goals of the Ideal Peering Forum
The primary goal of the ideal peering forum is to facilitate peering. The ideal peering Forum pulls together the “right” people into the room to establish, build, and maintain relationships conducive to increased peering between the attendee organizations.
There are also critical secondary goals. Chief among these goals is to increase the effectiveness of internet operations for this peering community. It is a goal that these issues are raised in this forum, and that we collectively highlight emerging trends and document best practices associated with the peering coordinator role. The forum assembles individuals aligned with these mutual interests and those willing to contribute their insights, and tries to pull them together into a cohesive community.
Another secondary goal is to ensure that the forum meets the needs of the attendees and their organization so they return to attend the next one.
It is for these Peering Forum goals, and from experiences at some of the best conferences all over the world that this paper tries to capture the best practices.
Attendee Walks Into the Ideal Peering Forum
When the attendee walks up to the Peering Forum Registration Desk he/she is greeted by name (a printed color peering yearbook with photo and name are taped to the table to help the registration staff), and are handed
• A very cool laptop computer bag, with logos of the sponsors of the event, with the name, location and dates of the forum stitched prominently on the bag. (Attendees get so many cheap bags that get stacked in a closet or given away, any viral marketing value of those bags is lost. This bag is intended to replace whatever bag they currently use, will become the bag of choice, and will show up at venues all over the world, making friends and family jealous and sure to try and get an invite to the next Peering Forum.) -- this idea based on Josh Snowhorn killer laptop bag given at a peering event
Inside the bag is
• A SIP phone, preconfigured with all attendees SIP phone #s so they can meet each other. This will facilitate the “I need to find Person X from ISP Y, but I have never met her.” Problem. This also is to demonstrate new technology in an open environment, and as a take home phone, can be configured for use at one’s home network. This phone during to forum is configured to allow the attendees to call home for free using the Internet and SIP gateways in each country that an attendee comes from. -- this idea from the APRICOT where Gaurab made a deal with a wifi phone mfg and preconfigured them with the SIP conference gear.
The attendee is welcomed and handed a Nametag with
1. The Name of the Person prominently displayed
2. the company and AS# prominently displayed
3. a clip-on and/or safety pin to hold the nametag to the shirt. The natural tendency is for people to position the nametag high on the chest, conducive to being read without detection by the attendee. (Note: lanyards are NOT handed out here; they hold the badge in a position below the dining table when seated, and force the eyes to look way down during casual meetings, leading the person to realize that you don’t remember their name.)
4. All nametag info is printed on both sides since lanyards (if an attendee insists on wearing them) often flip around leaving people to walk around with blank nametags around their necks! If no lanyards are used, the ideal forum can use the nametag holder to stow an agenda, meal tickets, etc.
More Giveaways
The laptop bag described earlier is packed with goodies including a logo’d largest possible memory USB sticks/drives. The USB memory stick contains, to the degree possible, the draft slides, associated white papers, the agenda and attendee lists in the form of a yearbook with photos, a couple of books on tape and eBooks for the flight home, etc. all in electronic form. In the evening social and the next morning, attendees can swap their USB pen for a new one that has all the material the original had but upgraded with an MP3/MP4 of the previous days presentations. That way, if there was buzz about a particular talk that was missed because discussions pulled an attendee out of the room, they can listen to the talk and review the slides on the plane ride home. When technology improves, the USB drives will have the previous days videos along with photos/video of the social events for perusal on the flight home, or for highlighting to the folks who approve the travel the value of the event. Since many people need to assemble a trip report, and/or presentation sharing the experience with the teams at home, having the materials handy helps them drag and drop the relevant materials onto an internal server or into a document with links to the sources on the server. The USB pen data will be erased eventually, but the logo on the USB drive will stay, providing the viral marketing benefit of the sponsor.
• Business Card carrying case preloaded with the business cards of the people organizing the event, perhaps with a personalized note to the speakers thanking them for agreeing to speak.
• Location Appropriate Goodies. For example, the NOTA 2005 forum handed out an additional beach bag with a large NOTA beach towel and hat. One could imagine since this NOTA Forum included a cruise to the Bahamas, that the bag might have also contained sun screen, sun glasses, aloe, some gambling chips from the cruise line to get the attendee started, etc. For the Equinix Ashburn tours in the winter in Virginia, a bus picked up the attendees, handing them a very nice black fleece logo’d jacket to keep them warm for the trip and while waiting outside for the bus to return. For Chicago in the summer, the giveaways included a Pizza Peering theme T-shirt, highlighting many types of peers as toppings on a pizza menu. These themes help reinforce the theme of the conference, and if done well, these giveaways represent a personal benefit to the attendee that decides (and successfully argues that they ought) to attend the forum.
• Quality Polo, Tshirt, long sleeve shirt as appropriate for the climate. Here again, quality is essential ($30 each in quantity). We like the NZNOG Polo shirt made of honeycombed mesh cotton as an example. It is lightweight, packs small, and doesn’t get wrinkled and is likely to get worn. -- Idea from Jonny Martin who did this for NZNOG
• Pad of 8.5 x 11 paper with a solid thick cardboard backing so that it can be taken out and written on while standing up. This can be logo’d with the event, dates, location. Waterman Pens engraved of course for writing, perhaps supplemented by LiveScribe smart pens and notebooks for electronic storage of meeting notes.
While perhaps cost prohibitive, the idea of including logo’d and engraved macbook airs in the laptop bag, preloaded with software, training material, video tutorials on various topics, welcome messages from the organizers, product spreadsheets, presentations, large video files, etc. is particularly appealing and will certainly increase the buzz for the following meeting. It is also a great way to incorporate social software to facilitate the meetings of folks interested in various things - kind of like an electronic bulletin board showing the ad hoc meetings, dinners, socials, etc. Having the software pre-loaded and configured removes some logistics hurdles.
Are these Bribes? Yes, there is an important word of caution here. Whenever the employee benefits (like giveaways at a forum) and there is an expense to the employer (flight to the forum, hotel, taxis, food, etc.) there may be a justifiable concern that all of this represents a bribe intended to curry favor with the attendees. A few people for example decided to attend the NOTA Forum during the day, but did not go on the sponsored cruise/social to the Bahamas. Some companies are more sensitive to this than others and may error on the side of caution to avoid the perception of impropriety. It is a difficult line to draw.
Giveaway Summary – all of these giveaways start the forum off to the right start. People will dig through their bags, discovering the goodies, talking about what is on the USB sticks, etc. Logistically bundling these things together in a bag speeds the registration process. The viral marketing accomplished by the bags, the USB sticks, etc. are proportional to the amount of time these things are seen by others. Others should become envious that they missed the forum and will try and curry favor with the organizers to make sure they get an invitation to the next Ideal Peering Forum. We want the attendees to recall in their discussions with other peers the Ideal Peering Forum event location, dates or at least year, and the best way to do this is with great usable giveaways. Peering Coordinators often recall the presentations and the event by the location or the social event, things memorable at the forum. This is something that can be socially engineered.
The Food Service at the Ideal Peering Forum
When you walk toward the room you notice a massive food and beverage selection on the table. The ideal Peering Forum has drinks available all day, not only at selected breaks. This forum is socially engineered to have the expected ad hoc breaks to have a discussion outside. We are trying to facilitate these interactions, and the break areas are the place to do this. In a variety of geographically dispersed break areas will be
• Coffee, variety of teas and chai, espresso drinks, all available in decaffeinated form.
• Fresh squeezed fruit juices
• Fresh vegetable drinks
• Bottled water and varieties of soda
• Healthy snacks as well as indulgent ones
Why focus on food and drink? At conferences, attendees are asked to wake up early (depending on timezone of origin) to drink up coffee and eat sugared sweets, and are then expected to sit still and listen actively to presentations. This works for a short while until the sugar crash occurs; the point is, these foods and drinks are universally delivered at conferences but are not conducive to the expected behavior. Therefore we need to provide and promote foods that help keep folks awake -- perhaps even educate the audience as to the benefits of longer tern sustainable energy foods and why the forum will make sure they are available.
This issue is particularly important in the afternoon. After lunch conference attendees suffer from ‘food-coma’, a sleep-like hypnotic effect that renders all but the most energized presentations incomprehensible. Heavy snacks for afternoon breaks such as brownies and ice cream have a similar negative effect when you want an attentive audience. Managing food is an easy way to indirectly improve the forum experience.
Also note that the first days of a conference have a more attentive audience, while later days the attention span drifts off. The number of attendees decreases over time as well, so a generally larger and more attentive audience is found in the early part of the first days.
Room Layout of the Ideal Peering Forum
The Ideal Peering Forum is held in a theatre-style venue with each row of seats descending down to a stage where the presenters present. George Washington University has such a theater and worked quite well for NANOG. Such accommodations are rarely available, so we will focus on the best practices assuming that the ideal peering forum is held in a hotel.
Classroom style with tables is used to facilitate note taking and use of laptop computers. There are two screens set as high up as practicable so all seats have an unobstructed view of the entire screen.
Each seat has access to power and hard wire Internet access via Ethernet plug for laptop users. Wireless may be substituted.
Video monitors on the floor in front of the speakers allowing them to always face forward, and to avoid them turning their heads to the side to view the slides on the screen which points their head away from the clip on microphones. The second monitor shows a countdown timer so the speaker knows how much time is left in their slot. The background color coding is Green until 5 minutes are left, Yellow until 2 minutes are left, and Red when 2 minutes or less are left. -- These best practices come from Next Generation Network conferences
There is a ‘green room’ where the speakers are wired such that the wiring is not easily viewable by the audience, nor is the microphone likely to be accidentally pulled out.
Each speaker has a bottle of water available on the table next to the podium should they need one during their talk. Each session should have a case of bottled water available so no one is ever looking around for bottles of water if a room monitor happens to be absent.
All (draft versions of the) talks are preloaded onto the presentation laptop, but presenters are allowed to update their preso up to the break before their presentation begins. In some case they may use their own laptop, but this is minimized to keep the speaker flow smooth and quick. The preferred method of updating slides is USB copying to the presentation machine with a quick walk through to make sure the transitions and animations etc. work as expected. This walkthrough detects the situation where a presenters newly linked file is absent from the shared laptop.
A transition title slide exists for each speaker showing a photo of the speaker and the title of their talk, with a URL linking to the presentation. This is especially helpful for the attendees viewing this cover slide as a thumbnail mode on their computers.
The laptops the audience uses have the ability to automatically follow along with the slides being presented by the speaker, with a notes page on the side for taking notes during the presentation. A forum wide instant message chat room is used for audience members to chat and ask questions during the presentation. A moderator will make sure this doesn’t get out of hand.
Agenda Selection
By far the most important preparation step of a successful Ideal Peering Forum is the agenda selection and its ability to pull together the right people into the room. To this end, select speakers who;
• Are social centers – they will bring/encourage others in the community to attend
• Are good speakers and have given talks in the past of interest to this crowd. Their talk title, their name, their contacts, their reputation will help attendees justify attending the conference. Think of the attendee asking their boss to attend: ‘What will you and the company get out of it?’
• Bring some topical material of interest to the crowd. For example, Todd Underwood from Renesys gave a last minute talk at NANOG about the recent level 3 – cogent de-peering. This is an area of interest to this crowd and was especially topical – it generated over 150 messages on the NANOG mailing list!
• Interactive Talks – find talks that have audience involvement. Examples: The Peering Simulation Game is a live simulation that brings audience members on stage to play the game. This successfully draws the audience in, and an on-line version would involve them even further. Another example is Peering Personals – a chance for the audience members to make a 2 minute ad hoc introduction of themselves. These are more dynamic, more unpredictable and more memorable than canned and approved presentations normally brought to these conferences.
Breaks and Socials
The breaks and socials are the most important times to accomplish the primary goals of the peering forum. The ideal peering Forum is careful to keep speakers on topic and on time, with the more controversial and those that inspire follow up conversations occurring just before the breaks and socials. Channeling these heated discussions and debates is a primary goal of the peering forum organizer.
Survey Forms
Surveys are on-line and hard copies are placed on each seat in the room at the breaks before the speakers for that session will speak. Each section of speakers has its own sections on the survey form so people who need to leave early are filling out the forms for the talks they heard. Some enticement is used to get survey forms filled out such as a raffle for an iPod, noise reduction headphones, PDA, etc. something that helps those who travel a lot.

Location, Access
The hotel is be reachable via a max of two plane segments from most origins, and be easy to get to from the airport. Directions and options for housing are listed on the event web site, complete with price points and pros and cons for each hotel based on personal experience. If possible, limos should be arranged from the airports to the venue for people traveling on the same plane together - start the social side early.
Social
Great care should be taken to select a social that accomplishes the following goals:
1. Encourages the interaction between the attendees
2. highlights an aspect of the location
3. is memorable – many times the event is remembered based on the social

We could go on - there are tons of execution ideas.
Consider these to be some ideas derived from some of the best practices of various operations forums around the world.

Dr Peering
DrPeering@DrPeering.net

The Ideal Peering Forum (Part 1 of 2)

Mon, 15 Mar 2010 11:14:49 -0700

Dear DrPeering -
Our Peering Ecosystem consists of several hundred Tier 2 ISPs and a few Tier 1 ISPs but we have never had a peering forum. We have a few exchange points but they don’t seem to be pulling together the ISP population. Do ISPs ever lead a Peering Forum?
Also - Will I see you at the Global Peering Forum cruise?
Greg Daniels
-------------------------------
Dear Greg -
Last question first... I am actively doing peering work for an ISP but, surprisingly, they seem reluctant to pay consulting rates for me to go on a cruise. Color me surprised. Not really.
The cruise venue has always been an issue with some companies - I guess it smells like a boondoggle, or a bribe, or worse (abuse of budget, collusion, pick your influence poison). Some folks can not attend for this reason, and this is unfortunate as it is simply a different venue.
I credit Josh Snowhorn (NOTA) with the first Peering Cruise. Josh made the discovery that hosting a peering event on a cruise was actually less expensive than hosting the event at a hotel (when you take into account catering, room nights, etc.) And the Peering Cruises for those who can attend have been very effective. I used to play down this “cruise” perception issue, until now, that I (sadly) can not attend.
Josh has also done a masterful job hosting, getting sponsors and creating great giveaways that people actually still use today, but that is another topic.
So back to your first question...
Peering Forums typically get started by an Internet Exchange Point (IX) that is looking to build a population of peers. By identifying a bunch of peers that can become members all at the same time, the IX can leverage their combined self-interests to build a critical mass of participants more quickly. In Europe, these seed ISPs might form a formal association and join en mass to bootstrap things and launch an IX. In either case, once enough participants are members of the IX, it is much easier to attract additional peers, and the peering forum helps achieve the ends of more peering for all. A real win-win-win.
But ISPs can start a peering forum, and might even be better positioned to do so. Why would an ISP start a Peering Forum?
1) Marketing Position. An ISP may seek to demonstrate leadership – in thought and in action. This messaging helps with customer acquisition, with regulators, and with popular opinion as the ecosystem grows and Internet prices drop. To be able to say “We have been taking a leadership role in evolving the peering landscape since 2008. Our goals are the same as the public goals – to evolve this ecosystem into a first class Internet ecosystem, with low prices, low upstream costs, and massive penetration of broadband Internet services” is powerful in emerging ecosystems like yours. Not only for goodwill; most people prefer to do business with a leader.
2) Regulatory. Since most governments prefer industry self-regulation, a peering leader has a bit more clout with the government, especially when guidance, information, or assistance / influence is desired. As issues such as network neutrality or de-peering disputes arise for example, key players may be consulted while followers and lurkers may not.
1) Market Research. Running a peering forum may actually be more effective and less expensive than paying for market research. The chart below shows the frequency and rough costs of running a peering forum twice a year, compared with the cost of purchasing a market research report every other year.
I would argue that one is more likely to hear timely information about happenings, and more likely to obtain the right answers to important marketing questions as a side effect of running the peering forums. Since you are in the industry you may be better positioned than an outside organizations to collect information directly from the field and running the forum gives you an excuse to talk with these folks in the field. You can also ask and evolve the queries to the right people at the right time, and change those questions over time. By running the peering forum, you stay in the loop by keeping communications channels open, collecting information before, during, and after the events that you run. Very valuable.
One final point that must be made - like IXes, Peering Forum demonstrate the characteristic that economists call “Network Externality” - that the value of the peering forum is proportional to the number and value of those attending. For IXes this effect is shown in the figure below.

Opportunity. Since you said your ecosystem doesn’t really have peering forums today, you have the opportunity build a peering forum to critical mass with no competitive peering forum in place.
Do it. It may be a challenge to start it up, but it is far more difficult to dislodge a competitive peering forum that is already at critical mass. Once you have critical mass, your peering forum can be very valuable to you and the population and be difficult to dislodge if you keep people happy. After all, there is a limited time and travel budget to attend meetings of this kind.
Next I will meditate on “The Ideal Peering Forum”...... Ummmm......

Dr Peering
DrPeering@DrPeering.net

Desirability of Content for Peering?

Wed, 10 Mar 2010 10:26:42 -0800

DrPeering -
I am excited about offloading my video content for free by peering. The ISP customer really wants my High-Definition content so the ISPs should want to peer with me, right?
Dana Eagle
---------
Dana - unfortunately, content to date has never had the upper hand in peering discussions.
Why?
Eyeballs are “captive” customers in that there are really only two alternatives for getting to these access customers : DSL and Cable. On the other hand, there are many alternative ways for access networks to get to the content since content is typically multi-homed. . . So when content wants to gain access to the eyeballs, the access networks own all the cards. Captive customers are why Comcast is able to charge content providers for paid peering.
Next, the peering ratios issue creeps in. An access heavy ISP will have their peering ratios adversely affected by peering with content. For those who are close to 2:1 out to in traffic may put their peering sessions in jeopardy by peering with content.
Next, the load that you will be placing on the access network will not generate any incremental revenue for the access networks, while they will see you as making ad revenue from the successful delivery of your HD video. They may see this asymmetry as unfair and not want to make it easier for you to load their network.
And your video content may be competing with their video offering. In the last few years, cable have migrated their peering decisions from a technical decision to a business decision.
For all of these reasons, the desirability of your content may not be as powerful a card as you may think.

Dr Peering
DrPeering@DrPeering.net

The Emerging Video Internet (part 1)

Tue, 2 Mar 2010 16:49:08 -0800

DrPeering -
What’’s the big deal about video? We are Peering coordinators. Bits are bits, right?
Angela Kinsey
---------------------------------------
Angela -
Why should peering people care?
Because the reality is, we are all in the video distribution business.
If we understand the video internet ecosystem we can better plan for growth, spot events, identify the next generation of customers, alliances, protocols, and services to better meet the unique and emerging needs of video distribution over the Internet.
It is important to understand the surrounding ecosystem that drives your peering bits since:
1. the financial value of peering is proportional to the volume of traffic that is peered “for free” compared against the cost of peering (equipment, colo, IX fees, etc.).
2. video is larger in volume than say tweets, email, or just about any other type of traffic. If 80-90% of all Internet traffic by volume is best characterized as video①, then I would go as far as to say that the Internet should even be renamed the Video Internet to reflect this dominant use. All other traffic (by volume) is insignificant in comparison.
3. the traditional 100-year-old video distribution system is migrating even more of its content to the Video Internet.
The traditional “Hollywood” system is where movies have been produced, licensed, distributed and viewed using tightly controlled methodologies over the last hundred years. The emerging “Video Internet” provides video content distribution with easy and in some cases free access to a global audience, or at least to those who have a robust Internet connection.
I have modeled the two systems, and in the next few ask.DrPeering articles, I will demonstrate how these two competing systems are in some ways parallel, in some ways clashing, and in many ways evolving in response to each other. I will highlight some of the mini-revolutions in each step of the video Internet supply chain, from producer to consumer, that have enabled the Video Internet.

The ecosystem at the top is the traditional Hollywood system, and starts with the controlled creation-production system. In order to have your idea for a film “green lighted” it must be one of the precious few screenplays that are accepted into the Hollywood system. Retired studio executives told me that inside the studios there are so many people adding their creative input into the process that the end result is often very different from the original vision. The point is, this is a tightly controlled process, and if you want to have your film financed and distributed into the theaters, you have to go through this process that has evolved over the last hundred years.
The ecosystem on the bottom is the Video Internet, a five year old process that is much more of a wild west frontier system. Anyone with an HD camera, some software, and an imagination can make and distribute globally their creative expression. While Hollywood focuses exclusively on blockbusters like Avatar②, there are opportunities for the Video Internet ecosystem to take over the lower and mid tier movie production and distribution. We will see examples of the Hollywood system migrating more and more of its content to the Video Internet distribution system. The unanswered question is “how do players in the Video Internet to make money?” The Hollywood ecosystem already has a mostly working economic model.
The economics of the Video Internet can not be ignored. A former Fox executive shared that it costs $1200 to make a celluloid film for distribution to the theaters. As a result, they are two-thirds of the way through a twelve year process to pay for the costs for the theaters to upgrade their systems to receive digital video. This has been an eight year process to get this far. A comparably slow process as Netflix can today stream a movie to your house for less than twenty five cents.
If the Internet is already 80-90% video, and Hollywood is only starting to migrate its library of movies to distribution over the Internet, we can expect to see increased dependency on the Video Internet for high quality video distribution. Peering coordinators should care about this because their value is how effectively their peering can move video traffic now, and the needs of this dominant application is all that really matters.

Dr Peering
DrPeering@DrPeering.net
Source:
① http://techcrunch.com/2009/06/09/cisco-by-2013-video-will-be-90-percent-of-all-consumer-ip-traffic-and-64-percent-of-mobile/
② “A World of Hits”, The Economist Nov 26, 2009

Comments from the field:
Patrick Gilmore correctly points out that statements like “the Internet is already 80-90% video” are hard to verify.
True, there are no probes uniformly distributed across the Internet that can accurately measure traffic to make ... Further, even if we had widely deployed probes capable of making the measurement, we don’t have the ability to look inside and accurately observe and categorize all types of video (streaming, downloaded, backed up, downloaded from file share server, encrypted or not, for personal use or not, pirated or not, etc.). We simply don’t know what percentage of Internet traffic is video.
DrPeering agrees. The data collected is based on the Cisco report cited, based on conversations with a lot of ISPs in Japan, Korea, etc. and presentations where no one raised objections to these estimates. Having said that, even if the percentage is less than that, say 40-50%, the underlying argument is still valid - this is the dominant Internet application from a traffic volume perspective. And traffic volume is key to whether peering makes sense or not.

Coming Attraction: Localization of Piracy

Tue, 9 Feb 2010 15:08:16 -0800

DrPeering -
Why is there metering of bandwidth for consumers in some regions, and all-you-can-eat for the rest of us? Is there a relationship to piracy - like, I would expect there is less peer-to-peer because of the metering?
Pat
----
Pat -
Most of the world doesn’t meter retail consumer bandwidth. Thanks to AOL in the U.S., we all now expect all-you-can-eat Internet and some of us make full use of it. The peer-2-peer folks for example keep their Internet connection open 24/7, feeding the world’s insatiable desire for all things available for free and getting greater ability to download what they want in return.
In some parts of the world, like Australia, New Zealand and South Africa, consumer Internet services are metered. English eyeballs love english content, and in these places, most of that english content is located on the other end of an expensive transoceanic cable. Or so the rationale for metered bandwidth goes. I call these areas “culturally contiguous zones”, and these island areas incur expense to fetch this remote content.
As I learned on my latest trip to New Zealand, these areas also get “legitimate” Hollywood movie and television content very late in the game (6-18 months after airing in the U.S.) if ever. Some kiwis complained that even then, there are so many commercials that the content itself gets cut out to make room for the commercials !
As a result, peer-2-peer downloads of edited television shows and captured movies is very high, and they don’t mind paying for Internet bandwidth to download the content. Australians communicated the same point as well. As one Australian put it - “we want it, and we are simply not going to wait.” And of course, the ISP services are more valuable to the consumers because they can get this content over the Internet services without fear of infringement enforcement.
But the meter does still run...If only there was a way to get the desired content without incurring any cost....
Direct Connect wifi Hubs
Well, there is. An ISP explained over drinks...
What was popular on campuses in New Zealand was swapping USB thumb drives with movies, music, etc. on it. Now, the concern is its replacement: devices like Direct Connect wifi Hubs / Wi-Drives. These are essentially wifi access points with a disk drive, so you and your friends connect your computer over wifi to the Wi-Drive SSID, and have access to all of the shared movies, music, etc. on the drive. It is a network replacement for swapping thumb drives.
All you need is a single copy of the latest hollywood movie retrieved over peer-2-peer or a mega download type site. Once copied up to the community wi-drive, everyone can fetch it during class. It is better than peer-to-peer since it is
Free. Since the content is shared across a local layer 2 service, it doesn’t go through the layer-3 campus metering as peer-2-peer does, so it costs $0 to share files.
A No Risk Exchange. Since it is a layer 2 service with the range of a wifi signal, Hollywood’s anti-piracy agents can’t see it. Once a single copy gets to a campus ad hoc wi-drive, the entire campus will have it and Hollywood can’t know about it.
Mobile. One can take the wi-drive to class, to a party, to a friends dorm, to a friends home and have a full movie archive swap fest and people may not even know who is sharing.
This isn’t much different from an Apple Time Capsule except that instead of seamlessly backing up, it is a platform for speeding file sharing. But what was interesting to hear was that the ISPs were so concerned about the resulting decrease in P2P traffic, and the adverse affect that would have on their metered traffic revenue.
It was also interesting to hear that it is illegal to capture Internet traffic in New Zealand, so ... even if there were anti-piracy agents in the field, they would be arrested for invasion of privacy before the copyright violator would be sued for infringement.
The only thing missing is the automation of the p2p download to populate the local layer 2 file sharing. It would be interesting if the layer 2 wi-drives could get automatically populated by people who roam between wi-drives propagating recently downloaded and copied content.
Video Internet Ecosystem and Hollywood Video Distribution Ecosystems Morph
The Hollywood Video Distribution Ecosystem and the Video Internet Ecosystems continue to morph and react to one another. As we will hear about in the “Emerging Video Internet Ecosystem” white paper, the Video Internet is evolving at a much faster rate than the Hollywood distribution system. Nowhere is this more pronounced than in Internet Regions where bandwidth is metered, and the traditional purveyors of content are not providing what the consumers want. Here, the incentive to bypass the system is great.

Dr Peering
DrPeering@DrPeering.net

NZNOG 2010

Mon, 1 Feb 2010 15:45:29 -0800

DrPeering -
I went to NANOG and RIPE and APRICOT last year. I can’t put my finger on it but they felt different. Is there a best or are they just different?
Scott Patterson
---
Scott -
I used to travel to as many Internet Operations conferences as possible -- I was on the road three weeks out of every month. My experience is the same as yours - each Internet Operations conference was different... But...
I just got back yesterday from the New Zealand Operators Network Operators Group (NZNOG) conference and I have to say - I’m impressed with this one. NZNOG is one of the best Internet Operations conferences in the world and I would like to enumerate why.
Top Ten Reasons why NZNOG is among the Best Internet Operations Conferences in the World
1) NZNOG meetings are informal. You feel this in all of the presentations, in all of the discussions in the hallway and in the room. There is a real give and take between the attendees, with courtesy, a respect, and an enthusiasm that is just contagious. I credit this ambiance to the grass roots nature of this conference - NZNOG started when a few folks wanted to get together and have a bunch of speakers and topics and found funding for it.
2) NZNOG opens with a pre-conference dinner. For folks coming in from out of town, this is a great way to ease into a conference; gone be the days when the first experience with an event is to stand in line on some floor in a cold hotel, taking a number to get a badge, then find people to speak with who are groggily slurping down coffee who are in various stages of jetlag. No, your first experience at NZNOG is walking together en mass to a nice Indian restaurant, with a couple informal welcome speeches, with people migrating around tables swapping seats and stories over beers and food. Just a perfect way to start a conference whose purpose is facilitating interaction among the attendees. Perfect.
3) Social Time is recognized as a critical part of the agenda with long breaks, finger food instead of full on breakfast (no seats required to eat), and hosted on a beautiful university campus. All of these things and other environmental factors seemed to work together to assist in the interaction among the attendees.
4) NZNOG organizers act more as hosts than colored badges to be honored. It felt as though I walked into a restaurant and the conference organizers were restaurant greeters, keen on making sure everyone had a good time and met the people it made sense to meet. For the people who know everyone to make the time to introduce people around is incredibly valuable.
5) NZNOG is still the right size - about 150 people. We could all fit into a large lecture hall, tightly so we could see everyone, and make ad hoc comments without microphones. I will confess that some of the comments went by so fast I couldn’t recognize what was being said in this New Zealand form of the english language - it mostly sounded like english anyway ;-). Anyway, the point is that NZNOG is still small enough that discussions happen between the attendees seamlessly.
6) NZNOG was chock full of solid topics given by eloquent speakers. This is important because often, with the laptop computers in front of people, there are many distractions from absorbing a speaker’s message. The topics ranged from technical (vulnerabilities in the routing architecture - Geoff Huston’s how routing announcements are lies), to practical (how we built a multi-10G interface router using off the shelf parts and open source software) to informative and future-looking talks. There was something for everyone.
7) NZNOG was adaptive to the interests of the attendees. How extraordinary that the conference would adjust its schedule and not try to hurry later speakers to make up for interesting discussions that arose from insightful talks. It is a time management challenge of course, but the focus being on the attendees rather than rigid adherence to an agenda felt right.
8) NZNOG allowed speakers to use their own laptops. This is commonly seen as a bad thing at most conferences because of the lag between speakers as they plug in. However, at NZNOG, this simply provided a break for people to check their e-mail. What I liked most was that speakers could update their presentations, animations, jokes, etc. based on what the previous speakers or discussions brought up. I liked, as a speaker, the familiarity with the presentation software and the content that was on my laptop, and that what I had practiced with would be available to me when I presented. To me, it is far worse to hear a speaker highlight that they had a slick animation to illustrate a point but that it didn’t show up because the conference insisted on using PDFs. The other thing that happens is version skew - the speaker updates don’t appear in the slides on screen and that causes distractions. I rather give the speaker the flexibility and creativity and allow the one minute break in the conference flow, which leads to my next point.
9) NZNOG meetings ran informally but very professionally. The video services provided were top notch and provided by a van filled with video equipment running outside the conference doors. With about five pro cameras and microphones brought in by an enthusiast (Richard Naylor I think), the video was captured and streamed, and from what I saw it looked great! As a side effect, It felt like a community event with this sort of community involvement. The video presence was not invasive - it didn’t feel like the cameras and bright lights were on and ready to record your talk questions and discussion - they kind of blended in. The room was comfortable, the speakers were well rehearsed and insightful, the Internet connectivity was solid, the talks were interesting, and everyone got a lot out of the event - what more could you ask for?
10) NZNOG remains a loose organization, demonstrating the flexibility to change over time. This is a broader point. Most Internet Operations conferences are operated by dedicated organizations, with formal roles and responsibilities, and perhaps somewhat formal community oversight. In the case of NANOG for example, Merit Networks runs the conference but has the program designed by a selected Program Committee of a dozen or so volunteers, selected by an elected group of Steering Committee members. All of this structure means that innovations have multiple levels of groups to go through - a real killer for creativity. This structure virtually guarantees that that the event will go on as it was when it received the organizational structure. Consider the uphill battle that a newly selected program committee person would have if they wanted to change something fundamental in the program structure -- no, they will go along for the ride for the first few meetings. If Jonny Martin and crew at NZNOG wanted to try something wacky, something never tried before, something that might fail completely or be a big hit, I believe they could and would try it.
In summary, NZNOG has something truly great. I believe that the Internet Operations conference breakthrough innovations will happen at NZNOG and other smaller Internet Operations conferences. This is one to watch.

Dr Peering
DrPeering@DrPeering.net

PS - It is important to acknowledge that there are real cultural differences across Internet Regions; a grass roots NZNOG launched in an Internet Region somewhere in Asia (for example) would probably look a bit different. These fora would probably evolve to meet the needs of the participants there much as NZNOG has done - it would just evolve differently to meet their needs.

Chief Technical Liaison

Wed, 20 Jan 2010 13:06:56 -0800

DrPeering -
What does a Chief Technical Liaison do for an Internet Exchange Point?
I have never heard of such a job... and none of my IXes have a person with such a title.
Jeff Hugery
---
Jeff -
Most IXes have a person who acts as a Chief Technical Liaison but without the title - you should get to know these people who know everyone. You may recognize the person after you read what the person actually does...
The Chief Technical Liaison (CTL) is the externally facing position responsible for working with the broad Peering Community to maximize the value of the Internet Exchange Point. This is accomplished in at least four ways.
A CTL Understands the Internet Region.
Who are the Tier 1 ISPs (those who do not pay transit fees to reach any destination in the region)?
Where are the Tier 1 ISPs peering with each other? What IXes or colocation centers? How they peer (public, private, cross connects, and at what capacity)?
Who is trying to get peering with the Tier 1’s and with what degree of success? Peering folks like to grouse about bad dealings with Tier 1’s, so there is much to be learned over beers. A CTL has heard all the stories and understands the dynamics of this ecosystem very well. It is also instructive to hear what Tier 2 ISPs are adopting a selective peering policy.
What is the market price for transit at different levels of commit, and what is the price of transport to each IXes in the region. The strong CTL has the numbers at the ready to discuss with peering folks how much it costs to build a backbone and peer in three locations in the U.S.
Who are the high growth Tier 2 ISPs and emerging network savvy content providers that are exploring entering the peering community?
Who was involved in the latest de-peering? What was the motivation for de-peering? The de-peered don’t like to talk about it (since they were the rejected) and no one wants to be seen as the bad guy, so the de-peer’er doesn’t talk either. It is difficult to learn about these things but the CTL has the finger on the pulse of the community so knows this stuff..
Liaison - a CTL does introductions.
The most important job of a CTL is building the peering population at their IX. To this end the CTL builds an extensive rolodex of peering contacts at each Tier 2 ISP, has a personal relationship with each one, knows who is moving where before it happens, and goes to every Internet operations forum.
Critical mass requires a peering population of folks who actively exchange traffic, so there is some sales and project management activity going on here. Every IX starts out with a small or zero population, and has to get folks to build in when there is no one there to peer with. This is called the “startup hump”, and is also known as the “Chicken and the Egg” problem. Some of the smartest IX operators in the world shared some of their tricks of the trade about this challenge with DrPeering in The Art of Peering - the IX Playbook.
The CTL is the person the ISPs gravitate to when they want to know what is going on -- because this person knows everyone. This is also the person who can help new peering people ease into the peering community. For engineering folks who may not be socially inclined, asking for peering can be an intimidating process. The good news is that most of the peering folks are very friendly. However, some are a bit gruff, and a few are arrogant, unpleasant folks that look like Jabba the Hutt with a goatee. Start with the wrong people and the task can be intimidating. A good IX CTL will assist with 5-6 introductions to the “cool Tier 2 peering folks” and the new peering person is off and running on their own.
A CTL seeks Peering Traffic.
An IX is in a difficult position - the value it provides to the population is the ability to peer with others. However, no one has to peer with anyone - it is purely voluntary. So, the value of the IX to the participants is completely out of the control of the IX operator.
Further, the IX can be valued based on the amount of traffic that is peered across it. Not only is this also out of the control of the IX... but the ISPs consider traffic information to be highly proprietary. So how does a CTL determine who to go try to bring into the IX if the ISPs won’t say how much of their traffic remains un-peered and to who that traffic goes to or comes from? It is like saying “Go do battle but I won’t tell you where or who the enemy is.”
The trick here is to generalize across the players by category and pull traffic information during informal conversations and in the sales process. “Tell me who you want to peer with and I will go bring them in” might come up at a sales call, followed by “and how much traffic would they allow you to offload?” Armed with this information, a CTL can assume that other companies similar to this one have a similar traffic profile and similar traffic vectors.
The final side of this has to do with technical trajectory - where is the next load of traffic coming from. It helps to be curious, focused on the emerging technology trends, etc. This is how DrPeering stumbled upon the massive growth of potentially peer-able traffic from youTube before they became insanely popular and started looking into peering. This interaction set also led to The Video Internet : The Next Wave of Massive Disruption to the U.S. Peering Ecosystem white paper. Where is the next big wave of traffic coming from?
A CTL does Peering Optimization.
The population assembled at an IX may not be optimally peered; there may be folks who would peer if they knew the other folks already at the IX would be willing. A good CTL has a grid showing all the participants at the IX ; where the columns and rows intersect describe the peering relationship between the parties:
1) already peered - these guys all set
2) Open Peering Policy meets Open Peering policy - PEERING INTRODUCTION leads to more traffic exchanged, maybe a port upgrade
3) Open Peering Policy meets Selective peering policy - MAYBE PEERING POSSIBLE
4) Open Peering Policy meets Restrictive peering policy - NO PEERING POSSIBLE
5) No Peering Policy - these guys don’t peer - NO PEERING POSSIBLE
Managing this quilt of peering relationships is a challenge but helps optimize the value derived from participating in an IX.
The other optimization that a CTL does is to suggest upgrades to peering ports when the capacity reaches a certain point -- this is easy revenue for an IX.
A CTL is a Peering Evangelist
A good CTL does more than just private back room introductions ; they travel around the world, become recognized as an expert on the topic, and speak publicly whenever and wherever possible. This outreach does two things - it gets the IX name out there and provides a welcoming hand for new participants coming in from outside the ecosystem. Having something to speak about means writing and researching and documenting and sharing -- all activities that most easily fall out of the activities listed above.
So, those are the broad strokes of what a Chief Technical Liaison does. A good one is invaluable to an IX, invaluable to the ISP Peering Coordinators and their activities can be directly attributable to the IX bottom line.

Dr Peering
DrPeering@DrPeering.net

Ray the Hoster

Mon, 21 Dec 2009 11:16:40 -0800

DrPeering -

In this economy, we are all nervous about spending money. Our exec team is stuck like a deer in headlights, requiring assurances that we never needed in the past. Do you have any advice about investing in infrastructure in this economy? I just don’t see our exec team moving on anything.

Wallace

----------------------

Wallace -

We were speaking of data centers at a VC dinner just last week and I got to tell my favorite data center story - the Ray the Hoster story.

In 2001 the economy collapsed. Ray had been waiting in the wings. Rather than fretting, he seized the opportunity to buy a recently moth-balled data center for literally pennies on the dollar. The GenSets were still shrink wrapped, servers were half unboxed and still there. Qwest had lit the building before the building was shut down.

Ray was network savvy enough to identify which networking HW he could get also for pennies on the dollar on Ebay to get the building connected and to start his hosting company. He found the right combination of gear to get things going, and contacted Qwest at the end of their quarter to negotiate a great deal on BW. He was the only one I have ever heard of being able to buy an OC-48 *leased line* and pay for how much he used it, simply telling Qwest how much he was going to pay each month based on his own measurements of his usage! What a dealer.

He sold the company after a couple years of filling the data center with paying customers in the upcoming economy recovery restart. He sold the hosting company for something like $40M.

I serve on a few boards and found that this story helps put things into perspective, especially for those who worry about the economy and making optimistic investments during downturns like this one.

Ray is my favorite example of being network savvy, opportunistic, and smart (frugal).

Dr Peering
DrPeering@DrPeering.net

...Dr Peering and the White Paper Process

Mon, 14 Dec 2009 13:59:05 -0800

Dear DrPeering -
Where do you get the information for the peering white papers ?
How do you distinguish fact and fiction ?
Are the white papers ghost written ?
Do the Ask.DrPeering articles go through the same process as the white papers?
Does writing the white papers make you an authority on peering?
--- interested parties
----------------------------------
Thanks for asking these questions. Many of you have graciously contributed data points and insights to the Internet Peering white paper series --- let’s talk about how the information gets assimilated into the final white papers and peering articles for the community.
White Paper Creation Process
The Internet Peering white papers are based on information pulled from some of the smartest networking people in the world. These experts were absolutely essential to the creation of the documents - they contributed their time and knowledge by participating in the white paper process described next.
Creating an Internet Operations white paper involves the following stepwise refinement process, face-to-face, over lunches, dinners, socials and in some cases over beers at operations conferences around the world:
1) Find an Internet operations issue related to peering that (a) is important and interesting, and (b) has not been specifically documented before.
1) Discuss the issue with anyone who has insights to share, and document what is learned. [ Fact from Fiction? The early drafts are based on conversations with only a few individuals and as a result, early phase white papers are rough and have errors in them. The next stage in the process tends to weed out at least the larger errors. ]
2) Walk folks through what the previous folks have shared. These folks either validate the points of view, correct them, or add data where it is important to add information. [ Fact from Fiction? Indeed, peering is a highly politicized and often heated topic, so there are sometimes “religious” and/or diametrically opposed view points. One person’s fact may sometimes be another’s fiction. Because of this, it is important for the white papers to either highlight these differences or focus on the commonalities. ]
3) Return to Step 2, stepwise refining until no objections are raised. This tends to be about 100 walk-throughs and takes six to eighteen months. After one hundred walkthroughs, the best way to explain a complex topic simply and clearly can emerge.
The end result is a white paper on a particular Internet Operations topic that pretty much represents the broader community mindset on a particular issue.
5) Share back white paper to the community
Finally, present the research to a broader group (NANOG, RIPE, APRICOT, etc.) to gain final feedback and share the information back to the community. After all, the community provided the content, so it makes sense that the broader community be provided free access to it as well.
All of the content collected for the white papers is still freely available at the http://peering.DrPeering.net web site.
Are the white papers Ghost Written? No, I alone wrote my Internet Peering white papers. It is important to note however that many of the underlying ideas are not my own -- the content is based on information collected from networking people around the world using the process described above.
Do the Ask.DrPeering Articles go through the same process? Unfortunately no, the white papers process doesn’t work well for a monthly blog because there simply isn’t enough time. These articles are based on conversations with networking folks, often at a private silicon valley lunches where techies get together to discuss such things. As with the early stage white papers, these articles are based on a few conversations, so are more prone to errors. So we continue to evolve this process as we move to a moderated DrPeering discussion area and an advisors@DrPeering.net support group.
If bonafide errors are identified, we will and do fix them right away on-line -- in this way the articles have the advantage of being easier to update than when different versions of the white paper PDFs are floating around the net.
Expertise in Peering? My expertise comes from over twenty years working on the Internet, writing the business plan for NANOG and chairing NANOG from 1995-1998, and then ten plus years of additional research in the area, assimilating data points not just from one perspective but from the varying perspectives of hundreds of peering coordinators, network engineers and architects from around the world. Very few people in the world have researched the topic or have exposure to more different peering perspectives than I. These papers are cited by academics and taught in universities around the world. The challenge is working through all of the passionate opinions, fanaticism, etc. and in some cases highlighting both sides of a debate as in the “Great Debate - Private vs Public Peering” and “The Folly of Peering Ratios as a Discriminator” .
Throughout this time, my role has been that of a neutral facilitator. I brought the Peering Birds-of-a-Feather (BOF) sessions back to NANOG and chaired seventeen of them in a row. To highlight different views on emerging peering topics, we launched the “Great Peering Debates” as part of the event. These sessions were fun, informal, well attended, and served the needs of the Peering Community. I assisted with hundreds of peering introductions, helped pull together the aligned interests of the cable companies, the Tier 2 ISPs, and started the precursors to what eventually morphed into the PeeringDB and the Global Peering Forum. All of these activities required a neutral stance and an inclination to facilitate the discussions among the parties. Over the decade, these activities provided me with a bird’s eye view of the Internet Peering Ecosystems that I have shared with the community in the form of the white papers.
I hope this helps -

Dr Peering
DrPeering@DrPeering.net

Asia Boomerang

Sun, 13 Dec 2009 14:04:45 -0800

Dear Dr Peering,

Asia is a strange place to try and peer in. An area of many incumbent telcos who are quite happy to peer on the West Coast of the USA but not in Asia. They seem to be willing to fill the coffers of the global Tier1 providers and the TransPac submarine cable owners rather than improve intra-Asian latency and reduce cost for all in a desire to pretend to be a "regional" Tier1 and make everyone pay a toll to access their particular bit of the market. Yet, each one of them laments that others won't peer with them in Asia either and so their costs and latency are held high.

Are there same sage words you could offer to help break this hypocritical stalemate?

Regards,
Matthew

Matthew -

I asked the very same question of about one hundred peering coordinators and documented what I learned in, you guessed it, a white paper called “The Asia Pacific Peering Guidebook.”

It does seem odd that a Tier 1 ISP in one Internet region would prefer to pay for an expense transoceanic circuit to the United States, and peer there openly, rather than let someone else build into their home country and peer for free there. Why not let the other guy pay for a circuit into your home market?

One reason was summed up quite well by John Milbourn (at DACOM at the time) :

“Protecting one’s home market is far more important than any benefit peering in one’s home market can provide.”

For the Tier 1 ISPs in a region, their home market is one where they pay no transit fees by definition. Anyone who wants to send traffic to their customers from anywhere in their home market will pay at least one of the Tier 1 ISPs directly or indirectly (by buying from one of their downstream ISP customers).

What would happen if a competitor (light blue circle in the diagram above) shows up in country and gets peering with a Tier 1 ISP in that region? The new ISP entering the market could cherry pick the best customers and offer transit prices below the prices the Tier 1 ISP has set. The entering ISP can simply send that traffic right over to the Tier 1 ISP’s network for free.

This exact situation happened when one International ISP went into Brazil, negotiated peering with one of the Tier 1 ISPs there, and was de-peered shortly after they started acquiring some of the Tier 1 ISP’s customers. Generally speaking, Tier 1 ISPs have a restrictive peering policy in their home Internet Region, and do not want any more peers. As Waqar Khan (Qwest) phrased the Tier 1 ISP perspective, “We have all the peers we need”, and he is correct -- a Tier 1 ISP already has access to all routes in the Internet Region.

Another example from the field. You will find that in peering negotiations, ISPs such as Korea Telecom will negotiate what Korea Telecom routes can and can not be announced where. In one case, peering was accepted only if the peer provided free transport from Korea to Hong Kong and peered with them there in Hong Kong, and under the terms that these routes would never be announced within Korea. So the peer could not compete in Korea. This example shows how protecting the home market trumps peering benefits and is negotiated accordingly.

The Tier 1 ISPs in Asia that have a restrictive peering policy in their home markets will often have an open peering policy on the west coast of the US. Why? The reason is that the peers on the west coast of the U.S. can’t compete with them in their home market; the latency is too high for the peer to be able to sell transit to the home market.

This is why you see a lot of Asian ISPs peering openly with other Asian ISPs on the west coast of the U.S., and traffic between Asian countries follows that boomerang path across the ocean twice.

Solutions? Many layers of political issues come into play that have prevented many initiatives except for bilateral peering across half circuits. Since most Asian countries wish to be the topological center of the Asian Internet, they are have a disincentive to helping one another be the topological core of the Asian Internet. DrPeering believes the solution is one that allows any Asian country to be a node in a distributed peering system that topologically replaces the U.S. West Coast peering.

Dr Peering
DrPeering@DrPeering.net

Paid Peering and Net Neutrality revisited

Mon, 30 Nov 2009 17:35:03 -0800

Dr Peering -
You seemed to have stirred up quite the flame war with your Paid Peering and Net Neutrality article. What is going on?
Felix
-------------
It appears that at least one person is under the mistaken impression that DrPeering went to the FCC to lobby for the regulation of peering, paid or otherwise. This could not be further from the truth. On the contrary, DrPeering, very publicly, raised a warning flag that section 106 of the NPRM describes a rule that appears to come strikingly close to the heart of peering.
As with all of these discussions, there are two sides to the coin. Let’s clarify and focus on the core of the issue.
Policy friends and indeed Jon Peha (Chief Technologist, FCC) is cited to have said here that sometimes these rules are intentionally vague in order to elicit comments. So since :
a) the section 106 rule appears to have the intent of disallowing access networks to charge for “enhanced” access to the eyeballs, and
b) paid peering enhances performance (lower latency to the eyeballs, more control over routing, more visibility, etc. by bypassing the transit providers route normally in the path),
raising the potential regulatory threat posed here seems appropriate.
At the heart of the discussion is the question, “Does this rule apply to peering?”
The folks who say “No” point to the FCC diagram in section 106 (see below), shown on top of an illustration we will discuss later.

The double arrowed line between the end user and the broadband service provider appears to mean that the rule is intended to limit what an access provider can do within it’s its own network. The belief therefore is that this rule does not apply to what the access provider does at its borders where it peers.
If this is indeed the FCC’s intent, then we can all go home and sleep soundly knowing that there is no regulatory interest in peering. (The FCC would not comment on this point when the question was asked directly. Apparently this silence is a fairness issue - if they clarify for anyone, they need to clarify for everyone, which would require a new release of the NPRM... So if anyone claims the FCC has offered clarification, then let’s cite the source and be done.)
However, the decision to peer or not to peer at the access provider edge, in effect, makes a big difference to the prospective CDN, content-heavy ISP or content provider peer. As shown in the diagram below, we see what, in effect, looks like two-tiered access to the eyeballs they want to reach ; enhanced (paid peering) for those who pay the metered rate for paid peering to the access provider, or the more circuitous transit path for those who do not pay the access provider to direct peering.

So if you take the bi-directional arrow in FCC diagram above to include the path from the access network peering router all the way to the eyeballs, then paid peering is a mechanism that provides that “enhanced” access to the eyeballs. At least, that is how it looks from the outside to the content provider.

So all of this fuss is about this interpretation.... if you believe that the intent of the rule in section 106 of the NPRM is to prevent access networks from charging content companies for enhanced access to its eyeballs, one could view paid peering as precisely this -- higher performance access to its eyeballs for a metered rate executed as a paid peering product, selectively sold and priced as the access provider decides.
If I am misinterpreting the intent, then we can all breathe a sigh of relief; peering is no where close to the regulatory scope.
If there is any correlation between the rule and paid peering we are all in for an adventure as this will affect network relationships worldwide and may invite further regulation on a country by country basis or a network by network basis or some unforeseen future consequences.

Dr Peering
DrPeering@DrPeering.net
------------------------------------------------------------------------------------------
Addendum - NANOG 48 in Austin TX Feb 23, 2010 included a Net Neutrality session with excerpts from the Q&A section transcribed below.

It is significant since Jon Peha (FCC) is consistent at NANOG as in private discussion with individuals in the industry.
(1) When asked “Are an access network’s upstream management practices, through their transit or their peering relationships, out of scope to the NPRM rules?” the answer was that “the NPRM does not have an answer to the question”, so the issue is in other words is ambiguous.
at the same time...
(1) Dan Golding said if you as an access provider “evaded” the spirit of the FCC rules with the network’s handling of its upstream relationships (prioritization,prepending, etc.), then the FCC would be “slapping you down”. So even if not explicitly articulated in the NPRM, under the right circumstances, an access provider’s use of paid peering for example may be brought into examination at the FCC. According to Dan anyway.
If the concern is that the FCC might be interested in peering and transit relationships, the NPRM is silent on the issue but Dan Golding says that even if not directly specified, the FCC will use the rules if the access network is evading the spirit of the rules (to prioritize one’s own services over those of competitors for example). Seems like that would make peering in scope. Too bad the FCC folks didn’t explicitly agreed with Dan Golding's statement, but alas there was no comment here from the FCC.
It seems that both side declared victory, that their interpretation was correct. Here is the edited transcript that only includes the Q&A portion of relevance. Hopefully this will be helpful in dispassionate discussions.
Transcript
[ 10:47AM At the tail end of Questions and Answers session (Note: ‘…’ means the transcriber couldn’t hear what was said). ]
(1) Dan Golding: Um, just before we get to your question, I, I have a question that I believe is very, very important so before we run out of time which is the question of peering. There is some discussion in the community about… Dave , would you like to pose your question about paid peering?
(2) Dave Tempkin: Sure, so the question I made reference to you earlier. You know, there’s been a lot of people who boil it down to a very, very simple question: Is paid peering really …. Am I no longer able to directly connect my network to some ther network for some undisclosed sum of money?
(3) Zachary Katz (FCC): I would say, nothing in the proposed NPRM has states that paid peering is forbidden.
(4) Dan Golding: fourth microphone
(5) William B. Norton: Well, You guys actually stole my thunder a little bit because I was going down that very specific path.
I will thank you for the clarification because I think that was the source of uh some degree of mild heated discussion.
I would ask the question a little bit more generically. Instead of saying specifically paid peering. … I actually wrote it down so um I can get it right…
Are an access network’s upstream management practices, through their transit or their peering relationships, out of scope to the NPRM rules?
(6) Dan Golding: Did you say carrier or peering?
(7) William B. Norton: Um, well, it goes to where the boundary is in that access network, and whether he sends traffic through a circuitous path or whether he goes through direct. Is that out of scope in the NPRM?

(8) Zachary Katz (FCC): I may not have the technical background to answer this, so with that caveat, I don’t believe the NPRM prohibit um..
(9) Jon Peha (FCC) – The simple answer is the NPRM itself does not, does not have an answer to that question, and that it is not explicitly….
(10) William B. Norton: and that is the answer that gets us in trouble Jon
(11) Jon Peha (FCC): No, no. This is.. If there are people in this room who think they have concerns one way or the other, they are welcome to submit them, in fact are encouraged to help us understand the issue, but there is also to be clear, I’m not saying this is, you know,… to repeat Zach’s point, there is nothing in there that discusses these peering arrangements. There are clear principles about access or not, uh…, and if you think that there are peering issues, then you ought to submit them in form(?).

(12) William B. Norton: No, I was looking for the answer that I thought I heard which was “No, that ‘s out of scope”, or “Yes, peering and peering practices may be a way that one would implement some of these practices that you are trying to make sure don’t happen” and that’s where the heat tends to come in.
(13) Dave Tempkin: There’s a diagram in the NPRM actually that is pretty compelling, I forget the page that it’s on, that basically shows…
(14) Dan Golding: 42, page 42
(15) Dave Tempkin that basically shows what’s in scope, so It is pretty clear that it is intended to be in the last mile, but whether or not the language needs to be cleaned up, um, you know, that remains to be seen. From the picture, that’s what it appears..
(16) Dan Golding: From my experience with the FCC that being said, ,if If folks are trying to do illicit filtering or prioritization on peering routers , or they they try to to do things … they say “Well we aren’t doing things in the last mile but we’ll makes sure our other peoples voice traffic is degraded by getting on the peering router with routes essentially being pref’d up, even if the NPRM right now doesn’t fit perfectly , my view of the FCC is that they are particularly humorless no offense, when it comes to people trying to evade the spirit of the regulation. You know FCC regulations are a little bit different from most conventional law in that, most laws, if you meet the letter, the spirit matters less. With FCC regulations if you try and get around the spirit they will end up slapping you down.
(17) Richard Steenbergen: Hi Richard Steenbergen nLayer. So my question is somewhat related to what Bill was talking about. There is a liong standing practice for certain eyeball networks where they intentionally congest their transit, their peers, run their network hot. And then they say “well, if you want good access to us then you can just buy directly from us.” And this doesn’t involve any QoS because everything that you don’t pay them for is degraded . Is this something that you guys see is an issue. Because essentially what these guys are doing is using their captive customers as leverage to force other people to pay them.
(18) Dan Golding: <Dan re-asks the question…> Is that something that would be covered under the scope of network neutrality?
(19) Zachary Katz (FCC): So, it’s a real interesting point. It’s not one that I personally I’m familiar with but one I would like to learn more about before we provide guidance one way or the other.

How Peering has changed

Mon, 23 Nov 2009 09:17:00 -0800

Dr Peering -
What has been the most significant change in Peering in the last 7 years?
Hank
----
Hank - Thanks for the question.
I would have to say that the most significant impact on the peering ecosystem over the last 7 years has been the peering behavior of the cable companies. Here is the transition as I saw it.
In the early days, @Home handled pretty much all of the Internet activities for the cable companies, and until it went bankrupt, it had a highly selective peering policy.
After @Home went bankrupt, the cable companies had to take over the reigns and grow their Internet businesses themselves. Some cable companies were more dependent on @Home than others, but insourcing Internet Ops represented a major evolutionary step for the cable companies.
~2002 - Most cable cos started out as Open Peers (the engineering teams would just peer with anyone to make traffic flow more efficient).
~2003 - The window for peering with the cable companies lowers - some traffic volume required to peer, and they stated that they required a couple locations, although that requirement was sometimes waived.
~2004-2005 - Peering decisions are no longer made by engineering teams, but by a broader set of business folks. Any words spoken like “set top box” for example by the prospective peer have immediately terminated peering discussions. Peering migrates into a broader business discussion; how will this impact the future business plans?
Today - many cable companies have peering policies and prerequisites every bit as strong as ISPs, and Comcast is selling paid peering and transit. Free peering with eyeball networks is increasingly harder to come by.
The second significant change is the fat middle of the Internet Peering Ecosystem.
Specifically, in the last 7 years:
A) CDNs peer, Large Content Companies peer ... this is not new to the last 7 years; This was a strategy for CDNs from the beginning but the amount of traffic peered by the CDNs and by the large network savvy content companies today makes the middle of the Internet Peering Ecosystem very fat.

B) Enter Video Internet Ecosystem - this is not a peering thing but the massive flood of video traffic has effectively amplified the benefits from peering for all those delivering/receiving the bits for free. Video streams just consume so much more bandwidth than tweets.

Dr Peering
DrPeering@DrPeering.net

Comcast Paid Peering Service Reviews

Thu, 12 Nov 2009 07:52:32 -0800

DrPeering -
[...stuff about the previous askDrPeering article.... religion, extortion, etc. deleted ...]
What about the actual Comcast Paid Peering Service? I’d like to know how well it works in practice.
Anees
---
At NANOG lunches, over beers, at dinners, and over a recent California Geek Lunch, a bunch of us peering geeks discussed the Comcast Paid Peering Service, and DrPeering got an earful. Almost all of it positive.
Aside from the religious fervor over the politics of paid peering, Peering folks said of the actual Comcast Paid Peering Service that:
1) Unlike the previous attempts at Paid Peering products, Comcast priced it right, down towards the lower end of the transit price scale ($1-$3/Mbps). (In the past AOL priced paid peering higher than transit, arguing that it performed better and was priced to cover costs. Very few people took AOL up on its paid peering product.)
2) Peering provides a shorter AS hop, so a transit provider that peers with Comcast tends to get more of the multi-homed traffic destined to Comcast. For ISPs this leads to not only better performance for its customers, but more traffic and therefore more revenue. (There have been heated debate in the past about this Dave Rand argument.)
3) Peering provides better performance than getting access to Comcast customers via their transit providers. One customer said they experienced 2-3 times better performance with the paid peering product. One other said that was not their experience. It depends on the previous path I suppose.
4) One peer said that Comcast paid peering helped them bypass the networks that experienced routing issues, maintenance, upgrades that failed, etc. issues; their traffic went directly to the eyeballs while before they saw these issues.
5) With paid peering there is support, while free peering relationships generally has less support. One peer said the Comcast Paid Peering support was actually too aggressive. Comcast apparently was over communicating, calling NOCs when maintenance or upgrade outages were already known, scheduled and accounted for. The conversation then turned to the fact that they preferred more communication to less.
I know this reads like a Comcast Paid Peering promotional piece, but what can I say? This is quite simply what folks are saying in the field about the service.

Dr Peering
DrPeering@DrPeering.net
PS - We do need to continue to explore the Paid Peering issue as it related to net neutrality, as there is a very real threat of regulation that could impact everyone in the Internet Peering Ecosystem and across the Video Internet Ecosystem.
They can’t regulate this? Consider for example that Australia government officials took a brief look at Peering and said “Peering looks like barter, and therefore should be subject to taxation as barter” !

Paid Peering and Net Neutrality

Thu, 5 Nov 2009 16:00:22 -0800

DrPeering -
Why hasn’t anyone brought up Comcast Paid Peering and network neutrality? Isn’t Paid Peering extortion? As in... Tony Soprano saying “You want access to my customers? You pay me.” <queue Sopranos theme music>
Sy Musico
----
Sy -
DrPeering sees two sides to this argument, and you are describing one side. Kudos for the imagery.
The Peering Community seems to have (at least) two views.
Here is one view: Comcast Paid Peering is extortion because
1) A CDN is paid to get content topologically close to the eyeballs.
2) Akamai already has its equipment embedded inside Comcast’s network [Some debate/heat has arisen on this point].
3) For a CDN to compete with Akamai effectively (on performance) its needs to get its content close to the eyeballs.
4) Comcast charges a metered rate in exchange for getting content topologically close to the eyeballs (aka Paid Peering).
5) There is no alternative provider that can be used to get closer to Comcast eyeballs.
There are even rumblings that performance of one of Comcast’s transit providers is sometimes congested, which makes Comcast paid peering even more attractive; paid peering is a lower-cost higher-performance alternative to traversing that transit provider to reach Comcast eyeballs.
But you have to pay for this high quality access to Comcast eyeballs.
How is this not two tiers of access?; a sometimes congested transit path, or a direct paid peering path that one network operator described as having twice the performance of transit. If you were a CDN, you would really have no choice.
[ Editor’s note: Time Warner Cable and other providers also have paid peering products. Several Tier 1 ISPs and AOL have had paid peering products for years and have marketed them to ISPs as stepping stones to free peering, and to content heavy ISPs as providing improved access to the Tier 1 ISP’s customer base. ]
Here is another view: Comcast Paid Peering is an enabler, a competitive threat to Google.
Google wants Paid Peering to be illegal, along with any form of access tiering.
This argument is draped in Internet freedom, let innovation thrive language, and Comcast made mistakes with the whole DPI and throttling p2p traffic thing. But let’s put the religion aside for a second.
If you were Google, strategically, why would you want paid peering illegal ?
Killing Paid Peering will increase Google competitors’ cost, and decrease Google competitors’ performance. Consider the cost of traffic exchange graphic below.

Any company can buy Transit (metered access to the global Internet) for around $2-$9/Mbps (Source: 2009 Field data). Google is paying about $0.50/Mbps for transit today (Source: RampRate).

To optimize performance and decrease costs, a company can siphon off the traffic destined to Comcast eyeballs and buy Paid Peering (direct access to Comcast network) for around $1-$3/Mbps.

Paid Peering enables Google’s competitors to get access to Comcast eyeballs for around the same price as transit ($1-$3 instead of $2-$9/Mbps).
Paid Peering provides better performance than Internet Transit, since the traffic takes a less circuitous route. As shown graphically in the movie below, direct Internet Peering can provide 2-3 times better performance than the Internet Transit alternative.

Therefore, Paid Peering allows Google competitors to more easily compete with Google on performance and price without having to reach Google scale.
What happens when Paid Peering is made illegal?
Killing off paid peering would force content heavy companies, ISPs and CDNs to pay a higher cost to reach the eyeballs and remove for all the performance benefits of paid peering.

[Editor’s Note: These graphics point to Google in the <$1/Mbps range but there are others who also do the volume of traffic to get into this transit price range.]
To finish up this discussion, consider the hypothetical network diagram below. Here we see several networks and CDNs that could be used to access Comcast eyeballs, each with different proximity values to the eyeballs.
Let’s explore a hypothetical example to highlight proximity choices facing content companies and see where paid peering might come into play.

In this hypothetical scenario, NetFlix could purchase from a transit provider like Sprint, but its content would be two network hops away from the eyeballs that the content guys want to serve. Purchasing from a Limelight in this scenario would help push the content a network closer to the eyeballs compared with transit, but purchasing from a CDN that peers with Comcast (BitGravity in this example) would get the content even closer. Akamai is rumored to have nodes embedded within Comcast which would be the best (topologically closest) option if true. If NetFlix saw it as strategically important to run their own peering, they could perhaps seek paid peering with Comcast.
One thing is clear -- if you are a CDN, you need to buy paid peering from Comcast to compete on performance with Akamai.
Summary - Paid peering appears to be similar to a two tiered access network, one of the very things that the FCC may seek to regulate as discussed in section 106 of the discussion draft.
We should explore next the negative consequences of regulation, some unforeseen negative consequences, and what the peers say are some of the benefits paid peering offers over unpaid peering.

Dr Peering
DrPeering@DrPeering.net

Transit Provider Selection

Mon, 26 Oct 2009 13:58:36 -0700

Dr Peering -
We are looking to start peering. Does transit provider selection have any impact on our peering strategy?
Alton Brown
-----------------------
In some cases, selecting the right upstream transit provider can increase your chances of obtaining peering. From “The Art of Peering: The Peering Playbook Tactic #5...”
Let’s assume that you buy transit from a Tier 1 ISP (Global Crossing (GLBX) in this example). As a Tier 1 ISP, Global Crossing by definition has access to the entire region through its peering sessions with the other Tier 1 ISPs.
Let’s further assume that GBLX has a less restrictive peering policy than the other Tier 1 ISPs and peers with ISP B, a large regional ISP that you wish to peer with.

Transit Pricing Menace ?

Thu, 8 Oct 2009 09:15:06 -0700

Dear Dr. Peering -
I stumbled across an article “Transit Pricing: A Hidden Internet Menace” and didn’t quite follow it.
Help me digest this article.
Sergey Redevic
( http://www.internetevolution.com/author.asp?section_id=561&doc_id=182563&f_src=internetevolution_sitedefault&piddl_msgpage=2#msgs )
-----
Don’t digest that article; it doesn’t smell quite right.
Instead, for a high level overview, take a look at http://www.youtube.com/watch?v=3sSPAW2_9Ng , a very well done film from the Euro-IX that covers high level Internet notions along with a description of Peering.
Then take a look at some of the community-based interconnection research white papers like “Internet Service Providers and Peering” :
http://www.drpeering.net/a/Internet_Service_Providers_and_Peering_Definitions.html
and
“The Business Case for Peering” :
http://www.drpeering.net/a/Peering_vs_Transit___The_Business_Case_for_Peering.html
These provide a better description of what is going on with money exchange and peering between ISPs.
We can then talk about the wholesale vs retail transit and the various peering arrangements made between parties that optimize traffic delivery. The article seems to infer that transit fees are somehow hidden fees like baggage handling fees the airlines charge, or security fees charged when you book a flight, or shipping/handling fees when you buy a ticket to a concert. This is really not the case.

Dr Peering
DrPeering@DrPeering.net

Why Peer when Transit is $2/Mbps?

Tue, 22 Sep 2009 16:14:58 -0700

DrPeering -
Why bother peering when I can buy transit at $2/Mbps?
Steve Rzenberg
----------------------
Steve -
The decline of transit prices has been going on for a long time and the story is always the same; when considering only the financial benefits (cost savings), peering makes sense when you can peer away a large enough volume of traffic for free to offset the cost of peering. Let’s build a simple example.
Let’s say you are an emerging video player, with a lot of traffic to offload to transit or peering interconnects. Transit. We will go with your assumption that transit can be purchased for $2/Mbps ( today it is rare to find a transit price this low, but let’s go ahead and use this price point).
Peering. Let’s assume that you are already at a populated colocation center with enough traffic to utilize a 10G peering port to capacity, which we will assume conservatively is about 7500Mbps. (Above this capacity you are more likely to see packet loss occasionally at peak traffic, and we want to avoid that.)
Let’s assume that the price for a 10G peering port is $2500/month, a pretty standard price in the U.S.
At capacity, you are paying
$2500 / 7500Mbps=$0.33/Mbps
for that peering port.

Letter of Intent=Fail

Fri, 28 Aug 2009 08:41:27 -0700

Hi DrPeering -
When should a colocation provider deploy a peering switch fabric for its customers?
Margie42@Wresting.org
---------------
I will save you a lot of time and effort by sharing the experiences of one IX who used the “Letter of intent” strategy.
Success-based marketing - get “Letters of Intent”
Faced with the cost of deploying a carrier grade ethernet switch, the marketing wizards at one IX decided to reduce their financial risk by only purchasing the peering switch in locations where the sales people can find 10 customers that say they will connect to the switch if one existed. A signed “Letter of Intent” would be the vehicle communicating this interest.
“If the customer is interested in public peering, tell them we will deploy one for the customer base when ten have agreed to buy ports when it is available. If they aren’t able to sign the letter of intent, then they aren’t ready to sign a purchase order for the product.” -- Marketing Lead
Sales people visited customers and prospects. When asked “Do you have a public peering fabric, and what are its price points?” the sales person replied, per script, “We don’t have one there yet but we will deploy one when we have enough people asking for one. We have a letter of intent to sign if you are interested.”
Did the prospects sign the letter of intent? No ! They wrote on their notepad
“No Switch”.
After 100 such conversations, very few signed the letter of intent, and many went to the competition and helped them build their critical mass.
Here is what the colocation provider learned from these interactions. You have to look at things from the customer perspective.
WHY DIDN’T PEOPLE SIGN THE LOI?
1) Not authorized to sign. The prospect for peering services were not in the mindset position to commit to sign anything; they were exploring the options. Further, they often didn’t have the authority to unilaterally sign a document on behalf of the company - it may be a team decision or a decision for someone else. In any case, they now have a slightly negative taste in their mouth since they were being put in the position of admitting to themselves and maybe others that they are not empowered to sign anything.
2) Not committed to a single solution yet. They were often not committed to the specific colocation provider yet, so they strategically wanted to keep their options open. It would be premature to sign any verbal or written commitment of any form to anyone.
3) No immediate benefit to sign. What is the benefit to the customer of signing a letter of intent? There is none, except perhaps the delayed benefit of a switch being available perhaps at some point in the future from this colocation vendor. There is zero benefit at the instant that the request to sign is being made.
4) A Brand reason not to sign. The negative of signing is that the colocation provider will no doubt use this letter of intent to help them sell peering ports, or get the peering switch in place, using the customer network as a lure.
5) IX Not committed. No switch signals that the IX is not committed to public peering. This represents a risk to the ISPs that commit to the switch with the anticipation of a sufficiently large population to exchange traffic with. If they aren’t committed to public peering, why should we take the risk and help them build critical mass?
Conclusion
When these experiences were assimilated and shared with the colocation executive team, the marketing mandate was overridden. Switches were purchased and deployed at all colocation facilities. Most of the switches reached twenty ports sold within six months.
The “Build it and They Will Come” strategy will not always work, but the LOI strategy for colocation operators is a well paved path to fail. During the time the salepeople were asking for LOIs, the customers were going to the competitor that already had a switch and were building critical mass.

Dr Peering
DrPeering@DrPeering.net

How many IXes per metro?

Thu, 27 Aug 2009 09:49:43 -0700

Dr Peering -
What is the optimal number of IXes per metro?
Dan
-----
Dan -

Dr Peering did an informal study with the peering community and they seemed to fall into two camps:
Camp 1 : Only 1 IX per metro. About half of the peering coordinators said that they preferred that there to be only one IX per metro, because more than one would splinter the peering community.
“Which IX to go to? If I go to IX A, I can peer with the population there, but I miss the participants that are only at IX B. If I go to both, I dilute my benefits from peering in that region since it will cost me twice as much. So everyone chooses one or the other or both, and the splintering continues until the two IXes interconnect, which presents a different set of issues.”
“I want to manage redundancy myself in my backbone. IX failures are so rare that I can handle the sub-optimal path during the short period that the IX is down. Peering is merely a local optimization.”
Camp 2: Exactly 2 IXes per metro. The other half of the peering community said that they preferred the resiliency of two IXes in a metro.
“I want two IXes run by two different parties using two different sets of operations practices, different security companies, different vendors for switches, etc. so we get the redundancy and resiliency of two heterogeneous environments. That way we reduce our systemic risk.”
Uniform OPINION: NO MORE THAN 2 IXes PER METRO !
The two camps uniformly said they did not want more the two IXes in a metro since that would splinter the community and provide little incremental benefits from diversity.

Dr Peering
DrPeering@DrPeering.net

Do Content Providers Peer?

Tue, 11 Aug 2009 08:26:44 -0700

Dear Dr. Peering -
I understand that peering is something that ISPs do. But does it make sense for Content Providers also?
Sven Zyborg-Price
-------------------
Sven -
The short answer is that the same economics apply to Content Providers as ISPs : if you can send enough traffic to peers (for free) to offset the costs of peering, then peering makes sense financially.
The same business logic applies as well - is it strategic for you to peer? For ISPs, a core competence is managing traffic. They already have, or are expected to have, a 24/7 Network Operations Center (NOC) and the expertise to chase down routing problems. Most public peering policies have a requirement for a 24/7 NOC and escalation procedures. Is there sufficient benefit to you by controlling your end-user’s experience?
In the Global Peering Ecosystem, the largest Content Providers in the world have deployed large scale backbones for peering away 60%-90% of their content ! Companies like Yahoo!, Google, Electronic Arts, Sony Online, Microsoft, etc. purchase transit at scales large enough to get the lowest possible transit prices yet still maintain that peering is the less expensive and better performing alternative. Most of them claim “It is all about the end-user experience.”
At the same time, peering is not for everyone.
There are at least 3 strong arguments for Content Companies NOT to peer.
1) Transit is Cheap. The cost of Internet Transit has never been lower; the cost savings by peering is minimal to non-existent in many parts of the world. If the last ten years are any determinant, the price of transit will continue to drop. Any cost savings today may disappear tomorrow. Apply the Business Case for Peering.
2) Peering is Not your Core Competence. The expertise for most Content Providers is in creating the content, supporting the user base, building the user base, optimizing the content for search engines, etc. This is a very different core competence than network engineering in the public Internet.
3) Peering is Not Strategic. For most Content Providers, having a reliable connection to the Internet through one or two upstream transit providers and/or CDNs serves their purpose well. Unless you are managing very viral content, content with unique characteristics, need to adjust things dynamically, are blazing a new trail, outsourcing is probably good enough.
In summary, similar logic applies to both ISPs and Content Providers. The rational peering decision always seems to boil down to financial logic and strategic intent.

Dr Peering
DrPeering@DrPeering.net

Who should I send to Internet Peering Events?

Mon, 27 Jul 2009 09:08:19 -0700

Dear DrPeering -
I was just hired to, among other things, offload as much traffic as possible across ‘free’ peering sessions. How can I get peering without coming off like a needy or lame newbie ISP?
Serge Itovic
MikNet
--------
Thanks for that question, Serge; this is well travelled territory. Everyone starts somewhere, and the DrPeering clinic has assisted many ISPs through this process.
There are definitely some do’s and do not’s.
1) DO send your lead techies - these would be your network architects, planners, and engineers familiar with the network backbone and traffic, current and planned. They will interact with the engineers from the other companies, and we have found that the quickest way to gain rapport is for people doing the same job at different companies to talk shop.
2) DO NOT send your sales guys. If you do, keep in mind that Peering Coordinators travel a great distance to meet other Peering Coordinators. They will potentially be wasting their time and the company resources speaking with you, not achieving their peering goal, so be very cognizant of their time and interests, or you will be flagged and not allowed to attend again.
3) DO have a target peer list. It is helpful to show up knowing who you want to speak with, how much traffic you have to exchange today and future, and where you and the target peer are currently and soon to be colocated along with the name of person attending the forum.
4) DO research the target peers public peering policy. It is acceptable to ask about exceptions to a policy clause, but not acceptable to not know the publicly posted policy before asking. (See the ask.DrPeering article about Peering Policy Clauses.)
5) DO start with the low hanging fruit. All DrPeering consultants start clients out working with the Tier 2 ISPs that are easy to speak with, ones with whom peering is most likely to succeed. After five or so assisted peering introductions, the peering coordinators generally get the hang of it and go off on their own to obtain further peering relationships.
6) DO bring lots of business cards. There is no excuse for running out of cards at an event where travel a great distance with the sole purpose of establishing relationships.
7) DO NOT expect signed peering agreements as a result attending peering fora. A reasonable expectation is that your peering team, being exposed to the peering community, is partially inculcated into the peering community. With a handful of business cards, with the initial personal relationships established, peering discussions will be much easier. The Peering Forum will grease the skids towards any possible peering relationship.
8) DO SEND THE SAME PEOPLE. A common problem we see is that companies apply the “fairness” criteria to business trips, and send different members of the team. The problem here is that no one person is seen in the community as a peering person for peering with the company. It takes three or so attendances to be seen as “in” the Peering Community, with relationships with and access to the leaders in the Peering Community. By sending different people each time, you pretty much guarantee your company will always be seen as merely an attendee, rather than a leader or an active member of the Peering Community.
9) DO bring all the resources into informal discussion. You are pitching your case for peering, and the resources you have at your disposal include current network maps, traffic patterns, growth over the years. More importantly, and more persuasively, you have projections and plans. Project the success of your network with realistic timelines and growth trajectories so folks will want to and will plan to peer your traffic at a point in the future.
10) DO leverage Peering Professionals that are already in the community. These people include IX liaisons whose job includes introducing new customers to other peering prospects. There are consultants available to assist in the planning and execution of peering activities as well. Finally, you will find that peering folks are social by virtue of their job, and generally will help each other out; find a person that seems to know everyone in the peering community and ask for some guidance.
Other resources that might be helpful: 
Read about the Peering Coordinator Job
Read what the Peering Coordinators talk about on line and at peering fora
Sign up for the Global Peering Forum
Read more about the Global Peering Community

Dr Peering
DrPeering@DrPeering.net
P.S.
No matter what anyone says, photos will be taken and posted

So you want a Peering Policy?

Mon, 13 Jul 2009 11:42:02 -0700

DrPeering -
How do I create a peering policy?
Does the old phrase “Good artists copy, great artists steal.” apply?
-- Arch Stanton
------
Arch - DrPeering just completed a quick study of 28 Peering Policies. Yes, most clauses in peering policies are similar or identical. Take a look at the groups of peering policy clauses and select the clauses that are most important to you. Also take a look at the text used to see how they phrase things. Finally, take a look at the policies we studies in the list on the bottom of this article.
I think these sample policies will give you a good starting point.

Dr Peering
DrPeering@DrPeering.net
P.S. - It might be fun to create a web app called Frankenstein Peering Policy that allows the user to click and select the clauses and sample text to put into a peering policy. Right now the copying of text takes way too much time.
Introduction
DrPeering reviewed 28 peering policies to examine the language used in peering policies, to see if one could categorize common peering policy clauses. As it turns out, many of the peering policy clauses were very similar if not identical. We placed these clauses into categories, each with their own web page so the community can compare the wording of their favorite peering policy clauses.
http://drpeering.net/a/Peering_Rules_of_the_Road_-_A_Brief_Study_of_28_Peering_Policies.html
Why might this study be interesting? Maybe you want to construct your own Peering Policy and learn from others. Maybe you want to construct a Frankenstein of peering policies - and based on the results of this study, you would not be the first. Peering Coordinators may want to peruse these lists of clauses to see if they should add or change language that deals with issues that other peering policies address well. And for some of us geeky types in the Peering Coordinator community, a few of these clauses are comedy gold.

Findings of the Study
We found generally three groups of Peering Policy clauses:
A) Operations,
B) Technical / Routing / Interconnect,
C) General
Each of these groups had several Peering Policy Clauses that we categorized and/or generalized below.
[Editor’s note: For on-line versions of this page, you can click on the links to see the actual clauses pulled from the peering policies along with their attribution.]
Operations Clauses
24/7 NOC - almost everyone (25 of 28) had this requirement but there were many different ways to say it.
Traffic volume requirement clauses were common (20 of 28) but the least similar - some were 95th percentile measures of minimum peering traffic volume, some average measures, some included measurement in monthly traffic volume, and some specified direction. Some had different measures for public and private, some mandated migration to privates once a volume was exceeded.
Interconnect capacity requirements was a popular (19 of 28) requirement one would expect.
Work to fix things clauses stated that both sides will work diligently, sometimes within a specified time frame. (19 of 28 had this clause.)
Interconnect Capacity, Geographic diversity and Peering in all places in common - these came up in some policies. We blended these together and at least 13 of 28 policies had these..
Traffic Ratio requirements were comparatively with only 9 of 28 having them.
Maintenance and Outage Notification and Interactions for network planning and Monitoring/Managing Interconnect (6 of 28) - focus on the interaction between the two companies.
Escalation Path (5 of 28) clauses specify that both peer will share their contact information and describe how and when they will be provided access to engineers to help solve the more difficult peering and routing issues.
Use of IRR - route registration wasn’t as common as we expected (6 of 28).
Registration in PeeringDB - only 2 of 28 had this. nLayer does not even suggest using it - surprising, since Richard Steenbergen from nLayer leads the peeringDB project.

Technical / Routing / Interconnect Clauses
Consistent route announcements was a common clause (21 of 28). It was interesting that BBC said “Due to the localised nature of BBC content, we reserve the right to advertise a different set of prefixes at each location.” All other clauses were requirements for consistent announcements.
“Hot Potato” or “Shortest-Exit” clauses came up (8 of 28).
MEDs don’t seem to be widely used (2 of 28 mentioned them). When they are discussed in Peering Policy clauses, they tend to say that MEDs will be ignored or that they require negotiation.
MD5 was required by only a few: AboveNet, BBC, wbsconnect, and Charter
Don’t Abuse Peering - was a popular clause (18 of 28 had some of these clauses). Here we bundled a list of clauses such as no pointing default, static routes, selling, bartering or giving away next hop, leaking routes, etc.
Filtering clauses, Prefix Length minimum clauses came up along wth a minimum number of prefix or ASes to announce clause - these and Single AS (8 of 28) requirements clump together.
Provide us with tools clauses - in some cases the ISP required access to in-network tools to diagnose/check routing.

General Clauses

Can’t Be Customer - (18 in 28 had this clause) - in some cases you can’t have been a customer for some period of time, and in some cases the policy states that you can’t be a customer of a customer or of a peer.
Peering request clauses were very common in peering policies (17 of 28 mentioned how to request peering.) But there was variance here: some added how often you can request peering, and what information needed to be included in peering requests.
Peering may be suspended, terminated, and we can make exceptions at will. In here we also categorized the clauses about “meeting these requirements does not guarantee peering.” At least 15 of 28 policies had these clauses.
Paid Peering product was mentioned by 3 of 28 as an alternative for those who could not meet the peering requirements (Comcast, AT&T, Cox, tinet).
Peering in Reciprocal Markets - this clause is new and as a result rare (2 of 28). With this clause, the peer agrees to peer in both foreign and domestic markets. DrPeering wonders if Comcast has International desires with their “Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI arrangement with Comcast in the Applicant’s home market.” (It is common for a Tier 1 in one Internet region to be much more open in a foreign Internet Region. They would deny typically peering in their home market.)
Non-Disclosure Agreements and/or Peering contracts were required by 9 of 28.
This Policy May Change with some notice (10 in 28 had this one).
Financially Viable clauses showed up in 2 of 28 policies.
Summary
This brief review of 28 peering policies highlighted a great deal of commonality among policies. One can break down these policies into common groupings, and then further into clauses within the groupings to understand the nature of peering policies.
In some cases the peering policy clauses are identical, in most cases there is some commonality. It seems that many of these peering policies have a common lineage, that is, descended from the same source or from a recognized common issue.

So you want to tour an Internet Exchange Point

Mon, 6 Jul 2009 10:26:37 -0700

Last week, the ask.DrPeering.net clinic took a tour of the Palo Alto Internet Exchange (PAIX)¹, one of the oldest and best connected Internet Exchange Points in the world. A couple newbie guests accompanied us and reminded us that everyone goes through the same FIRST Internet Data Center Tour Experience...
The problem everyone faces on their first Internet Data Center tour is that
1) you are entering into a cold and noisy environment with cages (enclosing equipment that doesn’t appear to have any interest in trying to escape), and
2) you are then being shown banks of tall beige or grey boxes that you have never seen before and will never have to understand how to operate, and
3) you are told no less than 50 discrete pieces of acronym laden information that may or may not be relevant to your use of the facility.
You leave overwhelmed from your first tour, unable to explain what you saw. There are no “You are Here ᐅ” maps to provide context.
All of this expensive equipment simply controls the environment to make sure that the power never goes off, and that the equipment is all properly cooled.
What helps us more seasoned folks is the conceptual model in our heads of what Internet Exchange Points have. This model conveniently helps us place the information we hear into a contextual map of what is going on here.
So, for the friends of DrPeering (all of you reading this), we have created a very rough work sheet to take along with you on your data center tours.
This will give you a sense for the kinds of statements/issues that will come up and where to place the answers to those questions in the model. There are lots of in depth articles about data centers on the net - you will find as much detail as you want about all the equipment you will see and the tradeoffs data center architects make.
So print out this “Data Center Tour Companion” for your first tour and see if it helps at all.

Dr Peering
DrPeering@DrPeering.net

¹ When AboveNet purchased the PAIX from DEC it duplicated the PAIX model and rebranded PAIX as Public Access Internet Exchange to remove the Palo Alto centric nature of the name.

Open Peers?

Wed, 24 Jun 2009 11:34:39 -0700

Dear Dr Peering,

How on earth can you promulgate that anyone other than a couple content
or anycast networks are "open". The linked-to policies have requirements
that are equal to some of the selective: WISC doesn't peer with customers,
even peer1 has capacity requirements, Bouygues calls themselves selective,
CAIW limits prefix count, steadfast has the basic 'selective' elements as
does NAC.

Is it just because they use the word open? Init7 is the only 'open' one with
no requirements listed.

I honestly don't get why this is a drum beaten so often.

Joe

-----
Joe - thanks for raising the debate about what being an “Open Peer” means.

Brokaw Price from Yahoo! coined the term
“Pulse Peer; if you have a pulse, Yahoo! will peer with you”.
But what if Yahoo! requires a peering prospect to repair peering problems on a timely basis? Does that requirement project Yahoo! from the “Open” peering classification into the “Selective” classification?
Another example that comes up: some “Restrictive” peers such as AT&T believe that they are more accurately classified as “Selective”, since they accepted new peers recently, demonstrating their willingness to peer.
Peering Inclinations tend to follow a continuum from “Open” to “Selective” to Restrictive” as shown in the graphic below. (The “No Peering” inclination is generally used to describe content companies that do not peer.)

What to look for in an Internet Exchange Point (IX)?

Thu, 18 Jun 2009 10:45:31 -0700

DrPeering -
What do people look for in an IX?
Andy Davidonson
-----------
Thanks for that question, because DrPeering has an answer based on extensive research.
The ISPs over the last decade shared nine issues they explore to select the right Internet Exchange Point for them.

European vs. U.S. Internet Exchange Points

Thu, 4 Jun 2009 10:09:46 -0700

DrPeering -
How are Internet Exchange (IX) points different in the U.S. versus Europe?
I.P. Godéss
-------
I.P. -
Good question. European IXes and U.S. IXes tend to differ in:
1) Organizational Structure
2) Scale
3) Pricing
4) Business Model
5) Colocation Neutrality
6) Cooperation
Other than that, they are identical.

Google Net Neutrality?

Fri, 29 May 2009 13:48:43 -0700

DrPeering -
What did you think of Google I/O 2009?
-- John Furrier

I was struck by something Google didn’t mean to present. Let me explain.

How Equinix Beat MAE-East : IX Playbook Tactics 8-11

Sun, 10 May 2009 06:12:41 -0700

Dear Dr Peering -
MAE-East was the dominant IX in 1999 when Equinix started. Today Equinix Ashburn is the dominant player. How did Equinix beat MAE-East?
Steve Weldman
Steve -
This is a good chance to go back in time and demonstrate a few of the IX Playbook tactics discussed last time, and highlight a few additional ones, starting with an attempt to cooperate.
Tactic 8 - If you can’t beat them, join them
When Equinix Ashburn was constructed (1999) it had no customers and faced a formidable, well entrenched dominant Internet Exchange called MAE-East, an IX run by WorldCom since about 1994. Everyone knew of MAE-East - it was the place you had to peer on the East Coast if you wanted to be seen as a big player ISP. How does one present an IX with no customers as an alternative to one that is well past critical mass?

The Art of Peering: The IX Playbook Tactics 1-7

Thu, 7 May 2009 17:17:26 -0700

Dear Dr. Peering -
My question is related to your article “Should I build an IX?” by Dr Blessing.
How does an IX get its first customer? How did Equinix become a powerhouse IX?
Roy Terrice
-----
Roy -
Those are good questions, and by that I mean that I already have graphics that help answer that... In fact, I have an active (evolving) research article that I use for consulting gigs: “The Art of Peering: The IX Playbook”. It documents the ways IXes build critical mass, and defend it against competitors. It is based on dozens of conversations over beers with some of the smartest IX operators in the world. I’ll share some of the highlights with you here today, and eventually get the whole thing onto the DrPeering website.
Let’s start with a little theory and then apply the theory with a handful of the “Tricks of the Trade”, the guerilla marketing tactics an IX uses to build critical mass and then defend it against competitors.

Transit Prices Race to the Bottom

Tue, 28 Apr 2009 09:54:00 -0700

Dear Dr. Peering -
I read about all of these video deals that Level 3 is getting I am curious about the coopetition that happens between the Tier 1 ISPs now. I am curious about the continually dropping transit prices - is this a race to the bottom? Does this lead to infighting in the Tier 1 club?
Just curious.
George
---------------
George -
Yes - Dr. Peering hears that it is getting really messy out there. Imagine the pressure to deliver revenue in a recession, losing deals to your competitor, but still having to carry the bits for free. But we are getting ahead of ourselves. Let’s build a foundation for discussion by describing the competitive landscape and a hypothetical deal that the Tier 1 ISPs compete for.
The Race to the Bottom
You have probably read the DrPeering page discussing the Internet Peering Ecosystem, so you know that the Tier 1 club in any peering ecosystem is a good place to be. As a Tier 1 ISP, by definition, you can reach all destinations in the Internet Peering Ecosystem (North America for example), solely through your free peering relationships. You get money from your customers but don’t have to buy transit from anyone.
See http://drpeering.net/a/Peering_vs_Transit.html for the current market prices for transit and a discussion of peering as an alternative.
Traffic exchange for Tier 1 ISPs is not free; there is a capital and operating cost of operating a large scale network and large (100’s Gbps) of interconnects. However, all traffic (and the corresponding revenue) that is not peered away within the Peering Ecosystem eventually trickles up to the Tier 1 club. For everyone else, there is a transit pricing floor that you can charge your customers that must be above the price your Tier 1 charges you.
But for Tier 1 ISPs, the price floor is the cost of getting the bits to the end users. That would be the cost of getting those bits across your network to your customers, or the cost of sending them across the room to your peer who has agreed to carry those bits for free. It is nice to be in the Tier 1 club; it makes the business math much easier.
Wholesale vs. Retail
So the Tier 1 ISPs sell wholesale transit to Tier 2 ISPs who in turn sell to the market.
But for the large deals (75Gbps+), the Tier 1 ISPs generally compete aggressively ($2.50-$2.75) to get the deal and don’t generally lose to Tier 2 ISPs that they sell to.
For these deals, the content provider generally doesn’t want to take chances either. They have a lot of traffic and money at stake.
So this deal is 75,000 Mbps * $2.50/Mbps=$187,500 per month or
this deal is 75,000 Mbps* $2.75/Mbps=$206,250 per month of new revenue.
Let’s look at the competitive fight for this deal.

Dr. Kuku on Circuitous Internet Routing

Tue, 21 Apr 2009 06:12:35 -0700

Dr Peering -
I’m curious about the circuitous path that traffic sometimes takes; I thought routing in the Internet used some form of shortest-path routing. What is going on here?
-- Cheryl Ladd

Greetings Cheryl -
Well, I have to tell you. On my many voyages,
I’ve circled cyber space 42 times,
battled the New World Order, and
avoided alien abductions with this:
my Thought Screen Helmet
in order to ensure I could provide you with these,
wonderful, wacky, and weird web sites to answer your silly question.
But first, thank you DrPeering Clinic for finally allowing me to highlight all the good work at http://stopabductions.com/ - Visit the website and learn how to make for yourself a thought screen helmet that has proven so successful in avoiding alien abductions.
Now, I know what you must be thinking. Will it work for the alien-human hybrids that have been integrated into our societies? Sadly, no - they will be abducted and the thought screen helmet can not protect them.
But for the rest of us, $30 is all it costs to make your own thought screen helmet.

Oh, Elusive Paid Peering

Sat, 18 Apr 2009 08:22:25 -0700

Dr Peering -
What is “Paid Peering” and when does it make sense? Why won’t anyone talk about it?
Gavdi Enron
------------------
Mr. Enron - Good questions.
DrPeering has been searching far and wide for evidence of Paid Peering, from the well connected peering ecosystems in the U.S. and Europe, all the way into the deepest darkest parts of remote peering ecosystems in south east asia. There have always been rumblings, but no one was willing to talk about it and be cited.
Let’s first remember that Internet Peering is defined as a reciprocal exchange of access to each others’ customers. This is typically a free relationship, as both parties feel they are getting approximately equal value in the deal.
Next, recall that Internet Transit is defined as a business arrangement whereby on Internet Service Provider (ISP) provides (usually sells) access to the global Internet. Here there is a clear customer-provider relationship; the customer pays who can get traffic onto the Internet.
These working definitions were more or less easy to observe, document, verify, and validate with the Peering Community. But Paid Peering has remained elusive; veiled behind NDAs and the closed doors of peering discussions. Over the last ten years Dr. Peering has spied some paid peering patterns of behavior in the ecosystem.
Paid Peering Discussions
Here is the typical sequence of events that lead to Paid Peering Discussion.
Peering is requested. To get free Internet Peering, two ISPs explore the mutual benefits of the proposed peering relationship. Network maps and mtrg traffic graphs are exchanged. If one or the other party feel there is not mutual benefit, if they are not indeed ‘peers’ in the value they bring to the relationship, and the peering doesn’t typically happen.
So, Peering is denied. Now, in some cases, the party denying the peering request suggests that the denied party buy transit. In some cases, the party denying the peering request suggests that the denied party buy “Paid Peering”, which is exactly like Internet Peering except that some money is exchanged.
In both of these cases, the denied party saw themselves as a peer, but were for some reason rejected.
When peering is denied, it is not surprising that
1) Neither party wishes to talk about the denied peering event. (No one wants to be seen as a network not worthy of peering, nor be seen as the “bad guy”.) There are the occasional whiners who complain publicly in the hope of getting sympathy, but Dr. Peering has never seen this leading to the peering that was originally sought.
2) The denied party feels slighted, and maybe even the employees of the company feel somewhat inferior. There may be anger, embarrassment, disappointment, resentment, tears in beers.
3) The denied party sometimes responds to the suggestion to “buy transit” or “buy paid peering” with one half of the victory salute. (For those who never heard this expression, it is an obscene gesture.)
Sounds like dating, no? Dr. Peering thinks so.
But Paid Peering may be emerging now as a product that makes sense for a particular segment... but more on that below.
Let’s speak to your next question...
Why don’t people talk about paid peering?
Those who sell paid peering will talk about it, but
1) You have to find the right people...They often don’t have a dedicated commissioned sales team for the product. Sometimes, this is because they offer the product as a stepping stone to free peering, once the prerequisites are met.
2) They get higher margins by selling transit, a well known product within the company. The margins and therefore the commission for the salesperson who bothers to learn about the product may be small, and the complexity of the sale may be greater.
3) They offer it at a price higher than the market price of transit. ATDN was famous in the Peering Community for this. The resulting “Are you crazy offering it at that price” doesn’t lead to many sales, so there is a disincentive to bringing it up. The counter argument is that the quality of the end-user experience is better with peering, so those that care about that will pay.
For these reasons, we haven’t seen a lot of force behind the paid peering initiatives in the peering ecosystem.
It is worth pointing out too that those who buy it generally don’t talk about it because by doing so, they are admitting that they are not peers with their paid peering partner. You sometimes see Non-Disclosure Agreements in place with Paid Peering Agreements as a result.
Why don’t all ISPs offer a paid peering product?
Dr. Peering asked many telcos about this and the answers were pretty consistent.
The most common answer was that it was too much effort for the telco to launch a new product, especially when it competes with a product they know and sell today (Internet transit).
The second most common reason not to offer paid peering was that there would be too much cost to offer the new service, relative to the margins they should be able to get.
Consider all the internal new systems to sell, monitor and debug, train for, pay commissions on, report on, etc. All of this cost, for a service that the market believes should be priced at less than the already low margin transit service.
These companies told paid peering prospects to buy transit and filter out the routes that are not wanted, as illustrated to the right.
When does Paid Peering make the most sense?
The best case for paid peering is starting to appear today: high-speed, high capacity last mile access.
Comcast for example has a paid peering business arrangement where companies can directly connect their content or CDN services into the Comcast last miles, and those doing so reported to DrPeering that they are getting 3 times better performance as a result. The better performance is compared against sending traffic through a potentially circuitous path through potential many ISPs that are managing their traffic to maintain their peering ratios.
Paid Peering with last mile may be the most effective weapon for Content Delivery Networks. Rumors are that the price of paid peering with Comcast is very competitive when compared against transit. DrPeering suspects Comcast will succeed in helping this industry uncover and embrace the Elusive Paid Peering.
End Notes.
There is actually another option seen in the wild along the continuum of alternatives between peering and transit: partial transit. Partial falls into three categories - exchange X only, region X, all peers
The first example of Partial Transit is when an ISP sells access to the routes it learns at an Internet Exchange. They can say “Hey, you are thinking of joining the GINX, but you will have to pay the membership fees, the colo costs, etc. and then establish all the peering relationships. Instead, I will deliver to your router all the routes I learn from my long lived presence there, all for less than you participating there yourself.”
The other two partial transit examples are similar but for the regional routes, and for peer routes - only routes learned from the peer’s peers.
James Blessing (consultant, Garou Ltd) points out “networks offering partial transit can have problems with their peering arrangements if they don't make sure they are in line with peering arrangements that include 'universal announcement' clauses."
He goes on to say “Another reason for paid peering is that as you are a contractual
partner of the company (i.e. have a contract with monetary values) its easier to get
capacity issues and network problems (such as ddos mitigation) addressed quickly.”

“For filtering of routes from a transit feed you really need good quality (documented
and maintained) community support within the transit supplier for this to work - is there
an article on the benefits of communities for the peering community somewhere?”

“As a well designed and maintained community structure can make sure that path
selection can take geographic impacts into account - i.e. not having traffic between
two uk networks travel to de-cix to be exchanged.”

Will Hargrave points out that the AS-PATH will be longer in the partial transit solution than the direct peering interconnect, and that it will be more difficult to migrate to a more fully peered network.
Pricing Paid Peering. The metered price of paid peering and partial transit has generated a lot of heat. Many believe paid peering is a subset of full Internet transit, and therefore should be priced a a fraction of the price of Internet transit. Others believe that peering provides greater performance and therefore ought to be priced higher than Internet transit.
In 2009, in the market where Internet transit can be had for $6-$9/Mbps, paid peering seems to be pricing out (and catching on) at $1-$3/Mbps.
Some Notable Paid Peering Providers
AOL http://www.atdn.net/paid_peering.shtml
Comcast http://www.t1r.com/client/view.php?rid=56457
Verizon http://blog.streamingmedia.com/the_business_of_online_vi/2009/01/verizon-cdn-announcement.html
References
Peering Coordinator Panel ; 28-Nov-2007 http://www.peering-forum.eu/epf2/presentations/7.PeeringPanel.pdf
AOL Paid Peering POP and Locations http://www.atdn.net/paid_peering_locations.shtml
Verizon Port Partner Program - Just Smoke? http://www.telecomramblings.com/2009/01/verizon-port-partner-program-just-smoke/
Net Neutrality and the Carrier Hotel - CRGWest - http://www.crgwest.com/PDFs/NetNeutralityandtheCarrierHotel.pdf
asr’s Paid Peering Blog http://www.voxel.net/blog/2009/01/paid-peering/
Complexity of Internet Interconnections: Technology, Incentives and Implications for Policy http://people.csail.mit.edu/wlehr/Lehr-Papers_files/Clark%20Lehr%20Faratin%20Complexity%20Interconnection%20TPRC%202007.pdf
Historic Documents on Peering http://drpeering.net/a/Papers.html
Market Transit Prices: http://drpeering.net/a/Peering_vs_Transit.html
Transit versus (Paid) Peering: Interconnection and Competition in the Internet Backbone Market http://www.econ.ku.dk/cie/Seminars/pdf%20-%20seminar/Jahn-Pruefer_-_Peering.pdf

Handbook of Research on Telecommunications Planning and Management for Business
http://books.google.com/books?id=8Py2YnZX_h8C&pg=PA143&lpg=PA143&dq=%22paid+peering%22+internet&source=bl&ots=p6GVALoIiW&sig=RVIKdD7B8QEyw7SoDMeTy7hC0lA&hl=en&ei=PyXeSeWQIJS8M42mnE4&sa=X&oi=book_result&ct=result&resnum=9

Ramesh Johari CS 40 Lecture 3: Internet Economics: https://ccnet.stanford.edu/cgi-bin/course.cgi?cc=msande40&action=handout_download&handout_id=ID117720276518369

Marty Hannigan’s Simple IP Transit Request for Proposal http://world.std.com/~hannigan/hicap-rfp.pdf

Dr Peering
DrPeering@DrPeering.net

Should I Build an IX?

Fri, 10 Apr 2009 16:09:17 -0700

Dear Dr Peering,

I want to build a peering exchange in my local city; do you think that this is a good idea?

Hugs and Kisses

Hope Phule
Bedroom ISP LLP

--

Dear Hope,

Whether you build an exchange or not depends on a large number of variables, but before we go onto those topics have you ruled out the following two:

1. Join an existing exchange in your city, most Internet Exchanges have been around a few years and have built up both a considerable amount of experience and (probably more importantly) a wide range of existing customers. This would give a fast route to establishing peering as other networks will have already built to this location and have infrastructure in place.

2. Join an existing exchange but not in your city (or even country) - see ep.net for a list of IXes and the Ask DrPeering article “Value of an Internet Exchange” for a guide to calculating the value of various exchanges. Remember that you will need to change the relative variable to suit your network type and properties

If neither of these is cost effective then you have other options before hitting the DIY route:

3. Talk to an existing IX operator about extending their exchange to your local datacenter (as LINX [http://www.linx.net/] have recently done in Slough [ http://www.reuters.com/article/pressRelease/idUS166839+29-Jul-2008+BW20080729 ]). For this to be cost effective you will need to be able to provide a list of local networks that are willing to join the exchange to justify the cost of the extension.

4. Talk to an existing IX operator about running an exchange on your behalf (as AMS-IX [http://www.ams-ix.net/] do for CAR-IX[http://www.car-ix.net/]). Again you will need to get the business case for the build but you have removed the technical and majority of the political issues around the operation of the exchange in a single step. Depending on the location of the exchange you might be able to get local/regional/national/European government funding for the setup costs.

If neither of these works then it’s up to you as to what to do next:

5. Create a community run exchange owned and operated by the participants is a fairly straight forward option as all you really need is a switch and a couple of volunteers. An exchange of this type is unlikely to attract the larger networks until it reaches a critical mass but if most of your traffic is local to the local area its likely to be worth the investment of time and effort. This option might also be able to attract funding but that would require that the project is formalized into a non-profit making organization with a proper company structure (this is how LINX and AMS-IX operate today).

6. Starting a IX business with a formal business plan and funding (either from a group of individual companies or private investment) is the hardest of these options but is still achievable (even in difficult financial times businesses that have a solid foundation will survive and grow). The key to this is researching the potential of your location before creating a business plan and getting networks to commit in principal BEFORE you start spending any money on hardware or facilities

7. The 'Ray Kinsella' [http://en.wikipedia.org/wiki/Field_of_Dreams] approach to building an Internet Exchange is not something that Dr Peering would recommend....

If you do decide that building an exchange is for you then have a look at [http://www.euro-ix.net/ixp/startingixp/] and make sure that you talk to members of the Internet community who are usually happy to provide helpful advice and information (frequently in exchange for beer, wine, tequila and/or scotch).

Dr Blessing

Dr Peering Clinic

--

James Blessing is an Independent Consultant providing network strategy and design services for ISP’s, Datacenters and CDNs via Garou Ltd [http://www.despres.co.uk/]

A Business Model for a Cooperatively Managed CDN (v0.91)

Tue, 7 Apr 2009 10:42:30 -0700

A Business Model for a Cooperatively Managed CDN (v0.9)

Abstract
This paper is a starting point for an entire business model ecosystem designed to distribute of latency and loss sensitive Internet objects. For a set of premium content producers, there is a willingness to pay a little more for a guaranteed high-quality end-user experience. For some Internet Service Providers, there is a willingness to expend resources if there is a corresponding increase in revenue. What is needed is an ecosystem that supports both of these goals.
An overlay network called a “Media Grid” is introduced in this paper. A revenue allocation model is introduced along with two scenarios; before caching, and when the object is served entirely out of last mile cache. These scenarios are discussed with the goal of aligning the interests of all of those participating in the cooperatively managed video delivery ecosystem.
Model 1: Traditional Cable Distribution
Paramount offers an early release Pay-Per-View Movie Premiere. Let’s assume that Paramount advertises an opening night pay-per-view movie for $13 on the cable system or on a subscribers Internet service, and the end-user clicks on the movie to purchase it. (The numbers used in this paper are used solely for illustration purposes and do not represent actual negotiated rates).
With the traditional pay-per-view cable revenue split model, Paramount would typically receive of the revenue from the movie, and the rest would go to distribution chain (the cable company), about per video.

In this model, the cable company essentially resells the studio content, so the studio doesn’t have a direct relationship with the customer. The studios said there were a couple reasons for this:
1) They want a bigger piece of the pie - 50% is a bit low, and
2) They want to market related products to their customers (screensavers, video games, bonus features, sequels, fan sites and subscriptions, etc.)
So these studios are seeking a more direct end-user relationship, using an Internet portal or a proxy (Hulu for example) to get their content to households. Let’s explore the revenue flow in this best-effort Internet distribution model.
Model 2: Best Effort Internet Distribution
Over the best-effort commodity Internet, the studio pays for Internet access from an Internet Service Provider (ISP) and/or a CDN paying on a metered basis. The upstream ISP or CDN would in turn pay for distribution as needed to their upstream providers along the distribution chain for best effort service. The revenue flow might look something like this:

After the end-user clicks to buy this $13 (4GB) movie, the packetized movie goes from the Paramount Internet channel web site to its upstream ISP (Tata) at a metered rate of $8/Mbps. Tata in turn sends that traffic through its peer (AT&T) for free, who in turn transports those bits to Cablevision (charging Cablevision $6/Mbps), who in turn charges a flat rate to its customer of $20/month. [In the diagram, the revenue figure is shown closest to the party receiving the revenue.]
Let’s assume that the movie is 4GB in size, and at per Mbps it costs per GB, or per movie to deliver. Under these assumptions, distributing this movie over the commodity Internet provides Paramount with profits of -=per movie as compared with per movie of revenue under traditional revenue sharing model.

So there is a larger profit margin here - this is certainly an interesting proposition but in reality is fraught with challenges.
First, the best effort Internet service may be good enough for highly compressed standard definition video distributed under no-loss conditions, but for large long-lived streams under conditions of network stress, the end-user experience can be perceived as quite poor. From the content provider perspective, anything but a perfect end-user experience is unacceptable, as it tends to result in increased credit card charge backs, loss of goodwill and loss of revenue.
Second, with free peering relationships specifically, and with the global Internet in general, there is often little incentive for upgrading the infrastructure that does not directly lead to incremental revenue. This lack of alignment is one reason why we see congested peering links and so many inter-provider peering disputes.
Finally, the best effort transit service does not scale well for content with unpredictable demand characteristics. Content Distribution Networks scale much better but can only serve up content when their path is selected. We have heard stories of ISPs preferring a more circuitous traffic path to maintain their peering traffic ratios with their peers. As a result, the shortest path from content to a CDN is sometimes not used to the frustration of the content provider and eyeballs alike. To solve this problem, one needs to actually embed the content within the last mile.
To fully realize the incremental revenue from direct over-the-Internet video sales, and to avoid these issues, producers of premium content recognize that it may make sense to pay a little more money for a better end-user experience.
Model 3: Media Grid Overlay Network
In this final scenario, Paramount is willing to pay Tata a little extra for a high-quality “Media Grid” service, a kind of “diamond lane” for the Internet. Let’s talk about how this might work.
Let’s assume that Tata offers Media Grid Service to Paramount for a fee of 75 cents per GB, or $3 in total to deliver the movie over the media grid as shown below.

From the Paramount perspective, it is still more profitable than the traditional cable company revenue share model.
Let’s take a look into the Media Grid Overlay network to see how the packets and revenue flow.
Traffic and Revenue Flows
Media Grid. The “Media Grid” is a global ecosystem consisting of network devices embedded within the networks between content provider and end-user. Each of these embedded network devices are called “Media Grid Engines”, and are dynamically connected to one another across the underlying networks. Think of the Media Grid overlay as a cooperatively managed Content Distribution Network, with guaranteed, measured and audited service characteristics, and with each participant having a financial stake in the successful delivery of their premium content.
Traffic Flow. In the Media Grid model, the traffic is delivered from Paramount to the first Media Grid Engine (the origin server) in the path, and from there, across to the Media Grid Engines along the chain of intermediary ISPs. When the traffic reaches the last media grid engine, the content is buffered (and potentially cached) for delivery to the eyeballs that ordered the movie.
Revenue Flow. The revenue (for successfully delivered content) is distributed across the operators of media grid engines in the path.
How is the revenue allocated across the participating members of the ecosystem? There are many possible solutions for the community to discuss – here is one possible solution.
Scenario #1 – Cascading Revenue Flow

In the scenario, Tata charges its customer $0.75 per GB, or $3 for the media grid distribution of its premium movie. Tata has in turn negotiated Media Grid Services from its peer (AT&T) for 50 cents per GB, or $2.00 per movie. AT&T has negotiate its Media Grid Service with Cablevision in this example for 25 cents per GB, so AT&T pays Cablevision $1 per movie. And finally, the video is delivered to the end-user.
The Media Grid Measurement and Settlement System operator (shown on the bottom of the diagram) receives $0.06 per video stream from Tata in return for monitoring the Media Grid Engines and generating the settlement statements on behalf of the participants in the ecosystem.

Implications. We accomplish the goal of allocating revenue to where the infrastructure is most heavily used. Tata gets a chunk of the revenue for sourcing the deal to the media grid, and AT&T and Cablevision gets incremental revenue for delivering the content over their diamond lanes.
The interests and incentives here are aligned; Tata, AT&T and Cablevision all receive incremental revenue, but only upon the successful delivery of the content. If there are problems along the path, all are incented to fix them in order to gain the incremental revenue.
Optimization. In a live streaming scenario, all participants are participating in the cost of delivery, but in a static movie download scenario, the content can be cached. When the content is cached in the Cablevision Media Grid Engine for example, the load on Tata and AT&T is potentially zero. Should AT&T and Tata get their chunk of the revenue if they don’t do anything towards the successful delivery of the movie?
Let’s explore a straw man revenue model for dealing with cached content delivery.
Scenario #2 – Revenue Flow with Cache
In this scenario, revenue is proportional to the cache hits; everyone along the path is compensated for the first distribution, and after that, only when their caches are hit. This incents big caches at the edge for popular content, and the widest dissemination of caches to increase cache hits and corresponding revenue.
Let’s walk through the example to demonstrate this.
Traffic Flow. Assume that Paramount has already uploaded the video to the Tata Media Grid Engine (origin server). Assume further that the first copy of the video has propagated to the last mile Media Grid Engine (cache).

Revenue Flow. In this model, the origin Media Grid Engine operator (Tata) charges Paramount for each movie and the last mile operator is paid the remainder for serving the content to the eyeballs on behalf of Tata. Since AT&T has no cache hit in this scenario it doesn’t receive any revenue, and incurs no incremental bandwidth cost.
Implication. There is $0 load on Tata, so why does Tata get a chunk of revenue? Two reasons. First, Tata is operating the origin server (which provides the server of last resort), and second, for originating the customer content onto the media grid. Tata again pays the Streaming Settlement fees of $0.06.
The last mile operator gets the largest chunk of revenue for serving the content out of their cache, thus incenting them the most to increase the performance of the last mile for their share of the associated revenue.

Discussion for the May Meeting. We need to discuss the mechanism for how the market determines the allocation of revenue between the origin and the last mile cache. One system would have Tata getting the same amount for every successful movie, and the last mile gets everything left. Another would be for the last mile guys to get the same amount no matter what, and the originator of the sale gets everything else. The goal needs to be an open market solution where the dollars flow to bottlenecks.
Guaranteed Delivery
There are several mechanisms to provide guaranteed high quality service to the customer base.
First, since the popular content will most likely be cached in Media Grid Engines located within the last mile, the content has fewer router hops to travel, so fewer places in which congestion may occur.
Secondly, we expect the last mile providers will be motivated to upgrade their infrastructure (or are already upgrading their infrastructure) in order to realize the incremental revenue described here.
Third, there is real incentive to prioritize these high-priority bits that will generate incremental revenue over those that are sold today as part of a best-effort Internet Service. We expect queue prioritization and similar solutions will be employed in hardware to maximize the likelihood of realizing the incremental revenue.
Finally, there are mechanisms within the Media Grid Engines to strategically time the injection of traffic such that it misses the encountered congestion along the path.
Details of these mechanisms can be seen at http://www.interstream.com/wiki
Net Neutrality?
Do these mechanisms for prioritizing traffic and missing the bottlenecks create a clash with Net Neutrality advocates? Not really.
There are several definitions of Net Neutrality. We will use the definition that Net Neutrality means that any prioritized service should be priced and made available in an open, transparent, and non-discriminatory way. By working with the ecosystem participants, the media grid association defines and stands behind the association defined prioritization system. This is a much stronger and more defensible net neutrality stance.
Streaming Settlement Company
The machinery to make this work is behind the scenes, a kind of Visa measurement, auditing, and settlement and archival system for Internet streams across the media grid.
Inside each Media Grid Engine are meters that constantly monitor and report the performance of the streams going through it. The results are fed back to a settlement system that issues periodic settlement reports to all intermediary ISPs. In this scenario, Tata would purchase streaming media credits from nuMetra for a price of say 6 cents per stream in return for nuMetra handling these settlement functions.
The InterStream Association is the participant body that directs the Media Grid ecosystem, establishing and evolving settlement rules, inter-provider performance standards, equipment certification standards, archival system rules, equipment upgrade and maintenance schedule, etc. for the operation of the ecosystem. It is within this open industry forum that the media grid ecosystem will evolve.
Summary
The Media Grid Ecosystem as described provides content providers with “diamond lane access” for its high quality streams. Mechanisms are intrinsic in the system to incent all players in the path of the streaming to upgrade their infrastructure to gain incremental revenue.
The big winners in this effort are the last mile providers, who, as a result of caching, play the strongest role in the collective ecosystem, and will realize the most incremental revenue as a side effect.
But really, the entire ecosystems wins since there is at last a way for those willing to pay for high quality access to the eyeballs, to get that access as required by their application, and those who provide their bandwidth to this end to participate in the revenue flow.
Comments from the field
This discussion paper describes an ecosystem in general terms, using illustrative numbers to best illustrate the ecosystem. Readers suggest:
1) It would be interesting to model the ecosystem using actual market data.
2) It was recommended that the paper include a sample settlement statement highlighting all costs and revenues generated by the system on behalf of the players.
3) It would be helpful for the operators to see what happens if they failed to upgrade their infrastructure or prioritize traffic. “By not upgrading this OC-12 last month we failed to get $12K in incremental media grid revenue.”
4) It would be helpful to map out the open source software that would be used, what would need to be developed and the costs associated with the integration.
5) It would be good to map out the operations activities and the roles and responsibilities of the ecosystem members.

Acknowledgements
This paper is the result of a lot of thinking by Jeff Turner, Scott Landman, and the team at nuMetra. Great feedback and suggestions came from countless anonymous contributors and many folks in the peering community, Keoki Andrus, Ken Florence (NetFlix), Peter Harrison, Ren Provo, Richard Steenbergen (nLayer), Barrett Lyon.
About the Author
William B. Norton is the Mediated Bandwidth Working Group Chair for the InterStream Association and the founder of DrPeering.net.
From 1998 to 2008 Mr. Norton served as Co-Founder and Chief Technical Liaison where he was instrumental in getting the Tier 1 ISPs, the cable companies and some of the largest content providers in the world into peering at Equinix.
From 1988 to 1998 Mr Norton served many roles at Merit Network, including leading the Internet Engineering teams and chairing the North American Network Operators Group (NANOG), the Internet Operations forum for North America.
Mr Norton received his MBA from the Michigan Business School and holds a Computer Science degree from Potsdam College.

Predictions for the Internet 2014

Mon, 9 Mar 2009 12:23:41 -0700

Dr Peering -
What does the future hold? Will I be able afford to pay my nanny’s parking and towing fees?
-- Unther Todderwood

Unther -
Dr Peering can only share some data points collected from the field.
It was a long bus ride through the jungles of the Dominican Republic going from the North American Network Operators Group (NANOG) meeting to the Global Peering Forum. Some people slept, some people discussed politics. Some people drank. And some of us predicted the future of the Internet.
You see, my feeling is that if enough people answer this question, people with different but experts vantage points of this future, we can use this collective wisdom to explore possible futures. Then we vote on which of the possibles is likely to happen. Those with the most votes, win.
Is this a guarantee? No, but you can see how many of the predictions came true when we played the same game back in 2006.

Dr Peering
DrPeering@DrPeering.net

Value of an Internet Exchange?

Wed, 18 Feb 2009 10:35:12 -0800

Dr. Peering –

What is the financial value of an Internet Exchange (IX) to its participants?

Thanks –

Stu Dent
------------

Hi Stu -

Tough question but let’s take a swing at it.

Let’s start the analysis by making a three simplifying assumptions.

First, the value an Internet Exchange point provides is facilitating the peering of traffic between participants. There are at least two dimensions to the value of peering: the value of the traffic volume peered, and the routes announced at the IX. We will only focus here on valuing the volume of traffic in this article; the routes announced deserves its own analysis.

Second, it is important to note that several large IXes have both public peering fabrics and private peering services. Private peering is traffic directly exchanged between two parties, often over a physical piece of fiber or a virtual private service. Being able to do both public and private peering represents a significant value of the IX, but unfortunately the traffic peered privately can’t be factored into the analysis since there is no visibility into the amount of traffic exchanged privately. For this reason, we will have to start out with the simplifying assumption that the value of the IX is proportional only to the value of public peering. (The LINX has migrated some large traffic flows onto private peering so as a result this analysis will consequently understate their value.)

Third, let’s assume that the alternative to peering is buying transit, so the value of the IX can be estimated to be MarketTransitPrice * VolumeOfTrafficPeered for free at the IX. Under these assumptions, if the IX went away, the community using the IX would have to exchange that traffic with their transit provider at a metered rate. We further assume that the IX peak measure approximates the 95th percentile measure upon which transit is billed.

We say ‘peered away for free’, but public peering is not completely free. The cost of public peering includes some monthly recurring costs for each participant at the IX. If we subtract these costs from the value derived, we can estimate the value of the IX to the population:

ValueIX = TrafficPeeredAtIX * TransitPrice – CostOfPeering*NumberOfMembers

Let’s plug in some public peering traffic stats (https://www.euro-ix.net/resources/list-eur.php) and AS counts (https://www.euro-ix.net/member/m/peeringmatrix) into the model and test out the formula. (Some in the Peering Community pointed out that the measurement methodologies are not consistent across IXes, so consider this simply an exercise illustration of the model.)

Under these assumptions, the model allows us to make the following statements about the DECIX for example:

Can Peering increase ISP Revenue?

Fri, 13 Feb 2009 14:54:39 -0800

Dear ask@DrPeering.net ,

Q: I heard that peering can potentially save money and improve performance, but can peering actually increase revenue?

Sincerely,

Pierre Conè
---------------
Dear Pierre -

Thank you for that question. We focus so much time on the Peering Break Even Point (the point where the cost of peering exactly equals the cost of transit) that we tend to overlook the other benefits of Peering, including incremental revenue flow. The logic came from the field, from Dave Rand of AboveNet fame in particular, and it goes like this:

Internet Service Providers charge transit fees based on the amount of traffic delivered. Therefore, the more traffic delivered, the more revenue the ISP makes.

So what hinders traffic volume?
a) Packet loss. When a TCP traffic flow encounters packet loss, the TCP window size (amount of traffic in mid-air between the two parties) divides by two, and then slowly grows back up. *So every packet lost impacts revenue! *
b) Latency. The latency between endpoints affects how long it takes for the TCP window to open, so especially for short duration flows (like web pages and p2p), high latency has the effect of slowing traffic flow somewhat.

Peering, by definition, provides the *direct* exchange of traffic to each others customers. The resulting shorter path (generally) means lower latency, so the TCP window opens up faster, more traffic is exchanged, more revenue is generated. More importantly, peers have some ability to shut down, or redirect around congested paths.

By the way....DrPeering has searched for years for some way to quantify the incremental revenue made by peering. How might one go about quantifying this?

It is also interesting to note that the content companies that peer say "It is all about end-user experience." The #1 reason they gave for peering was to have greater control over routing, so that when they see packet loss (visible in the host server statistics), their network engineers can do something to address it. When a page loads slowly, the user goes to an alternative sites, decreasing their page views and corresponding ad revenue. This is one reason why Google, Yahoo!, Microsoft, Sony Online, Electronic Arts, etc. are "open" peers.

We'll talk more about "open" peers next time.

The final argument heard is that
a) peering may provide multi-homed customers with a shorter AS-Path for traffic,
b) this traffic will prefer the short path, therefore
c) peering increases revenue from multi-homed customers

Dr Peering
DrPeering@DrPeering.net

Notes from the field... An anonymous source claimed that Amazon has a group of folks looking at the “cart abandonment rate” and has correlated that with packet loss, latency, and other network related performance issues. If linked, Dr. Peering would love to have a look at the data to see if the community can assign a revenue dollar amount attributable to the benefits of peering. Alas, Amazon and others hold that data very close to the chest.