Ask Dr. Peering

Back in 1999, MAE-East was the dominant Internet Exchange Point on the east coast. By 2002, most of MAE-East customers had migrated to the Equinix Ashburn Internet Exchange.

This provides an opportunity to demonstrate some of the techniques described in the “Art of Peering: The IX Playbook”.

How did Equinix Beat MAE-East :  Tactics

We will demonstrate some of the IX tactics discussed last time and introduce a few more. First, attack weakness.

A) Equinix focused on providing a quality facility that compared very favorably against MAE-East’s insecure garage (Tactic 6 - Attack weaknesses). 

B) Equinix presented MAE-East as a non-neutral IX, one operated by a competitor to the ISP (Tactic 6 - Attack weaknesses).  Why would you want to support a competitor who will appear better in the eyes of the customers because WorldCom was the “center of the Internet” by running the dominant IX?

C) MAE-East participants were identified and ranked by recruit-ability and desirability. (Tactic 5 - Divide and Conquer and Tactic 7 - Lure away weakly peered IX customers)

500. D)Equinix staff (such as the author) were well entrenched in the ISP community (Tactic 4 - Swim with the fishes) participating and speaking at more than 150 fora around the world. The author heard many stories about the security weaknesses of the MAE-East facilities: for example, that a car could crash into the MAE-East parking garage and take out the entire Eastern Coast Internet.  In addition, that many ISPs were leaving due to the high cost of peering there. These points were evangelized and led to a price proof that ethernet peering was cheaper.
     

But these were not enough to pull customers from MAE-East, because the value of the MAE-East IX was still greater than the cost of being there.

Fundamentally, CRITICAL MASS PROVIDED MAE-EAST WITH STICKINESS.

5. E)But ultimately, all of these tactics helped shift momentum, but were substantially aided by a perfect storm of
   

1. 1)the static and high cost of peering at MAE-East,
   

2. 2) the dropping price of transit,
   

3. 3) the political/religious backlash against ATM, and
   

4. 4)WorldCom largely ignoring MAE-East
   

A few notes about religion and politics of peering.

First, ISP engineers did not care for ATM as a technology, and did not care for the telephone companies; both were seen as the enemies to their way of life. 

So this tactic is to approach the dominant IX with an offer to cooperate.  One form of this is to offer to extend the dominant peering fabric into the challengers colocation center, pay the costs of that extension and offer to share the incremental revenue.  This way the challenger has a means to attract customers in (customers get the value of the opportunity of peering with everyone at MAE-East for example) and the dominant IX gets a split of the incremental revenue without the sales overhead and without using up what limited colocation space it had left. Equinix could still offer and operate a competing peering fabric for its customers and potentially grow that customer base to critical mass.

Let’s look at the power positions of the two parties below.

The Cost and Technology of Peering at MAE-East. At the time, MAE-East was based on ATM, a technology for which the interface cards were expensive. Further, since MAE-East was run by WorldCom, one had to purchase WorldCom ATM circuits to get in to the Internet Exchange.  Finally, the ISP community generally rejected the ATM technology as “packet shredders”, highlighting a layer 2 ATM failure mode. So the engineering folks didn’t like the technology there, but most importantly, as the price of transit declined, the cost of peering at MAE-East was too expensive.

The Cost and Technology of Peering at Equinix. Equinix offered an Ethernet peering service (which the ISPs preferred), at a much lower price for the first bunch of ISPs (Tactic 1 - Lower Peering Costs).

Tactic 9 - Build Peering Community with Peering Forums

Equinix started running “Gigabit Peering Forums” to pull the peers together to help shape the peering community. (MAE-East had very little marketing support from WorldCom.) A whole article could be written about the lessons learned running facilitation peering forums, but for now let’s just say that the focus needs to be on the mutual benefits realized by the success of the IX. As the first few peers connected to the IX for religious reasons (preferring ethernet), and the economics of plummeting transit fees (without decreasing MAE-East circuit or port fees) made MAE-East financially unattractive, and the momentum shifted.  ISPs started to prefer peering at Equinix in Ashburn.

The Gigabit Peering Forums played a key role as it presented the opportunity for Equinix to facilitate peering among the peering population, something that helped the peering population while at the same time helped Equinix gain critical mass.

After five or six successful peering forums, the peering community started rallying behind Equinix as an IX with aligned interests to their own. Equinix received all kinds of market intelligence, relationships formed.  In response, the other carrier-neutral IX called the PAIX started their own peering forum, as did a few others. The community provided feedback that there were now too many IX fora to attend, so they merged the fora into a single Global Peering Forum which is very active today.

Tactic 10 - Be Curious and Freely Share Research

The author helped shift the momentum by documenting what he learned from the peering community about leaving MAE-East in the form of a white paper. He wrote “Do ATM-based Internet Exchange Points make sense anymore” which documented and explored the reasons that the ISPs said they were leaving the ATM-based MAE-East peering fabric. This white paper was widely circulated and presented at many conferences, accelerating the momentum toward ethernet based internet exchange points. With the support of competitors that also advocated ethernet-based Internet Exchanges, the exodus from MAE-East to Equinix was expedited.

The information collected in the field was propagated back to the field so the contributors of data could see their input and name in written form so they could see their name in print. As a third party validator, they indirectly reinforced the authors credibility and reinforced the ethernet-based IX benefits to the community.  Finally, authoring industry resources leads to being seen as a thought leader, which in turn helps build credibility, and reinforces the notion that the IX will grow to critical mass.

Several of these white papers made their way to high level executives at prospective ISPs. The author heard stories of how these white papers were given greater credibility than the companies own experts, simply because they came from the outside, away from the political pressures that colored many position papers.  All f this helped champions of peering within companies push it through the internal processes.

Since the data from these white papers came from the field, it had further credibility in the sense that many perspectives and vantage points were leveraged to create the views in the paper. In this way, the research helped uncover for the community some hidden common issues.  Further, the content had passed through a large set of industry insiders, so had been vetted to some degree.

All of this lends credibility to the IX since it focuses so much attention on the customers.

Tactic 11 - Evolve Success from Success

When the Tier 1 ISPs met to discuss operations difficulties keeping up with the inter-backbone traffic volume, the author had already researched their issue in the form of a white paper “Interconnection Strategies for ISPs” and had proven in the paper that if the Tier 1 ISPs interconnected with four other Tier 1 ISPs in an IX using private cross connects, the cost of doing so would be entirely covered by the cost savings by NOT connecting using point to point circuits.

With the momentum shifted from MAE-East to Equinix Ashburn, it was easy to see Equinix Ashburn as the interconnect point of choice for the east coast of the U.S., but what about the other seven regions? Here the carrier neutrality really helped, as a few of the Tier 1 ISPs would have preferred to have the interconnections happen within their data centers, and most of them had their own regional data centers.  But to POP a competitors building was a strategic non-starter. As a Qwest executive put it “Qwest will never put Qwest fiber into a Level 3 facility” -- these guys are cross town rivals.

This leaves just the carrier neutral colocation alternatives, and the pitches that Equinix and Switch & Data were very different.  Equinix went in with the CEO and several others who knew the decision makers around the table.  The pitch was a high touch and consistent service in all interconnection regions, an open and extensible interconnect environment that provided the opportunity for other Tier 1 ISPs to buy the service of full mesh interconnections with the other Tier 1 ISP cages. The Tier 1’s individually could decide to use the cross connects dropped to their cage, so the interconnect decision was still theirs.

Switch and Data sent a lower level sales or marketing manager that didn’t have the relationships with the folks in the room and proceeded to use a “overly slick sales message” according to a couple of those in the room.

It is the authors opinion that quality sales work, all the research ground work, and the relationship development work allowed Equinix to leverage the success of Equinix Ashburn into Equinix becoming the dominant IX of choice for the Tier 1 ISPs.

“We can both win. Let’s work together.”

We’ll get back to you.... Silence.

We need more information.

OR

Inside Voice: We are out of capacity, so this sounds like a way to get more revenue without much overhead.

Possible Responses:

Inside Voice: We are already winning. You will benefit more than we will. Why should we enable a competitor?

Possible Responses:

We need some technical, legal and business logistics documentation

Equinix in the left has very little ability to attract a peering mass into the IX. MAE-East on the right has critical mass and the default action of ro ISPs in this market is to just join MAE-East. It is past critical mass, so by definition, the cost of participation is less than the value derived by peering, given the current price of transit. So when Equinix asks to cooperate, what will the two parties do in an objective analysis?

We need more time for internal review.

Tactics 12-20 are being documented. Please contact DrPeering@DrPeering.com for more information.

Dr Peering

DrPeering@DrPeering.net