The Art of Peering: The IX Playbook  Tactics 1-7

Thursday, May 7, 2009

Network Externality

The Internet Exchange Business exhibits the characteristics of what economists call the “Network Externality Effect”; that is, the value of the product or service is proportional to the number of users of that product or service.

The usual example of the network externality effect is the fax machine. The first fax machine deployed was useless - no one could send or receive anything because there was only one.  When the second fax machine showed up, all of a sudden there was some value to owning a fax machine; there was one other party who could send or receive faxes. When the third fax machine was connected to a phone line, there were two other potential faxes to communicate with, and it was therefore more valuable to own a fax machine than when there was only one or two. And so on ; the value of owning a fax machine is proportional to how many others own a fax machine.

Internet Exchange Points have a similar network externality effect - the value of an IX is proportional to the number and characteristics of ISPs participating at the IX.  We quantified this financial value of an IX more thoroughly in the Ask DrPeering article “Value of an Internet Exchange”.

Network Externality and the Internet Exchange Point

Let’s explore some of the challenges facing a new IX. First, let’s make the simplifying assumption that the value of an IX is proportional to the number of ISPs there. Let’s further assume that all ISPs peer with each other, and that all ISPs peer the same amount of traffic with each other.

Now let’s plot this value of an IX to an ISP against the number of ISPs at the IX. 

Here we see the stages that all Internet Exchange points go through.  The cost of peering at the IX is highlighted here along with the value of peering at the IX. Where these two lines meet is what I call the Critical Mass Point -- the point where the cost of peering exactly equals the value derived from peering.  After this point, the peering population tends to grow exponentially, because the value of peering there is so much greater than the cost of participation at the IX.

Before the Critical Mass Point however, peering at the IX is a tough sell.  There isn’t enough traffic to peer for free to offset the cost of peering there. So what do the smartest IX operators in the world done to get over this “IX StartUp Hump?”

Beyond the typical “concept sell” that everyone in the ecosystem benefits by a well run IX, and beyond the “future sell” that the population will grow, there are a few effective tactics for building critical mass.

Tactic 1: Startup: Drop Peering Cost

“On contentious ground, I would hurry up my rear.” -- Sun Tzu

When there is not enough value in the IX to attract NSPs, an IX operator must do what is necessary to find a reason for NSPs to join. One approach is to reduce the cost of peering for early adopters to entice them into participating. This tactic is shown graphically below. With this approach it will take fewer peering session, less traffic, and in general less justification for NSPs to build into the IX.

Public Peering Port Fee Schedule

------------------------------------------------

Participants 1-10: $100/month

Participants 11-14: $200/month

Participants 15-20: $300/month

Participants 21- : $500/month

One form of this is shown as a tiered pricing schedule, based on the number of participants at the IX. This recognizes that the value of the IX grows and therefore the willingness to pay should grow as well.

This tiered approach of course has the down sides that

1. a)The value of the IX is still $0 for the first participant, and there is a cost of committing to an IX and building in, so the problem isn’t solved, and
   

2. b)there is an incentive to commit early but this can also be seen as discouragement  if you happen to be the 11th, 15th or the 21st participant.
   

If I was an ISP I would certainly try to negotiate hard to get the first tier pricing regardless of when I signed up.

The other forms of this seen in the field include to

1. a)give it away for free as Terramark did when they launched NAP of the Americas, or
   

2. b)make it clear with prospects that the prices would be negotiated down from the list price on a one-by-one basis (which tends to make the customer try and figure out the value they derive from participation), or
   

3. c)buy them in as Equinix did by offering stock warrants to MCI/Worldcom to build a major POP instead of a tributary. (This occurred in 1999 and was disclosed in public SEC and debt documents.) As shown in the figure, this helps leap frog the IX value; in recognition of the value they provide to the building, they participate in the up side of the business. It is debatable what value MCI/WorldCom brought to the table since they don’t peer with anyone except the other Tier 1 ISPs, so don’t lure many others into the building.
   

Each of these approaches decreases the cost of peering to lower the critical mass point.  As more participants join the IX, and the value goes up, the IX operator has the leverage to raise the prices back up.

Tactic 2 - Bluff

“All war is based on deceit” -- Sun Tzu

There is a range of deceit in the IX business.  One has to present a working ecosystem when only a concept exists.

Bluff Carriers. Frustrated IXes dealing with the Startup Hump often employ tactics to signal greater value to peering prospects. Paul Vixie at an APRICOT conference in the late 1990’s described the StartUp phase of the PAIX; they needed to persuade the large carriers to build in to the PAIX facility in downtown Palo Alto. Pacific Bell was contacted and told that WorldCom was generating significant revenue from the PAIX population and that the customer base would like a choice in carriers. Not wanting to give up business to WorldCom, Pacific Bell built into the PAIX. WorldCom of course was not in the building at the time, but when Pacific Bell showed up and started selling capacity, WorldCom was contacted to come in and compete. Once these two carriers were in the building, the population of carriers and NSPs was able to grow.

Bluff two potential peers. These same tactics have been used by some IXes to attract customers in; identify potential peers that the prospect would like to peer with and signal they are there or “pretty close” to installing. The prospects desired potential peers are then also targeted with the same type of signaling in the hope that neither NSP will realize that the other wasn’t already installed at the IX. This tactic often fails, as the peering population is a tight knit group of professionals that talk to each other for a living. These IXes lose credibility and it becomes more difficult to build sufficient trust to build momentum out of the startup hump.

Misleading Customer Lists. Another form of this broadly seen across the IX industry is the misrepresentation of the Peering Participation on the website. The NAP of the Americas (NOTA) in the early days for example had a “Representative Members” list to imply the potential peering population. The NOTA was developed by a consortium of NSPs and carriers, and this consortium list was used in the early days to signal support for the NOTA but implied participation. As a result, some NSPs were led to believe there were more participants in the NOTA than there were.

Misleading IX Traffic Graphs. Since the value of the IX is proportional to the amount of traffic peered there, some IXes use different measurement techniques to demonstrate their value. This has caused some infighting among the otherwise cooperative exchange point operators, since they all know that they are being compared against each other based in part on the which one provides the greatest value (traffic).

These tactics are common in many that demonstrate the network externality effect. And yes, as demonstrated, it is seen in the IX community, particularly those that are struggling through the start up hump. If there is no value to participation then one has to sell a vision, paint a compelling view of the future of the IX. Scraps of interest are sometimes crafted into “marketing speak” in the form of “backed by” or other such words. The problem is the “well-chosen marketing speak” is often repeated without the same precision by the sales people, leading to charges of misrepresentation, etc.

Tiered Pricing Proportional to Value

Buying in the Big Peering Player

Tactic 3: Group Buy-in

“On open ground, do not try to block the enemy's way. On the ground of intersecting highways, join hands with your allies.”

“We cannot enter into alliances until we are acquainted with the designs of our neighbors.” -- Sun Tzu

When @Home went bankrupt it left in its wake a dozen or so cable companies that were left to find Internet service for their customers in thirty days! Very quickly they learned that transit was very expensive, particularly as the file sharing (Kazaa, morpheus, Napster at the time) was consuming 40% of all available transit bandwidth. The author spent time getting acquainted with this community to find a clear and compelling aligned interest – they each could save millions of dollars per year if they interconnected their networks together in peering relationships at an IX.

As a group, the North American Cable companies converged on an IX (Equinix), providing that IX with a huge gravitational pull. The cable companies were “Open Peers” so, generally speaking, were willing to offload as much traffic as possible with NSPs and content companies interested in peering. This group buy-in tactic helped build a strong critical mass and increase the value for the IX populations in which they participated. This is shown graphically below and can be a very effective start up tactic.

Tactic 4: Swim with the fishes

In respect of military method, we have, firstly, Measurement; secondly, Estimation of quantity;    thirdly, Calculation; fourthly, Balancing of chances;    fifthly, Victory.

DrPeering’s uncle used to scuba dive in lakes to find where out the fishes were, how many there were, what they were doing.  Armed with this intel, fishing became a lot more effective.

The IX business is unique in that the value provided is out of the control of the IX operator. Whether an ISP will peer, and if so how much traffic will be exchanged, and if that peering will last for the duration of the IX contract, are examples of variables that are part of the value of the IX calculation but  outside of the control and sometimes even outside of the visibility of the IX Operator. Many ISPs consider peering relationships, traffic volumes and projections, etc. proprietary and confidential. This makes gathering intelligence about the value of the IX service and the desirability of the IX very difficult to obtain. It makes collection and management of the intel that is available even more important.

Valuable intel includes

•contract terms and duration,

•peering locations,

•peering relationships,

•traffic volumes between players,

•financial situation,

•desirable peers,

•transit prices,

•POP locations,

•transport prices current and future,

•future deployment plans,

•peering challenges

Intelligence gathering requires constant reconnaissance missions. This includes travel to and speaking at networking conferences and particularly forums that attract NSPs and peering coordinators. Of particular interest are:

•What are customers and prospects saying about your IX?

•What are competitors saying about your IX?

•What are the trends in the market?

•What is the price and position in the market of your competitors

Use your contacts to identify problems with your service early and serve as the early problem solver.

With this information opportunities are made available and vulnerabilities in the competition are made available. The appropriate meetings, at the appropriate time, when armed with the appropriate information can make or break an IX.

Anyway - I hope this helps give you a flavor of some of the techniques used on the field.

Look for a future article on the defense side - those tactics the smartest IXes in the world have used to defend their position as the dominant IX in the region.

-- DrPeering@DrPeering.net

Tactic 5: Divide and Conquer

If he is secure at all points, be prepared for him. If he is in superior strength, evade him.

Regardless of the size of the peering population at the sub-dominant IX, there are usually unique networks only available at the IX.  These “unique networks” have a vested interest in increasing the peering population at the IX they selected and may be willing to share traffic information necessary to attract and build the IX population.

Specifically, this tactic involves pursuing strategic relationships between the IX and the NSP community to share prioritized lists of peering targets and measured traffic levels should the NSP peer at the IX. Armed with this information, business cases and compelling pricing can be constructed to attract these “key peers” into the IX, perhaps away from or in addition to the dominant IX. This can have a flywheel effect as these companies then can share their information to further accelerate the build up at the IX.

Tactic 6: Attack at Weak Points

When the dominant IX population is strong, a sub dominant IX may need to attack the dominant IX where they are weak, presenting reasons to select their alternative IX that do not involve the population. Seen in the industry, these weak point might include:

1.Dominant IX has no SLAs on services. Many IXes in Europe are run as not-for-profit associates providing best effort peering services and do not budget for SLA reimbursements. This provides some corporations pause.

2.Dominant IX does not allow Private Peering. Some IXes do not allow private peering but instead require all peering occur over a public peering switch. For some large NSPs this presents a significant downside: it opens up possibilities for bandwidth stealing by pointing default and makes it more difficult to manage peering bandwidth for peering sessions for example.

3. Dominant IX can not provide Terms and Condition required. Some NSPs can’t negotiate different terms than the rest of the membership. Membership organizations are driven by the participants who may not want to bend over backwards to get large competitors to attach and benefit. Commercial IXes may be able to price proportionately to the value the NSP provides to the community, thus attractive highly desirable peers with $0 cost ports to build population.

4.Dominant IX do not allow Content Providers. Some IXes do not allow Content Providers to peer at their IX.

5.Dominant IX does not provide sufficient Operations Support. Security issues, access issues, time to add racks and services may also be a differentiator.

6.Dominant IX does not allow transit sales. Some IXes do not allow transit traffic and therefore participation has only cost savings, not revenue generation. An new IX can reach this underserved market.

7.Dominant IX doesn’t serve some peering market. An IX can build a differentiated service along different dimensions including the population served. For example, in Japan there are three large dominant exchange points which service the domestic peering market. To compete against these local Japan-run IXes by attracting local NSPs would be very difficult. However, a focus on underserved International NSPs looking to establish regional interconnections, not necessarily public domestic peering within Japan, may be best served by a different perhaps International IX. 

The Leverage Differentiated Services tactic can best be diagramed below.

Competing against the Dominant IX

The next batch of tactics focus on how a new entrant competes with a dominant IX. This is a challenge since the value proposition of the dominant IX player is typically demonstrably better than the others.  Further, the existing population of peers benefits by the new ISPs coming into the dominant IX, so there is a further lure to the dominant exchange.  What are the tricks of the trade here?

Evade and attack weakness

Once the IX population is built up, this tactic can then be followed by the divide and conquer tactic.

Tactic 7: Lure away weakly peered IX Customers

“If the enemy leaves a door open, you must rush in.”

When IX participants are sparsely peered, they are vulnerable to moving. When the IX participants are unhappy with the quality of the peering service, if they have suffered from outages or decreased value provided by the IX, they are vulnerable to moving. This tactic requires an IX identify those vulnerable participants and their current and desired peers. Next the IX identifies how much peering traffic is being exchanged and calculates the value derived from participation at the existing IX. Those with few peering sessions, those with small amounts of traffic exchanged, are deriving the least value and therefore may be willing to provide details of who they would need to peer with in order to be worthwhile to move.

These existing and desired peers can then be approached and brought through the same process, possible in a group buy in tactic. This tactic rarely however succeeds in mass migrations due to the “Operations Gear Tenet” but often yields incremental adoptions for redundancy and future expansion. NSPs not receiving the value from the IX that they expect can be powerful advocates for other potentially interested in moving and can also provide a great wealth of intel on the competitive landscape.