IXes are all the same - a layer 2 switch connecting routers, right?

This article spotlights a few of the key differences between U.S. and European IXes.

vs.

IX

IX

European IX Colocation Neutrality

IX customers can choose colocation facility that meets their facilities needs.

European IXes spread across multiple colocation facilities interconnected with fiber

In the “classic LINX model”, the Colocation Provider may subsidize or pay for elements of having the IX within their facilities (e.g. space, power, fiber, equipment costs, etc.)

European IX Operated by formal Association, typically founded by a set of ISPs.

European IX Operator is typically a not-for-profit organization.

Prices approximate cost.

Everyone pays the same published fees.

Q: Why does colo operator pay for IX to be in building?

A: Colocation space more valuable with IX access there.

See “Value of an Internet Exchange” article for discussion of IX value.

U.S. Colocation Operator operates both colo and peering fabric

(one contract)

Colocation space is more valuable with well populated  IX access there (e.g. EQIX Ashburn, PAIX Palo Alto).

U.S. IX may be spread across multiple colocation facilities interconnected with fiber, but typically this is limited to their own colo facilities within a single metro area.

Colocation Provider/IX Operator pays for IX switch(es), fiber between and within their own facilities, then resells fiber capacity to customers in their buildings for private peering.

Physical cross connects relatively inexpensive (maybe $1000 non-recurring install fee) within building.

In some cases, ISPs can run their own wires depending on Colocation Operator rules

Physical cross connects comparably expensive ($250/mo) within U.S. Colocation  centers.

Only colocation operators can run cross connects.

U.S. IX/Colo Operator typically for profit.

Strategic differential pricing: Prices set strategically. At steady state they approximate what the market will pay.

The costs of private peering between two IX buildings (owned by the same IX Operator) are borne by the ISP. This makes the more densely populated colo more valuable and sought after since in-building cross connects are generally less expensive than inter-building circuits.

When multiple U.S. colo operators are cohabitants in a multi-tenant a building, there are sometimes conflicts getting inter-colo interconnections, since both competing colo operators have to agree to their respective customers interconnecting.

IXes across Europe tend to cooperate more with each other, and share info. Euro-IX facilitates this. (Competition creeping in now for largest/best European IX.)

IXes across U.S. primarily compete, cooperate only when customers push for it (e.g., GPF replaced IX mtgs)

U.S. Internet Exchange Point Model

(Separate Contracts with colocation provider and IX Operator)

European Internet Exchange Point Model

The European Peering community would point with pride at the European IXes saying that they are both carrier AND colocation neutral. This means that the IXes in Europe are generally available at multiple colocation facilities operated by different companies, none of which is the IX operator. ISPs can then decide which colocation operator provides the facilities services needed, and can connect to the IX whichever site they choose.

European IXes don’t usually have customers, they have members. They are often formed as a not-for-profit associations (44%) founded by members that are the ISPs that first connect to the fabric and seed its operation. The other common form of European IX organization is that of a service offered by an academic or regional development organization (42%). In either case, the focus is on providing mutual value, sharing the costs, and having some degree of community oversight over what they see as their organization. Even when they are not formal associations (e.g. NetNod), they tend to operate as if they were.

Many European IXes also have activities outside the mere operation of the IX, including political, legal, and community outreach and research activities. Results are generally shared with the broader community.

Pricing in Europe for IX services tends to approximate a cost-based pricing model. This is reflective of the not-for-profit nature of the operation; if the not-for-profit makes too much excess profit its not-for-profit status is in jeopardy. The pricing decline then makes it easier for other to justify joining, increasing the economies of scale and allowing the operator to decrease the prices further.

European IXes are distinct from the U.S. counterparts also in that they cooperate. There is a long history of European IXes sharing operations notes, growth trends, additional service offerings, pricing information, traffic statistics and customer status updates at public Internet operations fora like RIPE, APRICOT, and NANOG.

Now let’s compare the U.S. IXes across the same dimensions.

U.S Internet Exchanges. Internet Exchange Points in the U.S. are largely for-profit commercial operations run by the colocation companies that house them. This has several implications.

First, for-profit commercial IX companies generally have the ability to do strategic differential pricing. This means a U.S. IX can offer free colo and free switch ports for IXPs that will help lure in other ISPs into the building.  For example, when Equinix Ashburn was launched, and there were no tenants, WorldCom was incented to come in with stock warrants in the IX. In Los Angeles the Any2 IX was free for tenants in 1 Wilshire, and in Miami the NOTA IX offered free ports to all tenants. The ability to incent the right ISPs into the IX is a tool that the U.S. IXes have that the European IXes generally do not have.

Second, since they are for-profit, the pricing in the U.S. for IX services tends to approach the price the market will bear, as opposed to the European prices which tend to approach the cost of operations.

Third, in the U.S. you don’t tend to see many IXes interconnect with other IXes. The whole question of which IX benefits more comes into play as discussed in the previous Ask Dr Peering article on How Equinix beat MAE-East. Ultimately, these IXes compete aggressively on the colocation and IX fronts and, unless both sides think the deal is unfairly balanced in their favor, the inter-IX deals don’t go forward. Especially within the same building that houses multiple IXes, there is much tension surrounding inter-building cross connects between different IXes customers.

Fourth, the focus within commercial IXes tends to be more towards revenue generating activities. While there is much outreach done here, there is a commercial ultimately revenue generating motivation behind most activities.

There are no members of commercial IXes - peers are customers. Customers pay, they don’t vote. Decisions are made by the commercial IX operator, not by the population.

Next, a soft point. You tend to see that U.S. IXes tend to compete with one another more than cooperate. The exception to this is when the customer base insists that the different IXes work together. This occurred for example when each U.S. IX started running their own peering forums. The customer base, faced with constant travel to so many peering events, insisted that the IXes pool resources to run a single group event which became the Global Peering Forum.  It has turned into a singular must-attend event for U.S. peering coordinators that were otherwise splintered across many different events, so everyone won with this evolution.

Finally, U.S. IX public peering traffic volume is much lower than Europe - tens of Gbps vs hundreds of Gbps on each peering platform. Also, in the U.S. there is much more private peering than public peering. As DrPeering has documented in the past, there are valid arguments on both sides of the Public v Private Peering debate. The U.S. IXes are essentially venture capital-backed purpose-built secure 100,000 sq. ft. meet-me rooms facilitating thousands of private peering connections.

These are the differences that folks point to the most often.  I’ve held back some of the more controversial points surrounding contract negotiations and terms, quality of colo and SLAs.

-- DrPeering

Thanks to Stephen Wilcox, Colin Corbett and John Souter for helping edit these articles, and to ianai and ras for their observations as well.

Massive amounts of public peering traffic (the largest have several 100s of Gbps of publicly peered traffic)

Traffic stats are public at Euro-IX

Small amounts of public peering traffic comparably (10s of Gbps publicly peered at the larger IXes)

Much more private peering.

Traffic stats typically private

Seriously, there are a few things that ISPs have found surprising as they extend their networks from Europe into the U.S.

These differences between U.S. and European Internet Exchange (IX) points can be traced back to an evolutionary split that took place around 15 years ago. The European model for IXes is largely based on the successful model initiated by the London Internet Exchange (LINX), while the modern U.S. IX model is largely based on the carrier-neutral IX model initiated by the Palo Alto Internet Exchange (PAIX).

We will sketch the most notable distinctions graphically in this article.

In Europe, as shown in the figure below, Internet Exchange points have a handful of distinguishing features when compared with their U.S. cousins.

v1.0