Q:

DrPeering –

What is Remote Peering?

Josey Wales, Missouri Bushwacker

A:

Josey - thanks for the question as it allows me to discuss an emerging trend in the form of a Great Remote Peering Debate.

The Great Remote Peering Debate

As Internet Transit prices bob around $1 per megabit per second, ISPs and content providers continue to ask the question:

“Does peering makes sense anymore?”

To examine this we typically look at the peering versus transit graph shown below. Since Internet Peering has a fixed monthly cost, the unit cost of peered traffic (MonthlyCostOfPeering/PeeredTrafficVolume) decreases proportionate to the amount of traffic peered. Competing against peering is the metered Internet Transit service, which is a tiered service that has been dropping in price between 10 and 40% per year. This tradeoff is summarized in the Peering vs. Transit graph.

The point at which the unit cost of peering is the same as the unit price of transit is called the peering break even point where an network operator is financially indifferent between peering and transit. However, the underlying assumptions of the peering model are changing.

The Traditional Internet Peering Model

Traditionally, peering involves selecting an Internet Exchange Point and extending a network by leasing transport (e.g. circuits) into the IXP, and colocating a router in a colocation facility with a popular peering fabric. These monthly costs are shown in the figure below.

The Traditional Peering Model Challenge

One challenge that has always faced the traditional peering regime is that the cost of peering is relatively static compared to the perpetually falling price for Internet Transit. For example, colocation fees don’t tend to drop at the same rate as the price drops of transit. If colocation fees did drop at the same rate as Internet Transit, then powered racks would be priced today at about $1.20 per month. (The price of powered racks at network dense colocation centers remains between $1000-$2000/rack).

Routers and transport have dropped in price but not consistently, and not at the rate of transit price drops.

Peering port prices are under the control of exchange point operators and have historically dropped at a similar rate to transit price drops. These port price decreases have been largely driven by the European IXPs operating as not-for-profit associations both internal and external pressures to not make too much profit. This European pricing pressure extends overseas where the other IXPs are pressured to reduce their prices in kind. How often have we heard questions like “Why is my European xxxG port costing me $2500/month and my U.S. xxxG port is $4000/month? These peering port drops have lead to a delayed but inevitable drop in U.S. peering port prices, but with a slight time lag behind European IXP price drops.

In summary, while the cost of peering components under the control of the IXPs have dropped, the total monthly peering costs have not kept pace with the price drops of transit. This has been a continuous struggle over time, and the end result is that peering looks less financially attractive unless one can exchange a large number of bits across the peering infrastructure.

As the price of Internet Transit drops, the peering breakeven point shifts to the right, showing that one needs to be able to peer away more traffic for free for peering to be financially prudent.

For large volume traffic exchange, peering easily still makes financial sense, but the rest of the industry always faces these dynamics and the question “Does peering make sense anymore?”.

How Can Peering Compete with Transit?

From a purely financial perspective, peering makes sense as the unit cost of peering drops at a rate faster than the drop in the unit price of transit.

From a performance perspective, peering may provide sufficient benefits (control over routing, lower latency, lower packet loss, greater visibility, etc.) to make it acceptable for peering to cost more than transit.

Some segments of the market seem to be adopting a hybrid approach of blending Internet Transit and “Remote Peering.”

The Remote Peering Model

One way to reduce the monthly cost of peering is an increasingly popular method of peering called “Remote Peering.”

Definition: Remote Peering is peering without a physical presence at the peering point.

A remote peer connects to the peering fabric without colocating a router at the IXP, and therefore, without the expense of equipment or rack space. There is no deployment cost, no “remote hands” costs to deploy additional peering routers or to turn up circuits. Companies like IX Reach, Atrato, and others are providing access to multiple IXPs using a single router interface at a single metered rate (with commits). The remote peering model is shown architecturally below.

Remote Peering Today

Adoption of Remote Peering. According to Job Witteman (CEO, AMS-IX), of the 500+ customers at the AMS-IX, 70 new remote peers have been added in the last year.  Remote Peering is no longer a fringe market.

Peering Control without Peering Expense. Zaid Ali (LinkedIn) is a big proponent of remote peering solutions:

“Our engineering team initially expressed concerns about the reliability of a Layer 2 Virtual Private Network (L2VPN) service, but we have found that remote peering solutions like the IX Reach MPLS network, are very stable. Remote peering allows us to be in control of our own destiny and get closer to our eyeballs; with transit we don’t have control of the routes.”

Implementation of the Remote Peering Model

From the remote peering customer perspective, the router’s network interface has several sub interfaces, each essentially a VLAN extension to the remote peering fabric. This is how LinkedIn gets its content at least one AS hop closer to the access networks that are peering at IXPs across Europe.

IX Reach also streamlined the administrative process of joining multiple IXPs, handling the paperwork and speeding the deployment far faster than if LinkedIn had to deploy routers, circuits, and review, adjust, and execute contracts for colocation and IXP various services with all of these deployment facilities. Turning up peering was a lot faster with the remote peering model.

The Business Case for Remote Peering

Most of the remote peering cost reduction comes from removing the colocation, circuits, and equipment costs from the peering equation. Even if the MPLS metered cost is greater than the cost of transit, the network deployment may still make sense for those that want to peer to control routing to improve the end-user experience.  Consider two scenarios: remote peering from an expensive peering ecosystem and remote peering from an inexpensive peering ecosystem.

Case 1 : Remote Peering from an Expensive Internet Peering Ecosystem

Here are some price points collected from a content provider that compared traditional peering and remote peering from a European city where the transport and transit services were relatively expensive into well-populated exchange points where the price of transit and circuits was inexpensive (Frankfurt, London, Amsterdam, etc.).

The fact that the circuits were so expensive in this scenario makes remote peering seem a much cheaper option than the traditional peering deployment model. The bulk of these traditional peering model costs were removed from the peering equation and replaced with a metered rate to access four IXPs from one interface.

One will observe that in this context, $6000/month divided by 100Mbps is $60/Mbps, a price higher for peering that is much higher than the market price for transit across Europe today. This content provider remains convinced that the hybrid approach (buying cheap commodity transit while remote peering away strategic traffic sets) is a sound performance-enhancing strategy. As with every content provider I have worked with,

“the #1 priority is to improve the end-user experience. “

With their blend of relatively inexpensive transit along with selective remote peering to reach key players, they say that they are very satisfied today with their price/performance ratio.

Case 2 : Remote Peering from an Inexpensive Internet Peering Ecosystem

Price points for remote peering are much lower in more mature Internet Peering Ecosystems such as Frankfurt, Amsterdam and London. In these peering hot spots you can get a 1Gbps remote peering service and reach the major exchange points across Europe for a price closer to $1250/month. At the same time, note that the circuit costs that remote peering would allow you to avoid is much lower (only a few thousand dollars per month as opposed to ten’s of thousand’s per month.)

In both cases though, the logic is the same; remote peering removes the colocation, equipment and circuit cost from the equation and replaces those costs with a metered transport alternative.

The Great Remote Peering Debate

It couldn’t call this a debate without ferreting out some other viewpoints on remote peering, and as usual, it turns out there are strongly held, who are we kidding, almost religiously held, beliefs in the peering community about peering methodologies.  Over a few beers and dinners at recent Internet operations meetings I teased out a few of these contrary views of remote peering. Here are some of the criticisms expressed about remote peering:

1. 1.Remote Peering is outsourcing (and hiding) some important complexity.
   

2. a.From the customer perspective, the MPLS cloud that brings the VLANs to the IXPs is not as easy to monitor and debug as a point-to-point circuit. When things break, there is a lot of active equipment and multiple parties in between the customer and its peers.
   

3. b.In some cases, the added latency and complexity of the remote peering infrastructure model tips the scale to make the peer not want to peer with remote peers.
   

4. 2.Remote Peering can lead to routing inefficiencies.
   

5. We used to say “peering keeps local traffic local,” but remote peering changes this notion. You could be expecting to pick up routes in Amsterdam or Frankfurt only to find out that your peered traffic is traversing oceans over layer 2 where that traffic could be peered at a local IX in your home city! Why should the industry care?
   

6. a.The latency benefits of peering could disappear.
   

7. b.If everyone remote peers, then there could be quite a tangled mesh of routing anomalies that spans layer 2 and layer 3.
   

8. The counter argument to these seems to be that there has always been a diversity of local and international peers at the popular IXPs and that it is up to engineers to mange their own routing efficiently.
   

9. 3.BGP Timers will keep peering sessions alive long after a L2VPN has disappeared, and when it goes down it takes ALL REMOTE PEERING WITH IT!  Individual circuits don’t often all go down at the same time, and one can engineer across different circuit providers to mitigate individual problem. With traditional peering, the router is informed immediately when the underlying transport disappears. With this more complicated layer 2 VLAN system, all peering traffic could be black holed for minutes before the BGP timers shut down the BGP session.
   

10. The counter argument is that engineers can adjust the BGP Keep Alive thresholds before a BGP session is turned down. In the event of a L2VPN failure, peers can turn down the session within 30-60 seconds if they so configure.
    

11. Another person argues that traditional peering across an Ethernet switch also suffers from the same systematic failure – customer routers are not informed when their peers’ router gets disconnected from the fabric. The customer finds out when the BGP timers expire as well.
    

12. It is admittedly a failure scenario that all peering sessions might be affected when the L2VPN solution fails.
    

13. 4.Since Remote Peers are not committed to a real physical presence, they signal that they are not committed to peering in the region, and therefore, peering with them is less likely to lead to long-term peering benefits.
    

14. A counter argument, a pivot actually, is that peering is a local optimization and that if the remote peer does pull out, then that just means the optimization goes away. One is no worse off than before the peering.
    

15. 5.History has taught us that not everyone does the careful analysis of the routing impact before turning up a peering session. Some peers for example will peer with route servers to indiscriminately pick up routes. Remote peers that peer with Route Servers will inevitably lead to the suboptimal routing discussed above.
    

The counter argument seems to be that poor routing can be obtained equally at layer 2 and layer 3. It is an network engineering decision as to how to implement its interconnect strategy.

Summary

So there you have it - whether you embrace remote peering or not, you need to understand that it exists and seems to be getting more popular. I expect that there will be much religious debate about this trend. If there is any light generated (rather than the traditional heat) from these discussions, then I will extend this article with additional data. Until then, I hope that this helps.

DrPeering@DrPeering.net