Q:

DrPeering -

Are there Tier 1 ISPs that peer openly?

Enrico Pollini

A:

In The Internet Peering Playbook I pointed out that Tier 1 ISPs don’t peer with anyone else, because they don’t have to. By definition, Tier 1 ISPs have access to all routes within the country via their free peering relationships. They simply don’t need any more peering. So why should they peer with anyone else?

1996-2012 : Telecom Italia Peers Openly - Really!

I also highlighted an exception to this rule that I stumbled upon when I ran a workshop in Rome - that Telecom Italia (AS3269), a Tier 1 ISP in Italy with 51% of the market, peered openly! No kidding.

Over a lunch I asked the TI guys “Why?”

I pointed out that in the rest of the world, at least the Internet Peering Ecosystems that I researched, Tier 1 ISPs do not need nor readily accept additional peers. I thought that Telecom Italia was simply playing a trick - perhaps stating that “we will peer with you openly provided you buy $1M of services from us.” Or perhaps “We peer openly when you meet us in 25 locations across Italy with 100Gbps of capacity in each.”  We have seen these types of deceptive statements across the globe. But no, they said that they peered openly and actually preferred peering in only one or two locations within Italy! Very unusual, which is why I documented it in the book.

The answer to “Why?” was interesting. They said that they sold transport to all of these competing ISPs, making margin on their cash cow of copper and fiber. They also pointed to the fact that they were formerly government owned and it might ruffle feathers to not peer openly after the government divestiture. Finally, and this is a common one in other peering ecosystems as well, that the threat of regulatory intervention was strong enough to not want to provide any reason for the regulators to become interested in their peering practices. And so, Telecom Italia stood out in the world as a forward-thinking Tier 1 ISP that peered openly within their home market for compelling business reasons

2012 : The Telecom Italia Depeers Everyone

At the European Peering Forum in Reykjavic September of this year, a couple of the exchange point operators shared the story of how Telecom Italia changed their ways. As peering stories go, it is pretty interesting.

In June 2012, Telecom Italia sent disconnect notices to their peers. They offered to sell their routes via “Paid Peering.” The price of paid peering though was higher (3-5 Euros per Mbps) than the price of transit in the open market (2-3 Euros per Mbps). This did not sit well with the other participants in the Internet Peering Ecosystem, inferring a greed motivation was replacing an “improve the Italian Internet” motivation. In July 2012, the de-peerings began and ten’s of Gbps disappears from the Italian IXPs.

As a result, there was great uncertainty in peering ecosystem. Did peering still make sense? Was their enough traffic to justify peering? The value derived by peering there decreased, and the growth of new attachments slowed. The IXPs started promotions to grow the market. There was customer churn, and great concern about the future of the Italian Internet Peering Ecosystem.

2012-2013 Response

According to Mauro Magrassi (Milan Internet Exchange) who shared the story at the Reykjavik event, the event turned out to strengthen the collaboration among the Italian IXPs. They collectively raised awareness of the issue, attracted new CDNs, colocation centers, and content into the market. They “made a lot of noise” in the court of public opinion and with the regulatory bodies about the de-peering issue, with the hope of pressuring Telecom Italia in some way to reconsider their ways.

Recommendations

I was invited to give a talk at the Milan Internet Exchange meeting last week, sharing some of my latest peering research and perhaps commenting on the Italian situation. So I’d like to share some of my thoughts, drawing from what I have observed in other Internet Peering Ecosystems.

Peering Rules and Regulations won’t work. The ISPs are a clever species and will engineer a way to meet the letter of any regulatory edict without meeting the spirit of the peering regulation. Regulations are tough to get right, often introduce uncertainty (something that stifles investment and innovation), and have long-term consequences. This presents a combination that spells trouble for any well-intentioned set of peering rules.

Bucking requirements is in the ISP DNA. For example, when ISPs join the Hong Kong Internet Exchange (HKIX) they are required to sign the Mandatory Multi-Lateral Peering Agreement (MLPA) which requires them to peer with everyone as a pre-condition for peering there. Everyone signs the agreement, and peers with everyone, but the work around is to filter route announcements so they only announce access to their Hong Kong routes. Most international ISPs have very few Hong Kong routes and are unwilling to “give away” access to their overseas customer base. As a result, they meet the letter of the rule by peering with anyone but miss the spirit of the rule by not peering all of their routes.

Another example back in the day was when Sprint peered at MAE-East. By peering at these “NAPs” Sprint was able to service NSF-sponsored regional networks, indirectly receiving government grant money. But they only brought a T1 transport circuit, which was congested 24/7! When peers requested that Sprint upgrade the circuit because of horrible packet loss, Sprint forwarded the call to their sales team. After all, peers that wanted high speed access to Sprint customers were potential Sprint customers! The point is not to slam Sprint here, as this trick has been played many times by many players. The point is, that these maneuvers will circumvent any attempt regulate peering.

Name and Shame won’t work. We have seen this Name-and-Shame tactic applied in many Internet Peering Ecosystems. ISPs cry foul when they are de-peered, with the expectation that the rest of the world will see their perspective and rally behind them as the “bad guy” de-peers. However, Telecom Italia is now no different from any other Tier 1 ISP in the world, except that they peered openly for a while before de-peering and acting like the rest of the Tier 1 ISPs. Companies do what they believe is in their best interests, and they often don’t care what the rest of the people on the Internet think about it. If it makes strategic sense, they will circle the wagons and defend their position, and defend their right to take that position. Their position will harden in response. This is predictable.

Growing the Peering Ecosystem Is the Best Path Forward. The path forward is to make the peering points so valuable that Telecom Italia is drawn back to it. This is a tough road but is more likely to provide a sustainable peering solution than peering through fiat. Even if Telecom Italia does not come back to peer openly, at least the peering community won’t play the role of victims.

The new markets for peering include the enterprise and the small to medium-sized content providers. These folks need peering education to see that peering is indeed applicable to them, and they need remote peering service to ease the transition and provide a turn-key approach. These new peers need to see themselves as being able to leverage peering for strategic reasons, and do so without having to become network operators or change their business model. I recommend the “Peering Improves Security” pitch also, as security is a strong motivator for enterprises. Large credit card firms spend hundreds of millions of dollars per year on security, so the fact that security benefits can be derived automatically by peering should win a few accounts.  I recommend a few dozen peering workshops across the country to grow this emerging base of new peers.

Maximize the Peering Mesh. The IXPs in the market need to be laser-focused on maximizing the value that every peering participant derives from the IXP. This means, the IXPs needs someone to track peering, to understand every peer’s network inclination, and perform introductions and host socials to maximize the “meshiness” of the collective peering population. The more interconnections, the stronger the glue that holds peers to this IXP. It won’t matter so much that Telecom Italia pulls out if every other ISP in Italy peers to the extent possible at the IXP. From the description above, it sounds like almost 50% of the traffic could possible be peered away. This might be enough of a draw to get Telecom Italia to at least peer some traffic away at the IXP.

More Parties. It might sound flip but building a peering population involves parties, education, facilitation, and tracking the meshes. Parties are a great way to keep tabs on things.

These are some of the broad strokes, anyway.

In general, from my conversations in Milan last week, all of the right things are happening at the Italian IXPs, and they need to push through and roll with the punches.