Peering in Africa

Ask Dr. Peering

DrPeering –

We are exploring network and data center build plans in emerging markets. Where do you see peering and interconnection growth opportunities?

Don “Key” Kong

Nintendo Barrel Company

Don -

From a 30,000 foot perspective, peering is pretty well baked in the U.S., Western Europe, and much of Asia. There will of course be churn:

- new players with tons of new content (probably video) begin to peer,
- established players migrate out and purchase cheap transit instead of peering,
- established players that continue to peer even if it doesn’t make financial sense because the infrastructure is in place,
- seasoned network players migrate their content into commodity data centers but keep a small presence at the network dense location exchange points,
-etc.

Changes aside, for the most part, the “Peering cake” is pretty well baked. Those who should peer, do. In these parts of the world, it might actually be riskier to build a competing exchange point and compete against the incumbent than to build into an emerging markets and build a brand new exchange point or Internet service there.

As for candidate emerging markets, I am bullish on Africa.

The AFRICAN PEERING AND INTERCONNECT FORUM

I attended the third African Peering and Interconnect Forum (AfPIF-3) in Johannesburg, South Africa last week and I am impressed with this forum. I have attended all of them, from AfPIF-1 in Nairobi, Kenya to AfPIF-2 in Accra, Ghana, they get better attended, better talks, and better open discussions at each one. The peering forum was very well attended this time, with over 230 participants from all over Africa and the rest of the world. I found the intellects of the participants to be very high, the open discussions to be insightful, and there was a level of broad professionalism and respect for one another in this community that was inspiring.

One thing that was different from other peering forums was that there there were many suits and ties in the room. There was also a mix of jeans and slacks, and more casual attire, but apart from Asia, most peering forums tend to be casual. The language was English, and the audience actively engaged in discussions on the peering topics at hand, volunteering anecdotes, and sharing their experiences both good and bad. The people were very friendly, well educated, and the resulting hallway discussions lead me to believe that there are some really smart people here.

They just happen to live in early stage Internet peering ecosystems that suffer from expensive transport services, monopoly domination in the last mile, and therefore relatively expensive Internet transit costs. It is not surprising therefore that Internet growth has been sluggish across Africa. But that is changing. Rapidly.

INTERNET TRANSIT PRICE DECLINES

The price of Internet transit in 2006 in South Africa was quoted as being about $1200 per megabit per second (mostly satellite systems). In 2010 the price of Internet transit was cited as being about $400 per megabit per second. Today, in South Africa Internet Transit might cost closer to $200 per megabit per second. This South African transit price decline looks strikingly similar to the price declines I saw when I first started tracking price points in 1998 as shown below.

PREDICTIONS

If I was to make an uneducated guess, sadly, the only type I am qualified to make, I would say that this Internet Transit curve will be followed selectively across certain Internet Peering Ecosystems across Africa.

If the Internet transit price curve for South Africa follows the US Internet transit price curve, as it appears to be doing, Internet transit will approach $120 per megabit per 2nd in 2013. Then, in 2017, the price of Internet transit in South Africa will approach $12 per megabit per second.

To continue this hypothesis for other parts of the continent, I expect that we will see:

-a set of progressive countries (Kenya, South Africa, maybe Nigeria, etc.) will continue to push for deregulation and emerge as continental hubs for African Internet traffic. They will emerge as the AMS-IX, DE-CIX, LINX for the continent and the rest of the countries will grow slowly with grudging dependence on the dominant exchange points in the other country. Those countries that do not deregulate and engage in growing the Internet ecosystem will have high prices, have a hard time competing for information workers, and be left in the Internet dark ages.
-transoceanic capacity around Africa up to Europe will be around $10 per Mbps in these countries for large traffic purchases. International traffic price will not be the limiting factor to the African Internet - as today, it will be the in country problems that keep the prices high.
-locally peered traffic will take off and regionally peered traffic may emerge
-there will be 50 Internet Exchange Points across Africa at or near critical mass by 2017
-wireless Internet dominating the landscape

TECHNOLOGY RISK TAKERS

The other characteristic that I saw was the startup hustle. It is easy to build infrastructure when you have deep pockets - it takes skill, expertise, and vision to build cost effectively. I saw a fast and scrappy group of technology people here.

For example, in Kenya, the banking system has a lot of friction so they overcame this problem using technology in smart phones.

I saw people on the street selling sacks of potatoes and being paid via their cell phones. The money is exchanged between cell phone accounts and people pay and get paid for everything from groceries to rent with their cell phones. In some ways, these folks are less technology phobic or perhaps more risk taking than in the west, and I believe this will drive innovation here.

In the U.S. we don’t see people popping out SIM cards, but across Africa and Europe, this is pretty much a common thing. In fact, they will swap out SIM cards perhaps several times a day to take advantage of cheaper prices from one wireless provider during the evening and pop back in the other card during the day.

EARLY BIRD GETS THE WORM

As with most trends, the early adopters get to establish a presence, understand the market by directly interacting with it, and build momentum. By the time these markets look interesting to the rest of the world, the new entrants will have a more difficult time building in and gaining market share. We saw this when Level 3 dug fiber across Europe and found that the prices offered by the competitors dropped steeply on the very day they turned on the fiber, to a price point where Level 3’s investment would not pay off. Such aggressive maneuvers are common in the Telecom sector and I expect to hear these stories across Africa as well for those late to the market.

Of course, the beneficiaries of this predicted future Africa Internet boom will be the established infrastructure players. Companies with transoceanic fiber around the East coast of Africa (SEACOM) and with fiber assets around the West coast of Africa (Main One Cable), along with Liquid Telecom with terrestrially fiber from the bottom of Africa... these guys are all very well positioned to capture this late comer customer base. It is so difficult to build this stuff and they have already done the heavy lifting, so it will be a no-brainer to buy capacity from them.

COUNTER-INTUITIVE - Regulators are Facilitators in Africa

Here was a twist - something that I have never seen before - these guys like their Telecom Regulators! I’m serious. On many occasions several of the people in the room spoke gushingly about the regulators and the facilitation role that they played in developing their business! This wasn’t just helping them navigate the frameworks in the place - they helped evolve the and streamline the regulatory system to meet their business needs. The Kenyan regulator for example allowed the Kenyan Internet Exchange Point (KIXP) to bypass the requirement for licensing, as they require for all Internet operators. Following a complaint by an ISP, the regulator helped create a license for an IXP. I have also never heard of an IXP license before this. Different.

They spoke about the facilitation role the regulators played, like these guys were economic development people as opposed to the Telecom Police - the view that many in our industry hold for the Telecom Regulators. In Africa they truly believe that the regulators will help them build their Internet Peering Ecosystems in their country as partners. I expect that we will see active regulators participating in discussions at future AfPIF meetings. If the regulators are indeed partners in evolving these ecosystems then this might be a model of facilitation and economic development that we might start to see elsewhere. If you are skeptical, I am with you to some extent, but again, it is different from other parts of the world and I find this fascinating.

CHALLENGES IN AFRICA

Of course there are significant challenges across the African countries. It will take time to overcome them. Here are a few notable ones:

Power. There are the challenges associated with power reliability, but this makes for a healthy market need for data centers. I toured a world class data center in Johannesburg, the Teraco facility. It is modeled after the U.S. Internet Exchange Point model, where the colocaiton operator also operates the peering fabric, and does so in a carrier and ISP-neutral fashion. The facility is world class, every bit as solid as the 80 data centers I have toured around the world.

It was interesting to see in Nigeria that everyone owns and runs generators - many 24/7 ! They configure the generators as their primary and the utility feed is used as a backup, should the generator failed. Exactly opposite from the rest of the world. I asked why and they said that the infrastructure isn’t solid yet, and there were claims that vested interests in keeping the grid unstable. There is apparently quite an industry supplying generators, fuel, parts and repair services for these backup systems and they, as expected, defend their business interests. When I visited the hotel had 2-3 blackouts per day and it finally dawned on me why there are no alarm clocks in the hotel - they would always be flashing 12:00.

To reiterate the point, the power issues are real problems, err, opportunities.

Local Loops. The price to get to the exchange points remains a healthy chunk of the cost of peering in many countries in Africa. The government often owns the telecommunications incumbent so there is a disincentive to deregulate, and this may be required for peering to make sense here.

No Local Content. There is little content, but this will also grow when the eyeballs have high speed access and desires the local content. Estimates vary but it is said that 90% of all African content comes from other continents today.

Big Geography, Little Economy. The continent is big. One graphic showed the U.S. China and a few other countries could collective fit inside the continent of Africa. Add to that that the economies here in some countries is so bad that the Internet is the least of their peoples’ worries. There you have the essence of a challenge.

Infrastructure Builds are Slow. I again heard stories of peasants digging up fiber, expecting copper that they could cull to make jewelry to sell on the street. Stories of sabotage come up, etc. Street gangs demanding money from construction crews so they wouldn’t get hassled. (In this case the solution was to hire the local police friends and relatives to do the digging was part of the solution.)

I heard of a fiber cut caused by an elephant! That was a new one. The fiber is typically buried a meter down but occasionally they will hit rocks and sometimes lazy workers will let the fiber lay on top of the rocks less than a meter down. So, at 4AM, an elephant in a national park caused a fiber cut (they could tell from the foot prints) and they were not allowed to enter the reserve because of the dangers of interacting with the lions during their evening hunting period. Again, interesting because it is different. I have images of the rainy evening scenes from Jurassic Park for this one.

The point is, there are many obstacles to Internet development in this part of the world. Slowly but surely, these challenges are being addressed. For example, the African Union and ISOC are facilitating technology transfer to the goal of building the local Internet Exchange Point expertise across 30 countries in Africa! Many forces are pulling in the right directions here, so I don’t believe it is a question of if the Internet will grow here, it is a question of where and how fast the Internet will grow here.

Interesting Peering Discussions

Technology Broccoli. I had not heard this expression before. It refers to technologies that are perceived to be distasteful but ones that you know you need to explore and adopt. For some reason IPv6 and DNSSec among the list of common examples.

Are IXPs Innovative? In the book “Disciplines of Market Leaders” the author claims that leaders in any market are leaders in exactly 2 of the following 3 dimensions. Let’s talk about these disciplines as they apply to the leading IXPs in the world.

1)Operational Excellence - IXPs around the world seem to know how to keep electrons flowing to the switch and connecting new members. They handle administration things well and fail very rarely.
2)Customer Intimacy - The leading IXPs generally know who their customers are and to some degree what value they derive from the IXP service. They might not know the routes announced or the significance of having these specific customers as customers as it related to the rest of the population, but they throw parties and mixers, facilitate peering by introducing people around to the degree practicable.
3)Innovation - I argue that IXPs haven’t developed a breakthrough innovation since a neutral party turn on a switch to serve competing ISPs a few decades ago. Incremental innovations such as moving bits faster, sure, but no breakthrough game changers. This is the discipline that is not exercised.

And then the conversation turns to the thought that maybe that is OK. Maybe the appropriate role for an IXP is to move bits across fast and fat fabrics and not provide bleeding edge innovation.

It seems too that today, the Customer Intimacy leadership is a tenuous hold for IXPs. The leading Internet exchanges have grown into a 500+ customers operation and it is very difficult to facilitate the 500*499/2 potential interactions that optimal peering facilitation might require. There are just too many customers to be close to, and the remote peering trend allows customers to never even meet the IXP operator.

Anyway, this was one of the more interesting conversations / debate that I participated in.

AFPIF IS A HEARTBEAT FOR THE AFRICAN INTERNET

Discussion forums like AfPIF are invaluable for evolving the African Internet Ecosystems, and a really good source for on-the-ground research like what I do. A gentleman from South Africa named Graham Beneke (graham@neology.co.za) for example gave a great talk (video link here) that highlighted some routing and operational challenges in South Africa along with some specific solutions. His overarching theme was that some of the problems have to do with proximity of the countries on the continent to the rest of the world, but many of the problems are self inflicted, such as asymmetric routing sending traffic between African countries through Europe, announcing more specific routes unnecessarily, and using AS-PATH as the primary route discriminator when latency could be taken into account manually. Martin Levy’s talks as usually were great as well. The SEACOM preso was full of data points... Check out the videos for yourself - the video capture is quite good:

http://www.livestream.com/internetsocietychapters/folder?dirId=db5b9aae-d566-4d4a-801f-5209cfb6b830

I call AFPIF the heartbeat for Africa in the section because we all have day jobs, and we get together periodically at these forums, away from the daily grind, to learn from others. We establish friendships, maintain relationships, perhaps mend others. We identify potential future business, the ties that keep people and companies working together towards a common end. Then we go back and perhaps do our jobs a little differently, or perhaps we adjust the course of our company based on the corrective feedback that we get. From a 30,000 foot view, this AfPIF meeting resembles a heartbeat, syncopating with the flow of information and energies through the growing African Internet Peering Ecosystems.

SUMMARY: INVEST IN EMERGING INTERNET PEERING ECOSYSTEMS

For those looking for reasonable returns on investment, invest in infrastructure in developed countries. For those looking for home runs on their investments, Africa may be your best bet.

Anyway, I have a ton more notes from the discussions, but these were the ones that I highlighted in my notebook.