Q: DrPeering -

In the heart of Gotham City, I attended a conference put on by leading academics discussing the economics of the Internet, focusing on the dynamics of peering. There were a few network operators (ISP, CDN, Content Provider, Search, etc.) and a handful of PhD candidates.

It was like they were talking about different Internets.

Gadzooks but these were completely different views of the world! And clashes. Pow! Bamm! Zop!

Which brings me to the riddle -

Why are there such clashes between these non-competitors?

Adam West

A:

Golly Gee Willigers you are right Adam!

There has long been this underlying inter-group conflict for as long as I have been in the industry. The operators sometimes say

“the IETF standards-making body is openly hostile to the network operators that attend.”

On the flip side, the researchers and hardware manufacturers continually have their motivations questioned when they submit talks at the operator group meetings like NANOG.

With inter-group conflicts you often see a phenomenon called “perceptual distortion.” Each group ascribes motivations and characteristics onto the other group that reinforces their own world view, and they unite with the others in their group behind that “common enemy.”

A way to bridge the gap is to pull the groups together with a strong facilitator and talk openly about these distortions, maybe laugh about these extreme and untrue views, validate some and invalidate others, and find areas of common interests and alignment. Working on specific projects together where there is alignment helps breakdown these perceived boundaries.

Let’s talk about these two groups and the circles they travel in.

The Internet Engineering Task Force (IETF) meets in different locations around the world several times a year. To participate you are advised to participate actively in the working group mailing lists, to read the RFC drafts being discussed/debated along with the supporting groups of RFCs. In some cases participants are asked not to participate in the live discussions at the meetings unless they have kept up with all of this institutional knowledge. While these meetings are “open” in the sense that anyone can pay and attend, the amount of work required to keep up and participate under these informal rules, and the cost of international travel required to attend sequential IETF meetings makes it very difficult for network operators without a significant research budget to attend. As a result, only hardware manufacturers, researchers, and in some cases academics with travel budgets drive the IETF.

When operators participate at IETF, their numbers are small, and their voices are drowned out by the vendors that espouse second-hand (i.e. “a lot of customers have asked for this”) points of view. As a result, the operators view IETF as “not worth attending”, promoting their agenda at IETF as “swimming upstream”, and

“The IETF is openly hostile to network operators.”

The network operators instead tend to participate at network operators group (*NOG) meetings instead, where their numbers are strong (~40% of the attendees are ISPs), the meetings are local (relatively speaking) where they can compare operational issues like what works and what doesn’t. To do this they need a group meeting with a social environment conducive to identifying and comparing notes with their peers from other companies.

The conference submissions from network vendors are severely scrutinized and viewed with great suspicion. “What are they trying to promote here...what is their agenda?” is the common perspective in the operator community here and presentations are often dismissed based on the perception of spin rather than on the chance that someone might learn something from the content. As a result, the vendors have learned that the way to get on the agenda at operator meetings is to have a customer network operator co-submit or submit their experience with the technology.

Researchers are not viewed with suspicion because no one in the operator community really cares about “researchy” topics. So, they put the researchers into the “research track” opposite of the peering track, thus guaranteeing that the researchers don’t get any input from operators that they came to get input from!

Each group has their own private invitation-only conferences as well. Both groups focus on getting the right and manageable number of attendees in the room. Sometimes this is done with a requirement that the attendee be a customer of one of the sponsors, or a government-sponsored lead researcher. These private meetings have the side effect of reinforcing the boundaries between these groups. After all, if these “others” would be allowed to attend they would no doubt ruin the event in some way.

These are manifestations of this perceptual distortion, reflective of the underlying inter-group conflict.

Academics vs. Operators

Let’s look at the divide between the academics and operators. (These views are my own and no doubt reflect my own personal perceptual distortion.)

1. 1)Models vs. Practice.
   

The academics look at the Internet as a massively important infrastructure that could spin off tons of data that could be processed using their favorite advanced mathematical and modeling techniques. They want the data so they can do their work, identify something interesting that has been overlooked, and publish the results. The more counter-intuitive, the more important, the more interesting the results, the more quoted and read they become. Academic rock stars are made this way.

However, the data they seek is not typically been made publicly available, so in the absence of publicly available data they make simplifying assumptions about how the Internet works, and make models based on those assumptions.

The operators on the other hand are not looking for simplified models or interesting anomalies as much as trying to keep their network working, customers happy, and their business profitable or at least afloat. Fundamentally they are focused on how the flawed equipment that they put into their production network actually works when connected to the real Internet.

The operators are surprised and dismayed at the gap between their understanding and the academic understanding. It is enough to dismiss a speaker with great haste and there is not much tact displayed when doing so.

2. 2)Math vs. Money
   

Academics are not rich. They are not making a lot of money from their research in general and do not have the budget to build test beds of the scale of the Internet. Therefore, they count on their R&E networks for data, or more commonly, on simulations that are based on their hypothetical models of how the Internet works.

Without the raw data, they are forced into the world of theory. I have rarely seen an academic presentation based on truly relevant and interesting data culled from the public Internet. They are more typically simulation models, greek letters and graphs to prove importance and correctness.

The ISPs however keep their data proprietary and have since the NSFNET days. The data could (and will) be used by competitors to prove to prospects why they are better than the other’s own pronouncements. Sharing the actual data may materially harm the stakeholders, so the data is not made readily available. Even anonymized data that mentioned winners and losers made one researcher very unpopular with those that contributed anonymous data. So the risk is high and the payoff is low for sharing data, and it is all about the money.

3. 3)Theoretical optimization vs. stepwise refinement
   

Academics appear to be looking to apply math to identify things that are interesting and ideally lead to something akin to a mathematical proof of optimality.

Operators on the other hand appear to be engaged in the stepwise refinement of applied heuristics that make incremental improvements in the current operational context.

Different motivations.

4. 4)Cultural differences.
   

The operators are a bit more “rough and tumble” with their discussion styles than the academics. They may interrupt others more often, listen less well, perhaps speak louder and more often than researchers. This may be the group culture stemming from the years of working in a crowded team environment filled with uncertainty and the need to identify those few things that are solid truths. Those that assert most loudly, and rightly, take increasing responsibility and rise through the engineering organization. They tend to be a bit aggressive.

5. 5)Network Operators don’t need academics, but academics need network operators.
   

A number of academic papers published that describe models and make assertions that are pretty far removed from reality according to some in the network operations community. This reflects the desire for academics to publish without hard data and the cooperation of network operators, competing with the interests of the network operators to protect their information. As a result, the research done by academics is thought to be “irrelevant” and generally not needed by the industry. From this perspective, the academics need the network operators more than the network operators need the academics.

These cultural clashes have gone on for years and the bridge needs to be built. There are a lot of interesting projects that one group can do that would benefit the other if they can get over these differences.

DrPeering@DrPeering.net