In this brief retrospective, we talk about some of the undocumented behind the scenes history of NANOG. Specifically, we talk a little about what led up to its formation, and a bit about the choices we had to make operating and evolving the forum.
This may be helpful background for two audiences:
<Narration for the video.>
Hi – My name is Bill Norton and at the time I wrote these slides back in 2007, my day job was at Equinix where I spend 90% of my time working with the customer base, identifying areas of strategic alignment, and trying and find ways to pull these different communities together. At the time I was also an elected member of the first NANOG Steering Committee, an organization that I will discuss in a minute. So what qualifies me to talk about NANOG History?
I worked at Merit for 11 years (1987-1998), working on the NSFNET backbone project, the precursor to the modern Internet. During my last few years at Merit, I chaired NANOG and created the first NANOG business plan , articulated this spreadsheet.
I take pride in starting some of the early NANOG traditions including
All of these were new ideas and at the time caused great concern to those who didn't like change.
The point is, I lived through a lot of this early history and can provide some perspectives from both inside NANOG (when I worked on NANOG at Merit) and as someone outside of NANOG (during the last 10 years that I worked at Equinix and participated in NANOG). While I was at Equinix, I chaired 17 NANOG Peering BOFs ( click here to see some sample Peering BOF meeting minutes with photos, courtesy Matt Peterson June 2007 ),
introduced the "Great Debate" format to NANOG, and gave over 150 talks at conference around the world. I have a lot in the way of internal and external views on the NANOG as it compares with other fora.
Since a large percentage of NANOG attendees have always been first timers – maybe 30%-40% or so, this (web page and) video are intended to give these new folks some perspective of where NANOG came from - documented before it entirely evaporates from my aging memory cells.
NANOG has evolved over the years based on the feedback from the community, which is sometimes self-contradictory, but has settled in as a 2-3 day (S-T, S-W) conference forum.
In the early days we used to use Sundays for tutorials to test the A/V equipment to make sure that everything worked properly, well before the real meeting started Monday morning. We did this also so the hotel would not rent the room to someone else during the day on Sunday - that would have forced us to set up late Sunday night. We did that once or twice with folks up into the wee hours of the morning.
Since then, the Sunday tutorials and other uses of Sunday times lots have proven very valuable to the community. In fact, for some participants, this provides the added training benefit that allows them to come to NANOG! Since the NANOG community benefits from being able to meet/work with the broadest possible NANOG community face-to-face, the NANOG Sunday agenda has proven to be a big win for the community.
The majority of the content though is provided Monday through Tuesday and/or Wednesday, formed as a mix of large group plenary, small group tutorials, and birds-of-a-feather sessions. There are typically several socials scattered in the evenings to provide more chances to meet with and interact with peers. Many folks find that this face-to-face time is the greatest value and spend a lot of time in the hallways. There have always been unsanctioned, unofficial and slightly discouraged private suite parties at NANOG as well.
So why is NANOG convened here, in any single specific location?
Since the Regional Techs days and on into the early NANOG days, most participants' travel budgets were funded by groups funded by the National Science Foundation. Merit tried to spread the travel burden widely across the participants, so travel expenses did not fall too much on any regional groups. Merit used local hosts to handle the local arrangements, like identifying a good venue (universities and super computer centers in the early days) or hotels for the meeting (in the later days), making sure that Internet connectivity was available, that a regional airport was reasonably close, etc.
This distributed venue strategy has proven successful, but there have been challenges over the years. In almost all cases the hosts have done a great job. They handled all the logistics so the Merit NANOG support staff could focus on the internal meeting issues (registration, speaker and agenda coordination, making sure the rooms were set up properly, paying for cookies and coffee, etc.) In a couple cases, the local host volunteered and really wanted to host NANOG, but ultimately was too busy to handle the remote (away from Ann Arbor) work load. They were unresponsive, or otherwise did not provide the support Merit needed, so the venue had to be moved. The community has always stepped up to the plate, and there have always been alternative hosts willing and able to help out in a pinch, which is a testament I think to the value the community sees in this NANOG forum.
These kinks seem to have been worked out now that Gnawings are now planned a year or so in advance. And, as you can see by this map, the sites have been distributed across time zones over the years and we even expanded the North American hosts to the Dominican Republic.
In the early days, when caveman techs ran the original Internet core called the "NSFNET", the NSF awarded a contract to Merit to build and operate the new T1 (1.54Mbps) NSFNET with plans to migrate it to a whopping T3 (45Mbps) core to interconnect the NSF-funded regional networks across the U.S. as shown in the picture here.
As part of this contract Merit was to assemble the relevant regional network technicians to discuss the backbone core operations and performance as seen from the inside perspective (at the core NSFNET) and from the exterior perspective (from the view of the tributary NSF Regional networks). So we (Merit) used the "Regional Techs Meetings" to facilitate this interaction. Sue Hares ran these early meetings. The NSF funded Merit to provide logistics support for the meetings, assemble a useful agenda, and work with local hosts as appropriate, as stated before, to make sure no one was unduly burdened with travel expenses.
This worked during the contract term from 1987 until 1994 when the Internet growth led the NSF to the conclusion that the Internet no longer required government funding and could be continued as a self-sustaining private sector endeavor.
One of my first tasks as part of this transition away from NSF funding was to build a business model for the self-sustaining commercial version of the Regional Techs Meeting, called the North American Network Operators Group (NANOG), a name created by Elise Gerich and Mark Knopper, and reflective of the broader non-AUP nature of the new Internet core.
In this new model I named myself "NANOG Chair" mostly after the IETF Working Group chair function I mimicked. I set the agenda with the assistance of much smarter people than me - initially Merit staff, and later extended to helpful people that attended most of the meetings. We asked the community for hosts, volunteers to provide hotel and network. Merit used a $150 registration fee to offset the local host hotel catering and room costs, to pay for my silly NANOG T-shirts, and for systems support for the NANOG mailing list and the web site, etc. With this new business model , we covered the costs of NANOG operations with out government funding and NANOG continued to grow.
I called this model the "Strong Chair" NANOG model because I could pretty much do what I (thought the community) wanted in terms of agenda, logistics, location and timing, etc. When people complained about the content I recruited them to help me recruit agenda items, to volunteer to speak. Ultimately, when it came down to it, I was a strong chair that made the decisions. On occasion this meant doing something that the program advisors didn't like. From my perspective, it was good to be able to override the sometimes conservative volunteer program committee members and try something different. Committees tend towards conservatism. Sometimes things I tried worked, sometimes they didn't.
When in doubt on how the community would respond, I asked them. For example, we did a NANOG at George Washington University, in an auditorium used on occasion for shows like CrossFire and various shows distributed by CSPAN. We considered inviting CSPAN to capture a NANOG since they were already set up with all the equipment ports, and experience with the venue, etc. I did some inquiries with the CSPAN folks tofind out if/how that is done. Armed with this info, I asked the community what they thought of the idea. There was some concern raised that in this context, it wouldn't be the engineers speaking, but the degree of exposure would lead companies to replace engineer speakers with marketing speakers to ensure the right message went out. Another concern was that the cameras, lights, etc might change the "feel" of the venue, and maybe lead to less dialog. While I loved the idea of getting NANOG on TV, I decided not to pursue this path.
As mentioned earlier, we also tried the Beer-n-Gear event in an attempt to facilitate the interactions between the attendees and the vendors while getting some revenue to keep the registration fees from rising. No one wanted the trade show feel and the creeping commercialization concern was raised. This has always been one of the NANOG delicate balances:
How to accept commercial sponsorships and simultaneously make sure that commercial interests don't change the cooperative nature of NANOG? The older folks wanted to keep the academic feel of NANOG, often pointing to ISPCon/InterOp/EduComm/etc, where the trade show and party aspects dwarfed the learning and exchange of ideas aspects of the event. My view is that we still don't really know where that boundary is between what the vendor market wants and what the NANOG community will bear.
When I was chairing NANOG in the early days, we tried a bunch of new things, including beer-n-gear. We pretty much had to use the hotel services and catering - the costs were pretty high but the sponsors seemed to have the marketing money to get in front of the attendees. Then we started seeing more quasi-commercial activities we hadn't seen so much in the gov't-sponsored NSFNET days :
1) We started seeing folks having suite parties, in a couple cases these competed with the agenda or with the sponsored socials or BOFs. When I asked about their motivation, just to understand why, the answers for having these parties instead of participating in beer-n-gear were varied but seemed centered around the cost - that their little gathering was maybe one-tenth the cost of participating in beer-n-gear and everyone seemed to have a better time in this informal albeit cramped environment. To me, these parties felt more like a college parties vs. a formal event, and I personally liked the feel of these parties too.
We (the NANOG team at Merit) had to decide how to deal with this We had really three options: a) do we play hard ball somehow to prevent the parties? The hotel didn't like them either as they didn't generate any $ for them. b) Or let it slide by quietly ignoring (not condoning) the behavior? c) Or do we enjoy the party with the rest of the participants?
What actually happened was that people Merit folks were simply not invited to these parties for fear of what their attitude toward the party could be. There was a kind of "hope we don't get caught" on their side and our (personal) desire to socialize (be invited to the party) like everyone else while (Merit NANOG hat) making sure events didn't clash and the beer-n-gear sponsors didn't bail on the formal events. I think during my stead we slide towards enjoying the parties that we heard about, and a sort of *unwritten rule* emerged that the parties shouldn't clash with the scheduled agenda events. There was another kind of awkwardness as folks wanted to not clash, but didn't know when things occurred, so these unauthorized party organizers awkwardly had to keep checking the agenda to make sure their little parties didn't clash while not tipping their hat to Merit that they were doing something unsanctioned here. Even with this awkwardness, everyone kind of agreed and things kinda ran smoothly.
2) We started seeing people quietly passing out logo'd and funny t-shirts, one of the benefits we marketed to beer-n-gear sponsorship prospects. This too, during my time we let slide. What were we to do - police the event for T-shirts, vendor giveaways not done at the sanctioned times? What fun would that be? And for a 501.3c not-for-profit staff (not work for serious money compensation or stock), being aggressive about things like this tends to go against the personality grain.
3) And yes, over the years there have always been a few crashers - people attending the event without registering or paying. The question it seemed to me was the extent of the violation - how long were they there, did they eat or drink beer or get t-shirts at beer n gear, etc.
In one incident we know about, a person stopped at the event to say hi in passing, was actually called to the mike to answer a question and then community name-and-shamed / chastised the person for not having paid.
In another incident we know about, a person hung out in the lobby and was called out for reaping some of the benefits of NANOG (access to the population of people attending). To some it didn't matter that zero resources were consumed.
In the recent incident, a person looking for a lunch date with a person he wouldn't recognize asked for help meeting the person. I assisted in his failed search. He was there for only a few minutes and left.
One thing in common - These things sometimes causes some degree of uproar as everyone had an opinion as to where the line was. In most of these events, what seemed to cause the most problems to me was *how* the folks in charge of NANOG responded - if they did nothing, then people (especially people who paid with their own hard earned cash) felt a little cheated, and if the folks running things over reacted then the community responded with resentment of authority. This IMO was overreaction was one of the straws that broke the camel's back and helped roll the ball down the path towards revolution.
In 1998, I left Merit to help start Equinox and a sequence of NANOG Chairs from Merit stepped up to chair NANOG, each continuing this strong chair model.
One of the Merit strong NANOG chairs was not as responsive to the community as was needed, and perhaps applied a bit too much pressure on some activities such as mailing list administration (there were some bannings here) and at-NANOG commercialization activities (such as vendor logo'd underwear) than was appropriate. One long time attendee showed up at the hotel and hung out in the public areas of the hotel, but did not register or pay to attend NANOG - he was publicly scolded for taking advantage of the NANOG event without paying. This led to an informal referendum on the event and reaction, and the community had an allergic reaction. For these and a variety of other reasons, the community reached a tipping point that led the movement to restructure NANOG to be the communities' NANOG, not as an activity that Merit ran entirely as they saw fit.
Others have used the more polite term "NANOG Evolution", but the heat was substantial so I call it a revolution.
The goals of the revolution movement were to provide the community with transparency to how NANOG was run. When things happen, it must not be quite a mystery who is in charge of what. It sought accountability, a better agenda, and to make sure NANOG was more Community-driven and responsive to the needs of the evolving North American Internet. My benevolent dictator model crumbled.
So the model that was developed and codified in 2007 in a NANOG Charter that included 6 elected members from the community that were tasked with, among other things, selecting a set of program committee members from the NANOG community and selecting a set of mailing list administrators.
I conceptualize the new organizational structure with this evolution of the previous diagrams.
Instead of the Merit Strong NANOG Chair hand selecting program advisors and making the agenda, the NANOG Community elects the Steering Committee (SC) advisors shown as the inverted trapezoid connecting the NANOG community to the Merit NANOG operation. To solve the accountability and transparency goals, the COMMUNITY is charged with electing its NANOG Steering Committee members to act in their interests. Specifically this means that the SC selects a Program Committee (PC) from the Community and selects a Mailing Lists Committee (MLC) team to oversee the NANOG Mailing List. The Program Committee in turn puts together an appropriate agenda for the community.
I was one of those fortunate enough to be elected to the first NANOG Steering Committee, as part of this grand evolutionary experiment; we were the guinea pigs to see how/if this new machinery would work.
For transparency, the SC posts meeting minutes, and encourages the other groups to do the same to the extent reasonable. The team of the SC, PC, and MLC working with Merit has been mostly successful in my opinion, similar to a couple moving in together shortly after marriage. Both Merit and the various committees were getting used to one another. We were still working through issues such as "how transparent" and "how and when the SC-Merit relationship needs to be evolved, and who needs to be informed or involved in which activities associated with NANOG operations.
The end goals of the revolution were mostly achieved. The community had more visibility into the activities behind NANOG, with Steering Committee meeting minutes being posted and a recurring Sunday afternoon session called the "Community Meeting" where in theory issues can be brought up and discussed with the organizing groups behind NANOG. I say in theory because, as it turns out, some are uncomfortable discussing things in the open like this.
There is a fundamental dilemma between the need for Transparency and the desire to be sensitive to the Politics of any given situation.
As for accountability, the community has the right to elect or not re-elect Steering Committee members, and the term limits prevent the status quo from continuing indefinitely, and in effect require periodically new blood to the organization.
At the same time it is worth pointing out that this experiment in self-governance was not without failings. And we should recognize, and even celebrate that we recognize that we didn't get everything right the first time.
For example, with this model, when does Merit need to involve the SC? When the SC doesn't know something, they don't know that they don't know it ! This is a fundamental failure with the advisory role nature of the Steering Committee.
There is a lot of detail regarding selection of host, location and the number of meetings per year that Merit could derive from SC interactions. Should Merit just do what they do, or actively seek to engage the SC for opinions at every turn? What is their motivation? To keep things running. At times, to Merit, it might seem like the SC is dangerously close to micromanaging them, but to the SC it might seem like we have enough detail to represent to the community that we are doing our elected jobs.
With the Steering Committee oversight of the Program Committee – how does the Steering Committee get enough information to know who should be reappointed and who should not? One view is that the SC member participating on the PC should advise the SC on this, but if that were the case, then why not have that SC member make this determination alone? What value does the broader SC have when they have zero visibility into the internal workings of the Program Committee?
How does the Community know what the SC actual did for them? If the meeting minutes are sanitized to protect privacy then some crucial decisions are not available for the community or the next generation of SC members.
Anyway, to me, this should give you a flavor of some of the open questions.
Next I'll share some tidbits of history, or better described, NANOG folklore.
Back on 12/4/95 Bob Metcalfe attended his first NANOG at George Washington University and observed the geek crowd discussing some difficult scaling challenges with the transition to commercialization of the Internet. Specifically, there was packet loss at some of the exchange points, due to head-of-line blocking. Bob Metcalfe believed the insight to be documented was that there were problems with the Internet that the engineering crowd could not solve without the executives (the guys with ties and suits).
Bob's citation inspired one of my first T-shirt designs. My response was to make all the NANOG attendees wear T'shirts with stenciled ties as shown below.
When the Internet did not spectacularly destroy itself, Bob was going to eat his words. I spoke with Bob Metcalfe before etching his image onto a T-shirt - he loved it and asked for a T-shirt. That is still on my to-do list. Here is that T-shirt:
On the left here we see the tie thus meeting Bob Metcalfe's criteria for more ties at NANOG, along with a graphic depicting the end of the Internet, with engineer and network grim reaper arm in arm. Alien business people are shown showing up to save the day. The theme for these shirts was based on Bob's notion that the thing that NANOG really needed was more suits and ties, more business people who could make real decisions as quickly as needed to avoid a collapse. Walking through the Ann Arbor Art Fair that week, I saw a man walking around in a Sandwich board with the writing "The World is going to Collapse in 7 days - Repent!" --- and I thought, cool !
Anyway - those are some of the highlights of NANOG from the Regional Techs Meetings through a couple forms of NANOG organization.
UPDATE: Now, in 2010, there is a movement to migrate the Steering Committee "advisory role" into a more directive role. Stay tuned to the nanog-futures mailing list and participate in the new organization creation if you are so moved.
Internet Transit Pricing Historical and Projections
Index of other white papers on peering
Chief Strategy Officer, International Internet Exchange (IIX)
WIlliam B. Norton is a consultant, author of The Internet Peering Playbook: Connecting to the Core of the Internet, and highly sought after public speaker and international recognized expert on Internet Peering. He is currently employed as the Chief Strategy and VP of Business Development for IIX, a distributed exchange point provider. He also maintains his position as Executive Director and Consultant for DrPeering.net, a leading Internet Peering portal and consultancy. With over twenty years of experience, Mr. Norton focuses his attention on sharing his knowledge with the broader community in the form of presentations, Internet white papers, and private on-site consultations.
From 1998-2008, Mr. Norton’s title was Co-Founder and Chief Technical Liaison for Equinix, a leading Internet Peering Exchange and colocation center provider. From startup to IPO and until 2008 when the market cap for Equinix was $3.6B, Mr. Norton spent 90% of his time working closely with the Peering Coordinator Community. As an established thought leader, his focus was on building a critical mass of carriers, ISPs and Content Providers and documenting the core values that Internet Peering provided, including the Peering Break Even Point and Estimating the Value of an Internet Exchange.
To this end, he created the white paper process, identifying interesting and important Internet Peering operations topics, and documenting what he learned from the peering community. He published and presented his research white papers in over 100 international operations and research forums. These activities helped establish the relationships necessary to attract the set of Tier 1 ISPs, Tier 2 ISPs, Cable Companies, and Content Providers necessary for a healthy Internet Exchange Point ecosystem.
Mr. Norton developed the first business plan for the North American Network Operator's Group (NANOG), the Operations forum for the North American Internet. He was chair of NANOG from May 1995 to June 1998 and was elected to the first NANOG Steering Committee post-NANOG revolution.
Connect on-line: http://DrPeering.net – RSS: http://ask.DrPeering.net – e-mail: wbn@TheCoreOfTheInter.net
The Peering White Papers are based on conversations with hundreds of Peering Coordinators and have gone through a validation process involving walking through the papers with hundreds of Peering Coordinators in public and private sessions.
While the price points cited in these papers are volatile and therefore out-of-date almost immediately, the definitions, the concepts and the logic remains valid.
If you have questions or comments, or would like a walk through any of the paper, please feel free to send email to consultants at DrPeering dot net
Please provide us with feedback on this white paper. Did you find it helpful? Were there errors or suggestions? Please tell us what you think using the form below.
Contact us by calling +1.650-614-5135 or sending e-mail to info (at) DrPeering.net