The New NOC
Last fall I wrote about the changing state of the network management marketplace. This summer we did some research to test out a few hypotheses and explore a few others about network ops and the role of networking more broadly. The results confirmed some of my ideas and surprised me in a few waysaddressing areas ranging from application management, to the management of remote locations, to changes in organizational structures, to changes in attitudes and patterns of collaboration.
So, you may ask, what does this specifically have to do with your role as CIO? Well, for this article, Ive tried to select what I think are things you should care about because the roles, responsibilities and processes within networking are changing in many IT organizations. These changes potentially impact not only the effectiveness of network operations in of itself, but much of the rest of your IT organization. Ive chosen a handful of examples drawn from interviews and data that you might use as a litmus test but what it comes down to is one question: is your network team a leader, helping to pioneer new types of cross-domain collaboration or is it an Archie Bunker hold outclinging to the past and the notion that the network is a separate fiefdom with castle walls of politics and attitude protecting it from change?
First of all, based not only on our data, but on our active consulting work with IT organizations, Network Operations and its various affiliates, engineering, design and capacity planning, or IT infrastructure management isnt just about managing network devices in most shops. Our respondents (all of whom were affiliated with managing the distributed network infrastructure and the application services it supported) showed quite a range of skill sets. This included 70% of respondents with skills in systems management, 66% with security skills, 63% with strong backgrounds in change and configuration, 55% with storage backgrounds, 54% with application management backgrounds.
Overall, what seems to be going on is a move to manage the distributed infrastructure more holistically in support of the delivery of application services. This may sound cliché, but for years network managers have cared deeply about server performance with a growing attention to application services. Whats new is these trends seem to be accelerating. For instance, 81% of those in network operations feel that theyre responsible for managing critical servers in the data center. And 74% feel theyre responsible for managing some or most systems and other non-network devices in remote locations. Of these, the majority are responsible for the full lifecycle management of these devices, including active configuration!
The story on managing applications is just as interesting. In our survey, we asked our respondents about the veracity of the following statement: Informal (or formal) teams/meetings emerge to tackle problems regarding networked application service performance. These teams combine network, application database and other data center personnel. Eighty-two percent said this was either very true or somewhat true. What might be a shocker is that 46% claimed that network operations led these teams most of the time, while an additional 39% said that network operations led these teams only some of the time.
Taking the Lead
Networking is taking the lead in these triage teams in many IT shops for two fundamental reasons. First, its the first place of blame when users cant access information or application services. And thats, of course, not new. But what is new or new-ish at least is the second reasonnetwork operations tends to have the best triage tools to see where an application problem lies. In dialog with respondents, or in consulting, these are almost invariably flow-based tools that look at application traffic in terms of volumes, response time and in many cases even transactions. These tools often offer visibility from the application server right down to the receiving desktop. Nearly 70% indicated these triage teams were using these types of common tool sets to validate who owned the problem.
I should add that these triage teams are typically ad-hoc, they do not have documented processes, and while a number claim executive sponsorship, in most real-world cases this equates to some level of executive awareness and tacit supporttypically at the director or sometimes VP level. I would be curious to find out how many of you are aware of this phenomenon (feel free to let me know at email@example.com) but I suspect its a minority. Whats interesting to me is these efforts are very ITIL-like in spirit, supporting cross-domain collaboration in support of service delivery. But they are not at all out of the ITIL cannon. They are grassroots efforts almost invariably. In my opinion they should be nurtured and brought into more mainline processes.
But not everything is rosy in Network-land. There is at least one clear example of cognitive dissonance, the term for believing two contradictory things at once. While about 70% of our respondents said they used the same solutions proactively to support triage across domain, an almost equal number (one percentage point difference) said that they would not be comfortable sharing performance information outside of the networking group with others in IT. Of course, one question was specific and proactive, the other more general and point blank. The dominant reason for nay-sayingweve never done it beforewas followed by concerns about tool relevance and the very honest, we dont want anyone looking over our shoulders.
We also found out that networking overall feels that CMDB systems are as valuable to the networking organization as to the rest of IT, although there remained some fairly prevalent concerns about the viability of current offerings to support network ops personnel.
So, what does it mean when you add it all up? I would say that you have a clear case of a troubling-but-promising cultural revolution in the making. There are leaders and trends to be nurtured, and traditional ways of working and thinking that need to evolve beyond their castle walls. One executive I know just fired seven network engineers because they wouldnt come round to a more process-centric approach. But I believe that there are far more gracious and ultimately more productive ways to move networking into a more central and valued role within your organization.
Dennis Drogseth is vice president of Boulder, Colo.-based