The End-to-End Challenge

By Julie Craig

(Back to article)

In the IT world, it is easy to lose sight of the forest among the trees. Technology becomes more complex every year, and we're way beyond the point where a single person can understand all of the "pieces" required to run the average Web application.

Humans solve such problems by breaking down the big picture into consumable chunks. Despite investments in best practices focused on breaking down technology "silos", IT is generally still organized around technology teams specializing in managing infrastructure—networks, servers, and data bases. The problem with this approach is our job isn't really to manage technology.

Our raison d'être—the industry's reason for existing—is to add value that supports business goals and objectives. Most often, IT's primary value-add is the delivery of high-quality business applications. The problem arises when IT is unable to deliver these applications efficiently or effectively.

We are starting to see a trend in which demand for complex applications is outstripping IT's ability to support them. Online applications, and those built using Flash, web services, Web 2.0, and similar technologies are becoming increasingly ubiquitous. Convergence of business applications across networks also supporting voice over IP (VoIP), IP telephony, streaming media and unified communications has, to some extent, caught the enterprise management industry unaware.

Traditional enterprise management products, many of which have domain-focused roots, are still adapting to this converged, distributed, Web-based application focus. And although management products that address databases, networks, and servers are still a part of the big picture, the picture itself is much bigger than it was in the past.

While businesses are becoming increasingly proficient at developing applications, the statistics suggest they are not as successful at deploying and managing them. Enterprise Management Associates (EMA) research bears this out:

       Thirty seven percent (37%) of IT professionals report that they lack the tools they need to support their business applications.

       Unrelated research reports that IT organizations have an average of five, and as many as 25 or more, management products in-house.

       Forty three percent (43%) of application outages are still reported by users.

       Twenty four percent (24%) of IT organizations report that one of the biggest barriers to cross-functional application support is lack of effective diagnostic tools.

       Forty one percent (41%) of IT organizations surveyed report they prefer to use "expert opinion" over off-the-shelf application management products to diagnose problems. Only 21% preferred off-the-shelf products.

       Support costs consume 60% to 80% of the average IT budget. (Although this statistic is so commonplace as to have become a cliché, there is a good reason for these high costs. In lieu of automation, people are required to close the management gap—and people are expensive.)

The End-to-End Challenge

The high cost of support is one indicator that many vendor products and/or IT support practices haven't yet caught up with application complexity. In terms of products, the industry seems to be aware of these issues and has coined the term "end-to-end" to describe the types of products designed to address them. However, while multiple enterprise management vendors market end-to-end solutions, each vendor has a different take on what the term actually means. As a result, it is difficult to get past branding and advertising to decipher exactly what a given "end-to-end" product actually does, or to figure out where it fits within the arsenal of management solutions already in-house.

The term generates more questions than it answers:

       What does end-to-end really mean?

       Which part of end-to-end don't we understand?

       Does any single product solve this puzzle?

       We bought an end-to-end product—why can't we manage our applications?

A Multi-Dimensional Solution

Effective application management requires visibility to the application forest as well as the infrastructure trees. This requires depth of knowledge that encompasses both infrastructure and application performance. When IT relies primarily on people and their expertise for this knowledge, it comes as a result of cross-functional efforts among multiple technology teams. This is one reason why IT organizations have heavily invested in business process optimization projects, built around the IT Infrastructure Library (ITIL) or other best practice disciplines—and also a reason why IT administration costs continue to spiral upwards.

From the tools perspective, this depth of knowledge comes from building high levels of cross-functional expertise into automation. Long term, this is the most efficient and cost-effective way to manage increasing levels of application complexity. Short-term, some vendors, as well as some IT organizations, are having trouble making this transition.

EMA has worked with multiple IT organizations feeling the pain of the end-to-end challenge. With this in mind, EMA analysts specializing in application management, network management, and configuration management have collaborated to develop a cross-functional "End-to-End Application Management" model.

The model describes the vantage points and product categories necessary to analyze application execution in a way that provides a true end-to-end perspective. Combined, these vantage points yield the multi-dimensional application intelligence required to manage the broad spectrum of applications that execute in the enterprise today, in context with the infrastructure that supports them.

That being said, there is a world of difference between managing legacy applications running primarily on the mainframe, and managing SOA transactions executing both within and outside the enterprise firewall. While simpler applications might not require products from all five categories (see below), very complex application deployments very likely will. The model is designed to address this broad spectrum, with the expectation that most IT organizations are supporting multiple types of applications, and require the ability to manage all of them. It also assumes that, for the present at least, building this big picture will likely require multiple, interoperable solutions.

EMA's Approach

The model defines a taxonomy of product categories, each of which contributes its own unique perspective to the management fabric. For example, element managers are still important for monitoring and gathering platform-specific health metrics. However such "vertical" components are of minimal value without context to the applications they support. Likewise, products that provide a transaction-focused perspective are of limited value without visibility to the infrastructure elements underlying the transaction. At its most granular, the model is designed to expose these relationships in an automated fashion.

The model includes five broad product families, all of which are supported by, and contribute to, a federated configuration management system. These product categories include:

       Data Center Analytics: Traditional infrastructure monitoring and management products that discover, identify, monitor, and manage infrastructure and foundational technology elements.

       Application Discovery and Dependency Mapping: Products that discover, identify, and map applications, their dependencies and their relationships. Typically use application "signatures" or "fingerprints" to detect and identify applications.

       Application Code Analytics: Products that analyze application code and apply that analysis to discover, identify, map, and manage custom applications.

       Networked Application Management: Products that track transactions (including SOA) in an automated fashion by observing network flow, "real" or "synthetic" transactions.

       Application Integration Analytics: Products that track transactions through integration layers, technology touch points, middleware and across variable transaction paths.

       Federated CMDB: While not an application per se, the federated configuration management database (CMDB) ties together the information collected from multiple sources into a "big picture" with context to applications, the transactions that constitute them, infrastructure, cross-functional relationships, and organizational metadata.

Products currently available to address these dimensions are at varying levels of maturity. For example, products within the Data Center Analytics category are widely available, while relatively few management products can track a transaction through middleware and across dynamic execution paths. Likewise, some of the middleware management capabilities that are critical to managing SOA and web services applications are still scarce, and many management solutions do not address application dependency mapping at all.

EMA believes this is very much an evolving domain, and that remaining gaps will be filled over time. Meanwhile, working towards a vision of managing an application fabric, not simply point technology, helps bridge the gap between "as is" and "to be". Further information on EMA's research regarding the End-to-End Application Management model, and on the May Webinar detailing this new approach, is available at www.emausa.com.

Julie Craig is a senior analyst with Boulder, Colo.-based Enterprise Management Associates (www.ema-usa.com), an industry research firm focused on IT management. Julie can reached at jcraig@enterprisemanagement.com.