The Rise of Shadow IT

By Hank Marquis

(Back to article)

The loss of competitive advantage from IT may not be entirely due to its commoditization. It is starting to become clear that at least some of the responsibility lies with business activities taking place outside of the control of IT.

Today, business users and knowledge-workers create and modify their IT infrastructures using “plug-and-play” IT products. These commodity IT products are now so easy to use, cheap, and powerful that business users themselves can and do perform much of the work traditionally done by IT.

But without the planning and wider view into the ramifications of their actions provided by IT this often results in disastrous consequences. Forrester Research found 73% of respondents reported incidents and outages due to unplanned infrastructure modifications.

Welcome to the gritty reality of commodity IT. Aside from the opportunity costs and operational losses resulting from this uncontrolled plug-and-play free-for-all, many companies are missing out on the competitive advantage potential that harnessing commodity IT delivers.

Within this disturbing new reality lie both the seeds of competitive advantage and a viable model for 21st century IT. In the Summer 2006 issue of MIT Sloan Management Review , I proposed in “Finishing Off IT” that even though IT is now a commodity it can and does enable significant competitive advantage. Resource dependency creates complex relationships between consumers and providers.

These interdependent relationships in turn produce organizational problems that require organizational solutions. Offered as a solution was the notion that management and organizational structure, not technology, hold the promise of sustainable competitive advantage from IT, and that manufacturing process control techniques hold a viable model for the future of IT.

21st Century IT

To visualize how a 21st century IT organization could look, it helps to consider the production and consumption of IT services as a manufacturing micro-economy.

IT manufactures information processing, communication, and collaboration products that underpin nearly all business operations. Knowledge-workers consume these IT products in pursuit of business objectives using everything from simple emails to more complicated core activities like forecasts and audits.

A deeper exploration of what actually occurs within the IT micro-economy helps to further clarify the issue. Based on real events I documented between December 2005 and July 2006, the following dramatization presents a composite of the experiences reported by a number of mid-to-senior IT managers.

On the way to the office your Blackberry vibrates. It’s a message from your staff. Users on the east side have been tech-swapping again. You know how it goes: “I’ll trade you this color printer for your wide screen monitor.” You know this is going to raise flags with the auditors.

You get to your office and there is a note from the service desk about that system outage on the west side. It turns out the system went down because its users bought some high-resolution scanners and connected them to the system themselves.

You didn’t even know they had scanners until they called demanding support.

Downtown, a group of users decided that to improve performance they needed to regularly transfer gigabytes of video from the main conference room uptown to a storage area network (SAN) they built on their own. As you suspected, these transfers were responsible for slowing down a business-critical application that has managers all over the company grumbling.

An email from the PMO informs you of a new project that will require extra support staffing starting in two weeks; first you've heard of that. You look at the calendar and sigh—budget and staff reductions, increasing user counts, more audits, increased legal regulations, major new and unplanned applications, connectivity and collaboration requirements, and very powerful and unhappy customers to placate.

So much for delivering the IT projects you did know about on-time and on-budget.

This “bad behavior” by the business amplifies the already accelerating velocity of change facing IT whether in-sourced or out-sourced.

The true nature of today's average IT environment is not pretty, and it’s not something most senior executives have fully grasped. It may also turn out to be a critical factor in obtaining competitive advantage from commodity IT.

Rise of the Knowledge-Worker

IT commoditization changes the balance of power between IT and the business, and within the business itself. Within the IT micro-economy of plug-and-play commodity IT, the consumer/supplier exchange relationship has shifted. This requires dramatic changes in thinking and management.

Traditional wisdom holds that the consumer for IT services is a functional business unit—sales, marketing, and so on—but, today, the real consumers of IT services are ad-hoc teams of knowledge-workers spanning multiple locations, and crossing business unit and corporate boundaries.

This shift in the exchange relationship has profound implications for the business and IT.

The underlying cause is the unstoppable commoditization of IT as advances accelerate productivity: The ubiquitous availability of information and internet technology is enabling knowledge-workers to traverse geographic, political boundaries, and now functional barriers.

Called “Shadow IT,” they are the millions of knowledge-workers leaping traditional barriers and asserting themselves in ways that challenge traditional IT departments.

Knowledge workers perform vital business functions like numerical analysis, reporting, data mining, collaboration, and research. They use databases, spreadsheets, software, off-the-shelf hardware, and other tools to build and manage sophisticated corporate information systems outside of the auspices and control of traditional IT.

By creating and modifying IT functionality, knowledge-workers are in effect supplanting the traditional role of corporate IT. However, they do so in a management and process control vacuum.

While the business can do these things due to the commoditization of IT, few executives ask if they should do them, and fewer say they must not. Virtually none realize the impact or import. Instead, to the dismay of IT staff, most senior executives and most CIO's condone virtually any demand the business makes.

This lack of control is responsible for many of the problems associated with IT today.

While the IT center-of-gravity has irrefutably shifted to the knowledge-worker, they do not have the long-term vision or awareness of dependencies and planning that IT traditionally provides.

The business wonders why IT doesn’t get "it" and ponders outsourcing when instead they should be taking responsibility for their own IT usage. No product IT can buy, and no outsourced IT utility, can handle these and similar issues encountered in ever-increasing numbers by real IT organizations.

Yet, it is precisely this consumer/supplier shift, increasing dependence upon IT, and the product-oriented nature of commodity IT that provides companies with the opportunity to leverage it for competitive advantage. However many senior executives have so far tipped a blind eye to Shadow IT, implicitly condoning the bad behaviors previously described—and they are throwing away any advantage that IT can provide.

New World Order

This lack of management control over business IT consumption has a tremendous cost. It is partly responsible for loss of the competitive advantage that IT can and does deliver, and is directly responsible for many lost opportunities, increased costs, and service outages.

Over time the erosion of perceived IT quality usually leads to outsourcing, which is increasingly seen as an incomplete solution at best, and a disaster at worst.

In order to recover and expand upon the advantages promised by commodity IT, senior executives have to change their concepts of an IT department, the role of centralized control, and how knowledge workers should contribute. The issue is fundamentally one of management philosophy.

The Nordstrom way promotes a customer/worker management philosophy where management’s first commitment is to the customer. The customer is always right in the Nordstrom way. This accurately reflects is the hands-off position taken by many senior executive leaders with regard to out-of-control Shadow IT practices and bad business behavior.

A better management philosophy for commoditized IT is the ‘Southwest’ way. In the Southwest way, the worker comes first. The customer is not always right, and Southwest has been know to ask misbehaving customers to fly another airline.

Management’s first concern is the worker, because they know that workers following sound processes hold the keys to customer satisfaction, and in turn, competitive advantage.

Making the Southwest model work for 21st century IT requires a more comprehensive view of what constitutes an IT organization, a view that extends well past the borders of what most leaders consider IT.

Shifting Demographics

The rising sophistication and expectations of knowledge workers results in divergence in perceived operational goals between IT and the business—an indicator of task-uncertainty and a key contingency within structural contingency theory.

These changing demographics give new urgency to the need for coordination of knowledge-workers and IT, yet management is trying to centralize IT spend and control via the CIO role.

Instead of embracing Shadow IT, CIOs are trying to shut it down. Consider instant messaging (IM), an application many knowledge worker consider critical. IT's approach to IM is reminiscent of the early days of the Internet.

Instead of realizing the job of IT is to support the needs of knowledge-workers, most IT organizations are trying to stamp out IM—just as they tried to restrict and eliminate Internet access. How will traditional IT respond to Wikis and blogs as corporate IT tools in the future?

The Corporate Executive Board projects that the percentage of IT spend under central control to grow from 50% in 2002, to 95% in 2006, but this does not take into account the knowledge-workers of Shadow IT.

A study by Booze Allen Hamilton found that shadow IT personnel equal as much as 80% of the official IT staff. Clearly, despite the best efforts of senior leaders and IT, the business stubbornly refuses to succumb to centralized IT control.

The problem with the current direction of the CIO role is that is typically has responsibility to support the business without authority to control the business; a classic management mistake leading to the aforementioned dilemmas.

The lure of commodity IT is great. Since shadow IT is a direct result of commoditized IT and resource dependency, it also demonstrates that both corporate IT, and IT utilities, are not delivering the services required by knowledge workers.

However, most IT leaders do not understand the strategic contingencies within the commoditized IT micro-economy. They don’t know their marketplace, and they don’t know who their customer is. In effect, IT is manufacturing the wrong products for the wrong market. IT doesn’t get it either.

How to Manufacture Competitive Advantage

IT must evolve its focus from pushing projects to engaging customers via process in order to become an internal outsourcer. IT needs to create an inventory of compatible services knowledge workers can assemble as and when they choose within guidelines established by the business and IT.

IT needs to shift from telling the business what to do, to helping the business choose, implement, and use their solutions the right way. How will traditional IT respond to Wikis and blogs as corporate IT tools of the future? The same way they treated Internet access and IM? Or will IT take its queue from the knowledge-worker?

The answers to these questions are completely under the control of senior executive management.

From a strategic point of view, IT must adopt processes and practices for the right strategic reasons. Along with the change in management philosophy toward increased business responsibility for IT consumption, the IT organization must also change.

A sound model for the future of IT may be found in manufacturing—when organizational structure aligns with production and consumption the result is competitive advantage.

A survey report on the future of manufacturing by Deloitte & Touche reads as if written instead to describe the future of IT. The report states: “To satisfy customers, manufacturers will require a fundamental shift in executive mindsets and organizational cultures.

Manufacturers must eliminate traditional boundaries between customers and integrate more closely with them. This means partnering with customers and emphasizing the co-ordination of research and development (R&D), marketing and manufacturing.

Successful manufacturers will integrate the customer into the fabric of their organization ... top performers are proactively changing the rules of competition to their advantage, and to their rivals’ disadvantage.”

IT can learn a lot from manufacturing—they have mastered the sourcing debate, dramatically improved product quality while simultaneously reducing costs in the face of a rapidly changing markets, and have overcome the issues of aligning customer need with production quality to deliver competitive advantage.

These mirror the aspirations virtually every senior leader holds for IT.

So what do manufacturing companies have that corporate IT does not? The answer is maturity. Manufacturing is so pervasive there are branches of economic science dedicated to it. Answers to questions on sourcing, optimum organizational structure, and achieving and sustaining competitive advantage lie in the economic theories and best-practices surrounding manufacturing.

Examining how manufacturing companies evolved and organized to solve problems in pursuit of competitive advantage provides a roadmap to attaining the same from commoditized IT.

Consider hardware manufacturer Dell. Dell transformed how their consumers acquire products by removing barriers and incorporating consumers into their manufacturing process by letting them design their own unique solutions based on a series of interchangeable parts, options and accessories around a core product set.

According to structural dependency theory, there is no one best organizational structure for all companies. However, in 2005 Forrester Research reported that the traditional IT organizational structure supported vertical business units and siloed applications with roles, responsibilities, skills, and budgets driven around project-oriented activities.

This vertical technology-driven structure of most IT organizations is a legacy of the days before IT commoditization and the subsequent dependence of business upon IT, and its one-size-fits-all structure erects a number of barriers which prevent it from working effectively with the business to realize competitive advantage.

Structural dependency theory suggests the best structure is unique to the needs of the organization and that companies perform best when organizational structure aligns with their unique contingencies such as cost, regulation, and marketplace. This is especially true for commoditized IT.

“IT is changing the way companies do business, making them smarter, more productive, and more profitable,” said Diane Berry, managing vice present at Gartner. “But some companies manage their information technology better, faster, and smarter than others and a big part of the reason is the effective structure of their IT functions and the deployment of IT staff.”

Just Like Dell

The practical objective of the 21st Century corporate IT department must be to transform from offering static “one-size-fits-all” project solutions to allowing knowledge-workers to choose and construct the IT services they need dynamically, as did Dell.

Simply put, long-term goal orientation toward win/win relationships is more effective than short-term transactional relationships. This concept is vital to achieving competitive advantage from the IT organization.

An article in the Journal of Business & Industrial Marketing documented that many managers see business relationships through a war metaphor and their strategies derive from military philosophies. The article suggests: “Rather than bemoan the fact of dependency, we should seek to set that dependency in the context of an overall relationship strategy.”

Interdependent relationships are balanced or unbalanced. Unbalanced or outcome interdependence is competitive (win/lose); balanced or behavior interdependence is cooperative (win/win).

Interdependence causes uncertainty, and organizations take steps to reduce uncertainty, for example, choosing alternative suppliers. The actions taken in response to uncertainty can affect other organizations positively or negatively.

Positive solutions to outcome interdependence include increased coordination and shared control of resources, known as behavioral interdependence. Think of such interdependency as a form of branding, for example “Intel inside.”

While this may be frightening and challenging to IT, they can follow the World Class Manufacturing model by delegating the authority over what services they deliver to the knowledge workers, while retaining responsibility for how it delivers those services.

This is hard for IT to accept since they have always “known what’s best” for users, customers, and the business, and are still inclined to think users and customers don’t know anything about IT.

The business units in turn must accept responsibility for assembling and using IT services appropriately. History indicates that sourcing decisions based on task and process uncertainty deliver the most value, and thus outsourcing certain IT tasks and processes to knowledge workers is critical. However, they must realize they do not have the training, visibility, experience, or right to perform ad-hoc modifications to IT infrastructure.

Senior executive leadership must enforce and support this division of labor. The catalyst for successful behavioral interdependence is management commitment, and without managing business consumption, the result is chaos.

The successful 21st century IT organization must be a federation between decentralized knowledge-workers and centralized corporate IT, with engaged senior executive leadership and guidance.

The lessons of manufacturing economics—customer choice, supply and demand, quality and production, and organizational contingency theories—apply equally to IT. World Class Manufacturing, and its Kanban philosophy as exemplified by Dell provide a clear path for IT and the business.

This realization becomes a transforming moment for IT and the business, and while it represents uncertainty, all involved must embrace and nourish their interdependence.

Commoditized IT holds the keys to sustained competitive advantage, but obtaining it requires a significant organizational and managerial shift by all involved. Only senior executive leaders have the power to welcome, encourage, and initiate this transformation—the final commoditization of IT into a high-quality manufacturing organization.

The result is world class IT: Sustained competitive advantage from the organized interdependence between teams of decentralized knowledge-workers and centralized IT, formally supported and managed like a modern high-quality manufacturing operation.

Hank Marquis is a managing partner and CTO at itSM Solutions. You can contact Hank at hank.marquis@itsmsolutions.com.