During 2006, the "digital universe," i.e., all the bits and bytes created by everyone, everywhere, was 161 billion gigabytes or 161 exabytes. Over the next four years, IDC predicts that number to grow six-fold every year for a 57% CAGR.
To get an idea of just how much information is flowing around globe right now, 161 exabytes is "approximately three million times the information in all the books ever written," according to IDC.
"In a sense, this is really just putting a number on what people would have told you before. The bigger discussion is really around all the implications around this and what it means and what they need to do about dealing with all this data."
And that's the rub. While most information is being created by individuals, some 70%, 85% of that information is being touched by organizations, corporations and government bodies in some form or another. This then raises a whole host of questions over responsibility for the information and, therefore, how to deal with it.
"If it touches the organizations network anywhere then they can be held liable from a compliance point of view, from a legal standpoint," said Minton. "If that information is crossing their network then their responsible for it."
The Growth of ILM & BI
In an attempt to get a handle on all this data, corporations have turned to information lifecycle management (ILM) concepts and tools, and business intelligence (BI) suites. Which accounts for those two market segment's rapid growth over the past couple of years, said Minton.
But these two technologies are still, relatively speaking, in their infancies, said James Short, research director at the Information Storage Industry Center (ISIC) of the University of California, San Diego.
"ILM is the first step" towards a solution, said Short. "It's essentially a collection of a series of hardware and software innovations in storage that allow for a more economic utilization of both the hardware and storage medium as well as a first phase attempt at defining policies that will allow you a more policy-based storage approach in your IT organization."
Also, since the biggest problem with so much data isn't necessarily storing it or the cost of storing it Moore's Law has seen to that it's deciding what information is valuable and what it not, said Roger Bohn, ISIC's director. The volume only exacerbates this problem.
"Somewhere along the way classification of the value of that information has to be reconciled before the IT organization can make an economic system for storing that information," said Bohn.
While the vendor community works feverishly to develop the needed automation to tackle such a vast problem, their attempts so far have only begun to scratch the surface, said Short.