Solving the Problem of Large Project Failure

By Cliff Berg

(Back to article)

It is well-known that as software projects grow in size, their probability of failure escalates dramatically. (For our purposes here, “failure” means failing to meet its objective, either by being scrapped, or by the goals being substantially scaled back. A lesser form of failure is when the project drags out far beyond its original time line, by a factor of two or more, and at a cost many times it original estimate.)

So, what is it about large projects that is so difficult? Frederick Brooks pointed out in his seminal work the Mythical Man-Month that small teams are much more effective than large teams. But why? What is it about large teams, and large efforts, that is so intractable?

The “agile” software development movement is a reaction to much of what is perceived as dysfunctional about traditional methods of software development, and the use of large, process-intensive teams creating documents is one of those practices that is rejected by the agile movement. The feeling is that projects are better off if one simplifies how people coordinate their work. The best way to do this is to put the entire team in the same room and make sure that they talk. Documentation is largely dispensed with. The results are impressive. My former company Digital Focus (now Command Information) built its business around this approach.

Not every project can be run this way though; some tasks are large and require large projects and you cannot put 100 or 1,000 people in the same room. This problem must be solved so that large endeavors can be reliably undertaken. I contend that one of the major causes of the failure of large software projects is the use of a document-centric approach throughout the software lifecycle: from the development of requirements through the development of maintenance materials.

Not Knowledge

Documenting is a great aid for the documenter. As the author of four lengthy books, I can tell you the process of writing compels the author to organize one's thoughts. In a software project, the creation of documentation should never be seen as a milestone of progress. This is because documents are not actionable. Documentation of requirements or business processes does not convey knowledge. At best, it is only a starting point. Disbanding a team that assembles requirements as documents squanders the knowledge that team accumulated.

Only knowledge is actionable. Knowledge is the result of humans interpreting information – from documents, from the explanations of others, and from their experiences – and building understanding in their minds over time. The phrase “people, process, and technology” that one often hears is therefore misleading and dangerous. The flaw is in the concept of process, which is often taken to mean process documents.

I have seen too many projects planned around the concept that you just have to have someone create documents, and then bring in people to execute based on those documents, using some provided technology. It does not work that way, and that type of thinking leads to failure. You need to focus on the knowledge that is created as a result of creating documents, systems, and the other durable elements of your organization. The act of creating information creates knowledge, and that knowledge is the core of your business process; and that knowledge resides inextricably in those individuals.


Measuring Progress

If not documents, then what? Two important kinds of outputs of a slice of a software development process are shown in Figure 1: (1) knowledge and understanding on the part of the system builders, and (2) documentation of that understanding.

Figure 1: Milestones defined as measurable increases in knowledge.

Documents are merely evidence that a person has performed certain intellectual activities. For example, a test plan is evidence that a test planner has enumerated the tests that need to be done, and explained their rationale. However, one does not know if test planning is actually complete (has sufficient coverage) unless someone credible and impartial assesses the plan. That is, the plan needs to be verified.

Progress should be measured through tangible outcomes whenever possible, or through independent assessment when there are no tangible outcomes. The outcomes or the assessment are the credible indicators of progress, not the documents. For example, how do you know whether a design is robust enough to proceed with development? The assertion that a design document has been completed is not a reliable indicator, because it is well-known in software development that designs evolve substantially throughout implementation.

How then can one tell whether one is at a point at which proceeding with development will be productive or lead to lots of rework and perhaps even scrapping a first attempt at building the system? Prototypes are useful for this purpose, and so the successful completion of prototypes that address critical design issues is a better indicator of readiness than the completion of a design document. In any case, progress should be seen in terms of the creation of actionable knowledge, not artifacts.

The Scaling Problem

As projects scale, the effects of a document-centric process become more prominent, because those who create the documents tend to be less available to answer questions. Teams create documents and pass them on to other teams, and the original teams are often re-deployed to other activities. They might even be located at a separate site. Programmers, testers, and others are expected to pick up the documents and work from those alone. It is as if someone sent you a book of calculus and said, “Here, build a program that implements this.” No wonder large projects tend to fail. Due to pressure to optimize the deployment of resources, large projects tend to consist of many disjointed activities inter-connected by the flow of documents. But, since documents are information and not knowledge and are therefore not actionable, these flows tend to be inadequate.

Agile methods have been extended to large projects. For example, see Scott Ambler's article Agile and Large Teams. Ambler is Agile Practice Lead for IBM/Rational and tends to work on very large projects. The basic approach is to decompose the project into sub-projects, define interfaces between the associated sub-components, and define integration-level tests between these sub-components. This is very much a traditional approach, except that documents are not used to define all of this ahead of time. Instead, the focus is on the existence and completeness of the inter-component test suites, on keeping interfaces simple, and allowing interfaces (including database schema's) to evolve while keeping the inter-component tests up to date.


Documents & Maintenance

Artifacts are necessary for a system to be maintainable, but think of them more as a reference or as training materials, not as the primary mechanism for ensuring maintainability. For a system to be maintainable, knowledge must exist in the heads of those who will maintain it. If only documents exist, there is a substantial risk the system is not maintainable; either because the documents are inadequate, or they have not captured important design strategies (this is extremely common), or because the system is complex and it is simply too difficult for the maintainers to learn how to maintain the system from documents alone ... or all of the above. Also, documentation of the internal design of software is notoriously inaccurate.

In order for a system to be maintainable, there needs to be a plan for ensuring that the knowledge of how to maintain the system is transferred when the system is transitioned to the maintenance team. The plan must address how the knowledge will be maintained over time as the system evolves and as people come and go. A system of apprenticeship, in essence, is needed and the level of knowledge of the system's design must be tracked as an important asset to be maintained. This is usually not done, and, as a result, many legacy systems today are in a situation where no one can maintain them. Yet, no one can replace them because no one knows exactly what they do. The costs of this state of affairs can be large: inability to act (a lack of business agility), and chronic technical problems that cannot be properly addressed.

One strategy for preventing this untenable situation is to involve the individuals who will actually maintain the system in the creation of the maintenance plan, and simply pose the requirement that the system continue to be maintainable over time. In any case, both the system owner and the enterprise should actively track the level of knowledge and capabilities of the maintenance team, rather than treating this activity as a low-skill sideline task, as is often the case.

Please read more about how to create a non-document centric software development environment in my book Value-Driven IT.

Cliff Berg is a consultant in IT process, methodology, and change management, and a former enterprise system architect. Cliff has written four books, most recently Value-Driven IT, and before that High-Assurance Design. Previously Cliff was founder and CTO of Digital Focus, now Command Information.