Lean management failed in the automotive industry because stable systems do not respond well to change. For example, the lead time from requirements to implementation was typically around five years. This lead time prevented American companies from using the most recent technology in automobiles.
This lead time was acceptable when change tended to be slow. However, the industry was incapable of responding to sudden changes in customer demand such as fuel economy during the energy crisis of the '70's.
The pace of change is amplified a thousand-fold in the software industry. Massive change results from missed requirements, improper design decisions, changing technology, changing competitive landscape. In addition, competition appears suddenly because the barrier to entry for software is much lower than it is for manufactured goods.
Paradoxically, phasist process and metrics attempt to fit software development into a stable system by controlling the phases through the artifacts that are created on the passage from phase to phase.
In other words, these processes fight the very nature of modern software development based on a false assumption: change can be managed.
In addition, here is no evidence that software processes are inherently stable. More likely, it is these processes complexity and unstable dynamics that create bug spirals, cascading missed deadlines and exponential cost of change curves.
The spiral software development model, developed at Xerox PARC in the 1980s assumes that feedback and control must continually refine all aspects of the process.
Agile methodologies apply the spiral model to build in dynamic mechanisms to leverage the agility of an unstable process. However, like the jet fighter, this agility comes with a cost: non-linear or even chaotic response to input.
Agile methodologies must use accurate metrics and feedback to continually adjust the course of the project to keep it from spiraling out of control.
For example, extreme programming (XP) uses continual feedback with metrics that assess the creation of customer value as measured in ideal engineering hours. However, this is metric only measures forward progress.
XP uses unit tests and task estimates vs. real programming time to ensure that developers are following the proper course. Immediate customer feedback on completed features allows instant evaluation of the completeness of requirements and automation of the tests keeps the course moving forward as there will be no back tracking.
However, if any of these metrics are inaccurate or ignored, there is a possibility that the entire project will spiral out of control. Ironically, agile methodologies require systematic and even pedantic application of metrics to adapt to change without spiraling out of control.
Much has been written about the difference between agile and 'plan-based' software methodologies. Most of these articles are no more than thinly veiled advocacy for a particular development process without examining the first principles of design. This article discussed software process by applying the fundamental principles of control theory and revealed the fundamental weakness and strength of each approach.
Taylorist approaches assume and enforce the stability of the process. This stability allows Gant chart style predictability but mandates strict control of change throughout the process. Agile methodologies allow organizations to meet changing needs and requirements need accurate, instantaneous metrics with feedback and control.
Either process can be successful if these assumptions are met. However, in practice most organizations do not have the luxury of controlling change as reality has a way of changing the rules, the game, the goal posts and even the playing field.
Most important, modern management techniques, such as Bayesian decision planning, place the emphasis on agile business models that are proactive in changing markets and responding to customers needs before they need them. Leveraging the newest and most powerful technologies quickly and effectively can make or break an organization.
Unless the selected process embraces agility and is designed to handle and manage the dynamic stability of the process, agile organizations are doomed to failure. This is the lesson of the auto industry: process must support the corporate strategy and vision.
Cost-effective agility that leverages the correct technologies to not only meet but anticipate and consistently exceed the expectations of customers is essential. It is the duty of management to ensure that the selected process supports rather than sabotages these goals.
Carmine Magione is chief architect/technologist for Bridge Medical, a developer of patient safety software for the medical industry.