Pivotal Decisions, Process, Competition, and Success
Many experts now credit Japan's relentless application of just-in-time (JIT) manufacturing techniques and total quality management (TQM) or agile manufacturing to the automobile industry as opposed to the Fredrick Taylor models of lean manufacturing practiced in the U.S.
This shift represented a titanic change in processes, procedures and management techniques.
Currently, the software manufacturing industry finds itself in a similar situation. The hallmarks of most software projects include chronic lateness, over-extended budgets, and poor quality.
Software organizations struggle to reign in projects by imposing stricter software processes. These processes are based on waterfall or iterative models with phasist structures stemming from Taylorist theories of efficiency.
Over the past decade new software processes based on JIT manufacturing principles called agile methodologies have been introduced into the software industry.
However, there is huge reluctance in the software industry to make any adjustments even though, currently, all indicators point to something egregiously wrong.
This situation in many ways resembles the turning point of the U.S. auto industry from the '70s and early '80s.
Stability of Processes
Humans successfully control many complex systems under unpredictable and changing conditions. Airplanes dont crash into the ground even when buffeted by gusts of wind, and power stations do not melt down under fluctuation's in power draw.
These complex systems are controlled by making measurements and corrections to keep the system in balance. The study of the systematic application of measures and corrections is called 'control theory'.
The control of a system depends not only the system's current state but also how the system will respond to corrections or inputs. Control theory defines three categories of stability:
Stable: Stable systems resist change. When changes are made to a stable system, the system eventually returns to its starting state.
Neutral: Neutral systems respond to inputs in predictable ways.
Unstable or non-linear: Complex systems often respond to changes in unpredictable ways. In the real world, most complex systems are non-linear.
Many complex systems display stable characteristics when their elements are kept within certain parameters, called an "envelope of control." However, if the threshold is exceeded unpredictable results may happen.
Software Process and Control
Automobile and passenger aircraft stability are desirable features. However, there are many instances where the desire of a system to return to its stasis point is detrimental.
Fighter aircraft, for example, must be able to maneuver quickly to avoid other aircraft or incoming anti-aircraft artillery. Any attempt by the aircraft to return to its stable position, would slow its reflexes.
Traditional software engineer processes such as the waterfall model and iterative model are built on Taylor's theories of lean management. These processes are developed on two fundamental assumptions: First, that each stage of the waterfall is progressively more expensive (from specification through implementation) and second, that the process itself is dynamically stable.
These processes as a group are called "phasist."
The assumption of progressively increasing costs leads to the practice of attempting to fully define requirements and design before beginning implementation. Since changes to the requirements after implementation begins are expensive, system control becomes an exercise in managing change.
The stability assumption implies that Gant-style planning tools optimally allocate resources. Gant charts allow resources to be aligned and allocated based on estimated cost through the process of leveling.
This process depends on responding linearly to changes in resources, requirements and costs, in other words, the system must be stable or neutral.Dynamics and Software Creation
Lean management failed in the automotive industry because stable systems do not respond well to change. For example, the lead time from requirements to implementation was typically around five years. This lead time prevented American companies from using the most recent technology in automobiles.
This lead time was acceptable when change tended to be slow. However, the industry was incapable of responding to sudden changes in customer demand such as fuel economy during the energy crisis of the '70's.
Moreover, American industry was not able to leverage the latest in electronics and automation. The agile techniques of Japanese companies allowed immediate changes in both product and technology.
The pace of change is amplified a thousand-fold in the software industry. Massive change results from missed requirements, improper design decisions, changing technology, changing competitive landscape. In addition, competition appears suddenly because the barrier to entry for software is much lower than it is for manufactured goods.
Paradoxically, phasist process and metrics attempt to fit software development into a stable system by controlling the phases through the artifacts that are created on the passage from phase to phase.
In other words, these processes fight the very nature of modern software development based on a false assumption: change can be managed.
In addition, here is no evidence that software processes are inherently stable. More likely, it is these processes complexity and unstable dynamics that create bug spirals, cascading missed deadlines and exponential cost of change curves.
The spiral software development model, developed at Xerox PARC in the 1980s assumes that feedback and control must continually refine all aspects of the process.
Agile methodologies apply the spiral model to build in dynamic mechanisms to leverage the agility of an unstable process. However, like the jet fighter, this agility comes with a cost: non-linear or even chaotic response to input.
Agile methodologies must use accurate metrics and feedback to continually adjust the course of the project to keep it from spiraling out of control.
For example, extreme programming (XP) uses continual feedback with metrics that assess the creation of customer value as measured in ideal engineering hours. However, this is metric only measures forward progress.
XP uses unit tests and task estimates vs. real programming time to ensure that developers are following the proper course. Immediate customer feedback on completed features allows instant evaluation of the completeness of requirements and automation of the tests keeps the course moving forward as there will be no back tracking.
However, if any of these metrics are inaccurate or ignored, there is a possibility that the entire project will spiral out of control. Ironically, agile methodologies require systematic and even pedantic application of metrics to adapt to change without spiraling out of control.
Much has been written about the difference between agile and 'plan-based' software methodologies. Most of these articles are no more than thinly veiled advocacy for a particular development process without examining the first principles of design. This article discussed software process by applying the fundamental principles of control theory and revealed the fundamental weakness and strength of each approach.
Taylorist approaches assume and enforce the stability of the process. This stability allows Gant chart style predictability but mandates strict control of change throughout the process. Agile methodologies allow organizations to meet changing needs and requirements need accurate, instantaneous metrics with feedback and control.
Either process can be successful if these assumptions are met. However, in practice most organizations do not have the luxury of controlling change as reality has a way of changing the rules, the game, the goal posts and even the playing field.
Most important, modern management techniques, such as Bayesian decision planning, place the emphasis on agile business models that are proactive in changing markets and responding to customers needs before they need them. Leveraging the newest and most powerful technologies quickly and effectively can make or break an organization.
Unless the selected process embraces agility and is designed to handle and manage the dynamic stability of the process, agile organizations are doomed to failure. This is the lesson of the auto industry: process must support the corporate strategy and vision.
Cost-effective agility that leverages the correct technologies to not only meet but anticipate and consistently exceed the expectations of customers is essential. It is the duty of management to ensure that the selected process supports rather than sabotages these goals.
Carmine Magione is chief architect/technologist for Bridge Medical, a developer of patient safety software for the medical industry.