To Build or Buy? That is No Longer the Question - Part II

By Daniel Gingras

(Back to article)

There has been much study of the COTS selection methodology over the past couple of years, much of it done by Carnegie Mellon’s Software Engineering Institute, which previously had focused almost exclusively on software development methodologies, culminating with the seminal work Capability Maturity Model (CMM).

CMM defined an organizations sophistication level in the world of software development. The body of work, known as SA-CMM, or Software Acquisition-Capability Maturity Model, defines the level of sophistication of the organization in it’s ability to select software for commercial purposes. It ranks organization into one of five levels with one being the lowest initial level, which it defines as:

More Dan Gingras on CIO Update

Is the CIO Obsolete?

The ERP Renaissance

Surviving as the First CIO

The Death of the Internet?

If you want to comment on these or any other articles you see on CIO Update, we'd like to hear from you in our IT Management Forum. Thanks for reading.

- Allen Bernard, Managing Editor.

FREE IT Management Newsletters

"At the initial level, the project team typically does not provide a stable environment for acquiring products. The project team is staffed based on the availability of the individuals, resulting in a random composition of the acquisition skills. Acquisition does not receive adequate management visibility. The project is pursued in an ad-hoc manner. There are no key processes areas at Level 1."

This is the level at which most organizations acquire software which can have profound implications to their organizations. It also may be why studies have shown that 60% of all software implementation projects fail.

Step 1- The business case

The first step in formalizing the process is to create the business case document necessary to justify the project. It should clearly spell out the reasons for the projects, the anticipated steps necessary to accomplish the project and a preliminary budget to accomplish the project.

The key here is to create the financial or other justification for the project and to clearly spell out the benefits of what you are trying to accomplish. I like to tell both clients and students this is like navigating a boat on a long voyage; you may not know the exact route yet, but you should absolutely know the destination, or you may end up in a really bad place.

I wouldn’t even consider approving a project which didn’t have a well documented business case with specific and quantifiable benefits defined. This is the root of the entire project, and everything else grows from a well defined business case.

The key is the quantifiability of the benefits: You should know what success looks like before you start or you will not know if you succeeded.

It may be as definable as ROI, or somewhat less measurable, such as customer satisfaction, but in that case, you should have a good baseline against which you can measure. The bottom line is you wouldn’t think of buying a new factory machine without a thorough ROI analysis, so you shouldn’t invest in large information technology projects without the same rigor.

Step 2 - Governance and Control

Once you have the business case defined, you need to put in place a mechanism to keep the project on course. The course may not have been defined yet, (i.e., you may not have a solution or even an approach to the solution, but you really need to insure that the voyage has a captain from the business not IT).

Major projects like ERP or CRM really need a more formal oversight structure in the form of a specific “steering committee” or other governance body, which should represent both the business functions affected and have some participation from IT to insure there is an overall consistence with a corporate architectural plan.

In general, however, this body should be run by the business functions and not by IT to insure that true business benefits are achieved.

Step 3 - Constraints

Any capital expenditure has a pre-defined set of constraints, and it’s best to get these out on the table immediately. These include budget constraints — you want to set a fence around the amount of money you have to spend on the project early before you waste a lot of time and effort.

You may also be constrained by time — this is particularly true in the era of Sarbanes Oxley where new financial systems need to be implemented in a way that will not affect year end closes. You may also have timing constraints in manufacturing systems depending on your specific business environment.

Unfortunately, you may also have specific technology constraints too. This is unfortunate because in the “perfect world” you would select a COTS based purely on business needs and benefits but, in the real world, you generally have sunk-cost in specific computer architecture, and this translates to a specific skill base which can significantly affect costs of the project.

If you are completely an IBM AS/400 environment, for example, then you are adding significant cost and risk by looking at Linux solutions. The trade-offs may be worth it, but you need to factor them into your analysis early.

Step 4 - Requirements Definition

This is the most “technical” part of the analysis, and depending on your specific project, it may be of some benefit to bring in some help with this process. This is particularly true for big systems like ERP or CRM, where you may not know all the features available in the marketplace.

In general, this is the area that consumes the most time and resources in the selection process, and it’s also the most critical. The scale of the project will determine the nature of this step but a large scale, complicated ERP selection can take the better part of a year to do correctly.

The process varies considerably depending on the size of the project, but, in general, it involves fully understanding and mapping the affected processes within the business. Luckily, there has been a lot of new work done in this area over the past decade, and there are a number of new modeling techniques and tools available to help with this step.

Step 5 - Market Intelligence and Survey

This is really the easiest part of the process, and it can be very straightforward or moderately complex, depending on your approach.

The simple approach is to rank order the vendors of the specific product by market share. There is little risk in selecting the top three products in a market and limiting your selection to those products. This reduces risk because the “institutional knowledge” of all of the clients in the marketplace are generally embodied in these products.

These products will be coming from vendors who are also more financially stable and better able to extend these products to adapt to the ever changing market conditions we face today. Unfortunately, they also tend to be the most expensive, but not always.

You should be very careful about this step though because there are a lot of vendors with old “legacy” products, which have lots of installs, but are basically old and tired, which leads many companies selecting very expensive or complex products to seek out help with this process.

There are a number of good research firms with deep knowledge of the marketplace and, although this does complicate this part of the process, it does significantly reduce the risk.

There are good new products coming out every day, but I don’t like to be at the cutting edge for most processes, I’d rather have a lower risk proven product with a commitment from the vendor to continue development and expand the capabilities of the product.

Step 6 - Selection Parameters

Once you’ve completed your models and have a good understanding of the requirements, you can draft a request for proposal (RFP) based on those requirements and send it to the vendors you’ve identified in Step 5.

This is a straightforward exercise much like the selection of many capital goods. You will then rate the requirements in terms of their importance to your particular situation, so you can “weight-average” the answers from the vendors.

Vendors should answer each requirement with one of four answers:

  • This feature is currently available in the product;
  • This feature is scheduled for a future release on such and such a date;
  • This feature is available with a third party add-on; or
  • This feature is not available.
  • Clearly you are shooting for answer No.1, but knowing that a particular feature will be available in a future release may also meet your requirements — depending on your implementation schedule. Or that a third-party vendor can provide the functionality, while less desirable, may also meet your needs.

    Once you’ve rated the packages against the requirements, you will want to pare it down to a very short list, perhaps only two vendors and begin site visits and reference checks. In general, the vendors will try to steer you to their “pre-qualified” references, but I urge you to spend some time on Google searching for other users of the software and making some calls on your own.

    Ask specifically what they use the software for, how difficult it was to implement, the quality of the support they’ve been receiving, and focus on the problems or limitations of the software. Even if the specific problems don’t seem to affect you, you may want to keep them in your back pocket for negotiating leverage.

    Insure you are speaking about the save version of the software you’re going to purchase, because the problems someone might tell you about may have been fixed in subsequent releases. If possible, visit sites not selected by the vendor, but ones you’ve found yourself.

    Site visits are generally a nuisance for most companies, so try to be as unobtrusive as possible, and remember how valuable site visits are when someone asks you.

    Step 7 - Implementation Considerations

    Selecting a package is just the first step in longer process that involves actually getting the benefits you’ve identified. So, you’re going to have to get the package running, and that means the “I” word: Implementation.

    While this article discusses the steps necessary to select a product, it really represents 40% or less of the effort involved in obtaining the benefit — 60-80% of achieving the benefits defined in the business case depend on a successful implementation. And discussing how to implement a COTS package is well beyond the scope of this article.

    Most projects fail, not because the wrong software was selected, but because the implementation was poorly managed. In large-scale projects, you will need to bring in help to implement the software. This is the realm of the “system integrator” and selecting the right SI is another secret of the successful project.

    Generally, you should ask the vendor of each system on your “short list” to recommend their Top 5 implementers. Again, this is simply the triangulation of knowledge in the marketplace, thus reducing the risk. Certainly talking to other companies who have implemented the package you’re leaning towards would also be of value.

    Step 8 - Negotiation and Contracts

    This is a straightforward process, much like the negotiation of any acquisition of capital goods. It’s a good idea to get the “standard contract” from each vendor early in the process for legal review. This can shorten the process later on, and sometimes you might find a clause in the contract which would be a deal breaker, thus eliminating one of the potential vendors early on.

    Knowing the fiscal cycle of the vendor is also a good piece of intelligence because deals at quarter end and year end, as salesmen reach to make their quota, are legend. Here again, you can bring in an expert to help you negotiate, and there are a number of consulting firms with expertise in negotiating complex software agreements.

    Step 9 - Implementation Issues and PMO Formation

    Once you have the software selected, your work has just begun. Selecting the system integrator is as important as picking the right package, particularly in big implementations, because you are going to commit to using a specific methodology, or approach to the implementation.

    A small package may actually be implemented by the vendor, but in general, we’re really discussing major packages in this article and major package are generally not implemented by the purchaser or the vendor.On a large project, consider bringing in some independent help to oversee the project, so you don’t get overwhelmed by the process, and remember you must commit not only IT resources, but business resources to get major projects done successfully.

    In general, 80% of a project team or project management office staff should come from the business and not from IT. Having IT run a major business system project is a prescription for failure. Your team should include members from the mill, finance, operations, quality and every function which will be touched by the project.

    Step 10 - Training

    A major area where problems can develop is in training for a new system. You should prepare for three waves of training: The pre-implementation phase, which is generally to give an overview of the systems and create a “buzz” within the organization; the implementation phase, where the meat of the learning the system will be transferred to the organization; and a post implementation phase, where the training will have real value because end users will be able to relate to specifics they’ve seen while using the software in the business.

    Step 11 - Post Implementation Analysis

    Once you’ve completed the project, take time to go back and review the process. Having an independent audit of both the process and the benefits which were the original drivers is key to a process of continuous improvement in software selection, and reduces the risk in future projects.


    Obviously, you won’t follow this methodology every time (some application selections are relatively trivial) but if you want to reduce the amount of “shelfware," following this process has real value. On a major multi-million dollar project, you will certainly want to bring in outside help to insure that you get the most out of your selection, but the world now revolves around packaged software and the proper selection of software and systems is essential to maintaining a competitive advantage.

    Daniel Gingras has been CIO of five major companies and is a partner at Tatum, LLC , a nationwide professional services organization of senior-level technology and financial executives who take on leadership roles for client companies.

    He has more than 30 years of IT experience and teaches computer science at Boston University. He can be reached at dan.gingras@tatum.com.