Don’t ship broken software

There are two kinds of organizations: Those that ship faulty software, and those that don’t. Unfortunately, trying to change from one that does ship faulty software to one that does not is nearly impossible—in fact, I’ll go so far as to say it doesn’t happen to any significant degree. Yet at the same time, organizations that ship defects come under greater and greater pressure to stop doing so.

Why is it that changing into a “defect free” organization is so difficult?

The root cause of the problem is defect accumulation: The fact that shipping defects becomes a never-ending cycle that creates more defects than an organization can sensibly resolve. The acquiescence to ship defects tends to be ingrained into an organization and, consequently, becomes a way of life during early stages of a system’s life cycle. As a product begins to gain customers the high cost of defective software in turn becomes apparent—in the unhappy customers, high customer support costs, excessive software maintenance costs, and a host of internal conflicts.

Closely related to defect accumulation is the nature of the majority of defects: They are deep-seeded, rooted into the architecture of a system. In many cases, the defects have been “baked in” from the early days of a project’s life cycle. This essentially means that the defects are now regarded as behaviors within the system. They directly affect the system’s characteristics. Changing these deep-seeded defects can often cause a cascade effect throughout the system, requiring changes in outlying components that have come to rely on defective behaviors. The result of fixing a known defect therefore causes several other defects to arise, leading to a vicious circle that becomes increasingly difficult to break out of.

This leads to the most dramatic and most difficult aspect of large-scale defect elimination: The problem has a tendency to be a logarithmic one. As systems become larger, more defects are “baked in” to the core architecture. Defects become an integral part of the system, causing inter-depencies upon other defects. This cascades throughout the system, creating an increasingly complex web of defect-relationships.

Most software efforts suffering from this problem tend to demonstrate a number of signature worst-practices, including poorly thought out project staffing, inadequate project and architectural support, no formal inspection process, lack of documentation, non-modular code development, minimal code reuse, high interdependency between software modules, and duplication of algorithms. This “spaghetti code” leaves the disabled organization with a fragile, high-maintenance system: Every change tends to cascade into undesirable side effects.

Ultimately, an organization in this position finds that it cannot overcome the weight of heavily defect ridden software. The true cost of developing and releasing defective software becomes clear: The software is a dead-end. Undesirable side effects appear with every attempt to fix the product. The cost of applying a simple change rapidly turns into a major development effort with extensive integration testing and release testing—and often the end result demonstrates even more defects than the previous release.

The final result: Complete re-architecture and development of a “second generation” becomes the only way out. Unfortunately, this is often perceived as too high a cost. Software projects are written off as a failure, or become stagnant as organizations try to milk them for everything they can. Facing what seems to be starting over is a traumatic event.

Fortunately, lessons learned can translate into a solution—provided the organization can reinvent itself as “defect free.”

It’s not easy. In fact, I’ve seen only a few organizations do it successfully.

Organizations that commit to the change from the top-down are most successful in making the change. Without a doubt the greatest motivator for making the transition to a “defect free organization” comes down to cost and, fortunately, this is a motivator that is tangible enough to act on. Understanding the value of becoming defect free requires studying the problem and accepting the benefits in terms of time-to-market, reduced resource costs and reduced maintenance costs. Because of the clear benefits in developing defect free software in the first place, this is often not a difficult decision to make (particularly with the benefit of hindsight and direct experience with the hidden costs of defect ridden projects). Unfortunately, in the case of legacy software that is already highly defective organizations find themselves in a precarious place. To adopt change the high cost of correcting all the existing defects—or of flat our re-architecture—can be staggering: 50% of development cost and as much as another 100% of maintenance cost.

In fact, $100 spent to find and fix a defect during the requirements phase corresponds to $10,000 to find and fix the same defect during the operations phase, after software is in production (based on Boehm, 1981).

While cost is the most startlingly obvious justification for adopting a defect-free methodology there are many others. Efficiency of execution from better architecture and better use of resources translates into a more agile organization. Fundamentally more modular, more robust software translates into greater flexibility. Resources that would otherwise be squandered on the high cost of maintenance and support can be deployed to affect positive improvements to a project, such as new features or greater responsiveness to user feedback.

A zero-defect focus implies improvements in quality standards and tends to demonstrate far greater modularization and component reuse. This often translates into greater reuse of software modules externally, making positive contributions to other software projects as well. The organization as a whole benefits from this.

Fundamentally, the most important step in moving toward a zero-defect organization is empowering the Software Quality Assurance and Improvement team. In today’s modern, start-up driven world this often means creating such a group, as quality assurance is often treated as an afterthought—one of the root causes of highly defect-ridden projects. Quality assurance needs to be given consideration from the very inception of any project. Poor software quality always follows any organization that does not place an emphasis on developing quality from day one.

A fully enabled quality assurance organization contributes to a project during requirements gathering and specification phases. Quality assurance is not merely “testing,” it is a far more comprehensive process that pervades the software development life cycle. This process of improving the quality of work product is particularly crucial early in a project, during requirements gathering and design. It sets the foundation of the overall project. In other words, it becomes the job of the quality assurance team to make sure that requirements and system designs provide good blueprints for the development of the project. A full explanation of the mission and scope of the quality assurance team is beyond this article, but I’ll discuss a few of the key areas of quality assurance and improvement that drive defect-free software.

The essence of software quality assurance is preventing problems from occurring, removing defects and contributing to the usability and maintenance of the software. Essential to these goals is developing an integrated, comprehensive quality program. This effort begins at the inception of a project’s life cycle and continues through the post-operational maintenance phase.

This means that the quality assurance group will participate in efforts to define the project scope and requirements. Involvement at this stage assures that the goals of the project—and thus, the tasks of development—will be clear and concise. Are the requirements clear and understood? Does the customer agree with the intepretation of the requirements? It is the quality assurance groups job to step in and say, “Hold it, things are not right!”

The group should also assure that the development organization has a suitable development process and defect management process defined and in-place early in the project. This means applying quality assurance principles to the software development plan, resource allocation and staffing plan and mechanisms to review progress and assess status.

Also essential to the quality assurance effort—although often driven by a different team—are formal inspections. Formal inspection refers to a structured process of trying to find defects in development documents, programming code, specifications, designs and others artifacts during various phases of the software development program. Sometimes called a “Fagan inspection,” after Michael Fagan who is credited with being the inventor of formal software inspections, the process has a dramatic effect in reducing defects early in the project life cycle. This of course translates into dramatic project cost reduction.

Inspections comparison, Fagan, 1986
Inspections Comparison, Fagan, 1986

Studies related to formal inspections give us a number of well-documented findings regarding the inspection process. Projects that do not use inspections delay defect discovery until late-stage testing phases and the operations phase. Also, projects that avoid formal inspections consume significantly more resources. This may not appear intuitively at the start of a project, since zero-defect projects tend to require more resources early on. This gives the illusion that attaining zero-defects requires dramatically higher staffing throughout the project—in fact, just the opposite is true. Because of the gains realized early in the project life cycle, resource requirements are considerably reduced during later phases.

By adopting a formal inspection process defects are discovered and eliminated early in the project life cycle, dramatically reducing costs:

  • With no inspections:
  • 50 defects found at testing stage
  • 10 defects found after the project ships, at operations phase
  • Average industry cost: $700,000
  • Using logic and code inspections:
  • 35 defects found in logic and code stage
  • 25 found at testing stage
  • 5 found during operations
  • Average industry cost: $525,000
  • Using requirements, design, logic and code inspections:
  • 16 defects found in requirements and design stage
  • 30 found in logic and code stage
  • 13 found in test stage
  • 1 found in operations stage
  • Average industry cost: $300,000

Clearly, the most dramatic improvement comes from adopting a formal inspection process as early as possible in the project. These findings have been born out at companies such as American Express (13,000 LOC and 0.3 operational defects/KLOC), Aetna Life and Casualty (4,439 LOC and 0 operational defects with a 25% reduction in development resources) and BAE Systems (before inspections, experiencing 50% defect detection during operations; after implementing inspections, 75% of defects are detected prior to testing and integration).

The investment in formal inspections is surprisingly light given the dramatic effect it generates. The inspection itself is limited to a two hour session at most. Preparation time should take no more than three hours per inspector. Rework and follow-up on an inspection should require no more than five to eight hours.

Documentation is also a clear area in which the quality assurance group should be involved. Poor documentation is a frequent source of technical shortfall within a project. By improving documentation standards software quality also improves: The documentation process helps focus on clean interfaces and reusable code. Well documented modules become components that can be distributed throughout a system—thus eliminating duplication of functionality, a major source of software defects.

Also critical to the zero-defect program is the concept of “whole teams.” This principle is an essential component of the Rational Unified Process development in the 1980’s by Rational Software (now a part of IBM).

The core concept behind whole teams is simply this: Without representation from every stakeholder in a project, the project will be incomplete. The corollary is that an incomplete project will by nature be a defective project. Whole teams combat this incompleteness by assuring that every party concerned with the project is involved throughout the project’s development: Business stakeholders, marketing teams, engineering, quality assurance, user sponsors as well as target product customers, even budgetary management should be involved. Whether these stakeholders are involved in a steering committee or participate less formally is dependent on the nature of the project environment. The most critical point is that once the stakeholders are identified, they must remain involved throughout the project through regular interaction as a whole team, working together.

Changing an organization from one that enables defect-ridden software into a zero-defect organization is a fundamental shift. It needs to be adopted and embraced throughout the organization using a comprehensive program that attacks the source of defects and eliminates them mercilessly. It’s not an easy transition to make, but it can be done—and the rewards are well worth it.

Organizations that successfully make the transition will experience dramatically reduced project costs, more rapid time-to-market and market response capabilities and far superior use of their own resources. But these are just the most obvious, readily tangible benefits. In time organizations will experience greater software reuse and agility, opening new doors to opportunities that would otherwise remain closed. Improved software development techniques will also lead to reducing reliance on a single, critical resource, in favor of a more collaborative and resilient development organization. New product developments and sharing of technology between working groups becomes a possibility as good documentation and software reuse principles are adopted. Overall health of the organization improves too, evidenced by morale improvements that come from a productive, forward-looking organization as opposed to an inwardly focused one bent on defect elimination. The benefits grow over time, becoming evident in both dramatic and subtle ways.

I think the evidence speaks for itself. The key indicator of the value of developing a defect-free program is simply this: No organization that has made the transition has ever gone back.

For more on this topic, look for my upcoming book Finding Your Way: Navigating the methodology maze, a roadmap to successfully adopting process in your organization, or sign up for one of my classes offered through Hyrax International LLC.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s