The price of software problems is very high: As much as 50% of development and 100% of all maintenance costs can be attributed to software defects. Often, this price becomes apparent late in the software life cycle—quite often after the software has reached its operational phase (after the software ships)—as previously undetected defects are discovered to be the root cause of incomplete function or poor reliability in the product. On top of this we see that $100 spent to find and fix a defect during the requirements phase corresponds to $10,000 to find and fix the same defect during the operations phase (based on Boehm, 1981).
It turns out that when you search for defects determines how much it costs to identify and fix those defects. The formal inspections process emphasizes discovering defects early in the software life cycle. This is reflected in the average industry costs associated with defect resolution:
- Without formal inspections:
- 50 defects found at testing stage
- 10 defects found after the project ships, at operations phase
- Average industry cost: $700,000
- Using logic and code inspections only:
- 35 defects found in logic and code stage
- 25 found at testing stage
- 5 found during operations
- Average industry cost: $525,000
- Using requirements, design, logic and code formal inspections:
- 16 defects found in requirements and design stage
- 30 found in logic and code stage
- 13 found in test stage
- 1 found in operations stage
- Average industry cost: $300,000
Attaining such drastic reduction in defect prevention costs did not become possible overnight. Looking back over the history of software development and quality assurance methodology there are clear evolutionary breaks that have fueled strides toward better, more efficient defect prevention. Early efforts in the software industry focused on process definition as a primary means to eliminate defects. By the 1970’s, efforts in process measurement and improvement, combined with formal defect detection techniques, made significant strides in software quality possible. It wasn’t until the 1980’s that defect prevention techniques—such as formal inspections and unit testing frameworks—began to evolve.
Formal Inspection refers to a structured process of trying to find defects in development documents, programming code, specifications, designs and others artifacts during various phases of the software development program. Sometimes called a “Fagan inspection,” after Michael Fagan who is credited with being the inventor of formal software inspections, the process has a dramatic effect in reducing defects early in the project life cycle. This of course translates into dramatic project cost reduction and more reliable schedule projections.
In fact, according to a JPL Executive Briefing given to NASA in the late 1980’s, NASA’s projected budget for the International Space Station software program without the use of formal inspections would top $6 billion—compared to a revised budget of $4 billion after applying the formal inspection process.
So, what are formal inspections and how can they improve your projects?
Formal inspection is a defect detection, removal and correction verification process carried out by a small group during the pre-test phases of the development life cycle. It involves the interaction of the following five components of the software life cycle:
- Well-defined inspection steps
- Well-defined roles for the participants of the inspection (moderator, reader, recorder, author, inspector)
- The software product subject to inspection
- An infrastructure that supports the formal inspections
The primary objective of formal inspections is to remove defects as early as possible in the development process. Formal inspections achieve this objective by:
- Identifying potential defects during individual preparation
- Verifying that identified defects are actual defects
- Recording the existence of defects
- Providing the code author with a list of defects to fix
- Ensuring that fixes are performed and are correct
Formal inspections were designed to help organizations involved in software projects develop better products. While its primary focus is on software quality there are a number of peripheral benefits—dramatic budget benefits being the most obvious. The overall software life cycle cost is lower since defects are found early and are easier and less expensive to fix. By introducing formal software review and testing steps early in the software development life cycle a more technically correct architecture is developed and tested earlier. This fuels improved component reuse and drives development efficiency up. The effectiveness of the testing process and closely-related software quality assurance processes are improved because testing begins early, as well. This generally means that less time needs to be devoted to testing processes on a recurring basis. Another benefit of formal inspections is the immediate feedback on the software from a group of peers and target users. This creates a support loop that integrates well with modern ideas of iterative development and creates an environment where agility and end user feedback become a target of the development cycle.
There is a significant emphasis on the “formal” part of the inspection process. Formal inspections, while designed to be effective without impacting team schedules heavily, are more rigorous and well-defined than other peer review techniques such as pair programming and walkthroughs. Because of this they are significantly more effective, but they do not take the place of milestone reviews, progress reports, status review, testing or walkthroughs. Also, the process clearly defines phases of the software life cycle at which the inspections should take place.
Inspections must be carried out by peers representing the areas of the life cycle affected by the material being inspected. The team should be limited to six or seven people at most, and everyone participating should have a vested interest in the work product. This is a particularly important requirement of the process. Also, inspections must not be used to judge the quality of the software work product or the authors that developed the product. For this reason managers should not be present in the formal inspection itself; findings should be presented to management statistically or as a group of work product findings. This will demonstrate the value of the inspection process without singling out any individual author. Using the inspection to judge the capability of authors may result in less than honest and thorough results: If coworkers feel their efforts could result in a poor performance review for the team they may be reluctant to identify defects.
The Formal Inspection Team
Every formal inspection is carried out by a team of inspectors. The inspection team consists of four to seven individuals, and no more. In fact, the team should be kept as small as possible and yet be able to complete the job at hand. The goal of keeping the team small is to keep decision making and process to a minimum. It is also important to minimize impact to the overall development life cycle—superfluous involvement ultimately means time taken away from productive work on the project.
The inspection team consists of a moderator, a reader, a recorder, the software author and other inspectors. Each team member has a specific, defined role to fulfill. In addition to each individual role, it is everyone’s job to find and report defects; all team members are considered inspectors in this regard. Where necessary, more than one author will be involved in the inspection, although it is best to keep the inspection focused to an area that involves a minimum number of authors.
The moderator’s primary role is to achieve a good inspection. This includes planning the inspection, assembling the team and managing the overall process from beginning to end. The reader’s responsibility is to guide the inspection team through the work product being inspected (keep in mind that “work product” can include specifications, designs, actual software code or functioning software). The recorders job is essentially to record every discovered defect. The author’s job is to answer questions throughout the process, present the work product as needed by the inspection team and, ultimately, to correct the defects. In addition, larger projects may require support from the project librarian in keeping track of status, statistics and overall progress of the formal inspections throughout the development life cycle.
The most important guidelines to keep in mind when creating the formal inspection team are:
- Use a fair and unbiased moderator
- Keep inspection team size reasonable (between 4 and 7 individuals)
- Select inspection team members who have a vested interest in the work product
This last point is a particularly important one. Only those individuals that care about the project will be invested in the inspection process. Team members should be close to the project; peers working in the same life cycle phase, authors of the relevant specifications or code, direct users of the work product, product quality assurance and testing personnel are all excellent candidates.
Formal inspections are an organization-wide activity. As such there are several supporting roles that are essential to assure success from the process.
Most visible is the need for support at the Project and Program Management level. This means that the appropriate manager needs to establish a schedule that allows adequate time for all stages of the inspection process. This includes monitoring the inspection process and making sure that there is sufficient inspection preparation time, that individuals are not over-burdened with tasks and that all team members understand the critical nature of the inspection process. Ensuring that all team members are properly trained in the formal inspection process will help in maintain the schedule and making sure that everyone is prepared for an inspection in the alloted time. After the formal inspections have taken place, managers must meet with the moderator and author to review open items and rework estimates. The project manager will likely review inspection summary reports for defect trends and perform defect trend analysis.
Organizations may also find a need for a Chief Moderator. The Chief Moderator will chair monthly meetings of all inspection moderators and guide the evolution of the inspection process, forms and checklists, and information gathering. The Chief Moderator may also analyze the effectiveness of inspections across participating projects, provide support in the form of guidance and answers to questions and keep current on recent information and developments in the formal inspection arena.
Stages of a Formal Inspection
The formal inspection process is broken into seven stages, although two of these are optional and can be excised from the inspection process on a case-by-case basis, at the moderator’s choice. These stages are:
- Planning—The period of time used to determine whether the product to be inspected meets the entry criteria, set the inspection schedule, plan the inspection itself, select a team of inspectors and assign respective roles, and prepare and distribute the inspection materials. This is when the moderator decides whether an overview will be necessary, as well.
- Overview—An optional stage in the inspection process. The overview provides the inspection team with background information for the inspection. This stage may not be necessary if the team is already intimately familiar with the work product being inspected.
- Preparation—A critical stage during which each member of the inspection team individually prepares for the inspection. It is crucial that individual inspectors be given adequate time to prepare, otherwise the inspection process will not be efficient and may well fail to identify defects that could otherwise be discovered. Each team member prepares for the inspection by reviewing and finding potential defects in the product being inspected before the inspection meeting. Potential defects are then discussed during the inspection meeting as a group.
- Inspection Meeting—Meeting in which team members, as a group, review the product to discover, categorize and record defects. Defects are not resolved during this meeting.
- Third Hour—Literally, a third hour to the inspection meeting (the formal inspection meeting is limited to two hours). This is an optional additional time that can be used to discuss, possibly solve or further investigate defects that have already been discovered during the Inspection Meeting.
- Rework—The period of time that the author uses to correct identified defects.
- Follow-up—A short meeting between the author and the inspection moderator used to determine if the defects found during the inspection have been corrected and to ensure that no additional defects have been introduced.
The following few sections discuss each of these stages in more detail.
The Planning Stage
Chiefly, the moderator uses the planning stage to prepare for the inspection process. This entails making sure the product is ready for inspection, selecting the inspection team and assigning team members appropriate roles, scheduling the inspection meeting and making sure meeting facilities will be available, and distributing inspection materials such as forms and background information. The inspection materials should also include detailed information on the work product being inspected, the scope of the inspection and any specific components or functionalities that will be excluded from the inspection process, and individual checklists for each inspection team member to follow.
Another key role the moderator will play is making sure that the work product being inspected is of appropriate size for the inspection. If not, the product is divided into manageable pieces and separate inspections are scheduled for each piece.
The moderator also must decide whether the inspection team members are adequately familiar with the material to be inspected or whether an overview must be held. The team should know the background material from which the product was derived and should know how the material fits into the overall system being developed.
This is an optional stage that chiefly flows out of the moderator’s work in making sure that the team is adequately familiar with the work product being inspected. If any team members do not have sufficient depth of knowledge about the product, an overview is warranted. At the overview, the author will present the rationale behind the product, its relationship to other components in the system, its current state of development (e.g.: how complete it is and how well integrated into the overall system), its function and intended use and how it was developed.
An overview should be scheduled if any of the following criteria exist:
- The inspection team is not familiar with the product
- The product is new or is being inspected for the first time
- Inspections are new to the project
- Novel techniques have been used in the product
This is the most critical stage of the formal inspection process. During the preparation process, each inspection team member individually prepares for the role in the formal inspection meeting. Each inspector will review the work product thoroughly; user interface inspections will be conducted field-by-field while code level inspections will be conducted by reviewing the work product line by line. Higher level specifications should be reviewed by reading the specifications and comparing them to system requirements and expected system behaviors. Comparison against relevant work product and best practices should be made during the preparation process, providing a basis for quality standards and acceptable behaviors. Checklists should be used in the work product review for guidance on typical types of defects to be found. All potential defects are carefully logged and will be discussed during the formal inspection meeting. Complete preparation logs are submitted to the moderator prior to the meeting.
Prior to the inspection meeting the moderator will review preparation logs and draft defect reports. This review will both ensure that the inspection team is ready for the formal inspection meeting and also highlight any areas that may require additional attention. If the logs indicate that the team is not ready, the inspection meeting should be rescheduled. Areas of common concern between inspectors can often provide valuable guidance on areas of the work product that require particular scrutiny.
The Inspection Meeting
The inspection meeting is the period during which the inspection team reviews the work product as a group. The meeting is held to a limit of two hours and is carefully moderated. The meeting is run by the moderator and usually starts with a very brief restatement of the goals of the formal inspection. If an overview has been scheduled it is held prior to the formal inspection meeting itself.
During the meeting, the reader interprets and provides a reading of the work product, the author clarifies information as needed, and the team identifies defects that are then classified and recorded by the team recorder.
The reader’s description should note the function of components being review (whether paragraphs of text, code blocks or a functioning user interface) and their relationship to the product and higher level documents. The moderator is tasked with keeping the meeting on track, but the reader can be interrupted at any point a potential defect enters into the discussion. A short discussion of the defect is permissible, but the moderator should limit time accordingly. Any issue that cannot be resolved within the time limit will be marked as an open issue and can be revisited during the third hour.
The team should reach consensus on whether a potential defect is, in fact, a real defect. Quite often what appears to be a defect could be a mistake on the part of the inspector. At the same time it could be a flaw or deficiency in the requirements or other specifications (in fact it is very common to discover edge cases that were never considered during the specification and design process). These are considered defects as well and must be logged as such, even though resolution steps outside the bounds of the author’s work. During this process the recorder will note down every defect, its reproduction criteria or location as appropriate, and classify the defect. Ideally, if defect tracking software is readily available and defect entry is efficient enough that it will not slow down the inspection meeting, direct entry into the system is favorable. This provides the team with an opportunity to review the defect as it is recorded, avoiding possible interpretation or notation mistakes.
If the work product cannot be fully inspected within the two hour limit, to avoid fatigue and missing defects, the moderator should conclude the meeting and schedule a follow-on inspection at a later time.
The moderator will conclude the meeting with a brief recap of the number of defect found and the severity of the defects. All information should be recorded to track the effectiveness of the formal inspection process and to look to defect trends at a later date. If a third hour is needed, the moderator will assign action items to individual inspectors at this time.
The third hour is used for discussion or to close open issues—it should not be an extension of the formal inspection meeting. The third hour does not need to take place immediately after the formal inspection meeting either, nor does it need to be limited to one hour: It should be scheduled at a time when the author is prepared to discuss corrections to discovered defects or if major issues, such as defects or omissions in the higher level specifications, have been discovered. Attendees to the third hour are free to obtain support from outside sources or invite third parties, including managers or technical experts, to the third hour purely for technical reasons.
The rework period is used by the author to correct discovered defects, or to coordinate the correction of defects that fall outside the author’s domain (for example, if the defects occur in a higher level specification document). The author is responsible for ensuring that all defects have been corrected before the follow-up period with the inspection moderator.
This is an optional stage; the moderator and the team will jointly decide whether a reinspection is warranted based the inspection meeting findings. Reinspection may be required when there are a large number of defects in the product or when one or more defects require extensive or complicated corrections. Reinspection allows the changes to the product to be reviewed by the entire team instead of just the moderator.
The goal of the follow-up meeting is to confirm closure on major defects that were discovered during the formal inspection and that no secondary defects have arisen during the process. The meeting is held between the moderator and the author. The author reviews the steps taken to correct the defects and the moderator ensures that all open issues have been addressed. Minor defects do not necessarily need to be addressed in the follow-up meeting. Additional reviewers can be present during the follow-up meeting if either the moderator or author feel it will benefit the process.
Once all open defects have been corrected and any other open issues have been resolved, and once the work product itself passes the exit criteria (this could be reaching a compile state or passing quality assurance testing, depending on the nature of the work product) the moderator marks the work product as having “passed” the formal inspection. The moderator will note the product as having passed the inspection on the formal inspection summary report.
The scheduling of formal inspections throughout the software development life cycle targets specific phases to optimally eliminate defects as early as possible.
Traditional defect discovery and removal techniques target defects significantly later and less frequently than the formal inspection process. Expectations with traditional methods are correspondingly limited: Experience is that 60 defects escape pre-testing phases for every 1,000 lines of code written. Traditional testing steps are each approximately 50% efficient in the identification and removal of defects. Provided a typical, traditional quality assurance method that emphasizes testing of a coded and working product, followed by repeated late-stage testing, an average of 3.75 defects per thousand lines of code reaches the operational phase—and ends up in customer’s hands. The following figure demonstrates the effect of traditional testing steps on the development process:
While traditional methods tend to focus on reviewing and testing completed software, formal inspections begins the process much earlier at the requirements, specification and coding phases. In comparison to traditional defect discovery and removal this is much more effective at identifying defects early and correcting them before they become a problem. The formal inspection process achieves this by:
- Inserting inspections into the pre-test phase of the software life cycle
- The strategy is to find and fix defects when and where they are injected
- Inspections mean more defect detection steps: At least 9 as compared to 4 traditional steps
The frequent and repeated inspection process also keeps defects counts manageable, as shown in the following figure:
The formal inspection process introduces inspections as early as possible into the software development life cycle, and continues to repeat the inspection process at key stages of the life cycle. This means introducing formal inspections after the program functional design, requirements, architectural design, detailed design and software coding stages. Iterative projects can take advantage of additional inspection steps after operational code delivery as well. Note that source code level inspections should take place before unit testing, as the corrections made during unit testing can hide larger scale defects that should be exposed.
There are two striking differences realized between the formal inspection process and traditional detection process: First, the average defect rate in operational systems drops from 3.75 defects per thousand lines of code to 1.05 per thousand delivered source instructions, about one fourth the defect level. Second, because defects are identifying and corrected early in the process total defect counts remain much lower throughout the software development life cycle. This has a positive, cascading effect throughout development: Fewer defects means fewer negative behaviors become “coded in” to interfaces—and that means codependent defects are also kept to a minimum.
This implies that formal inspections are about 7.4 times more productive than formal testing practices alone (based on 3.25 errors per unit effort with inspections compared to 0.44 errors per unit effort with testing, given that inspections yield a lower return to effort against testing).
Planning the Inspection
Inspections should not be cumbersome. In fact, the formal inspection process is designed to be quick, efficient and require surprisingly little preparation time. Even so, it is important the the inspection team be well trained in the inspection process and its benefits—and fully realize the relative effort that can be saved through formal inspections.
The inspection moderator will invest more time into the process than other team members, chiefly because of the advanced planning and preparation that goes into each inspection. Likewise, the author is likely to invest more time because of the rework phase—however, if we discount the actual defect correction activities, the author’s investment in the inspection is largely in line with the rest of the team: Only a matter of a few hours. Following are the approximate time estimates recommended in a single formal inspection:
- Planning (moderator): 2 to 4 hours
- Overview: Between 30 minutes and 1.5 hours
- Preparation: 3 hours per inspector
- Inspection: 2 hours maximum
- Rework (author): Up to 5 hours
- Follow-up: 2 to 3 hours
Keeping the inspection process tightly focused on the work product and on defect discovery will help keep the inspection on-track and minimize time taken away from other tasks.
It is crucial that the inspection team be fully trained in the formal inspection process. The moderator and project manager must also make sure that there is adequate time available for the inspection, and that the team has properly prepared. If there is less than five hours available for preparation time, the inspection should be rescheduled.
Formal inspections have proven themselves through the test of time. We have an extensive body of work that demonstrates their value in effort, time and cost savings across many projects. Reports produced by organizations such as the IBM Federal Systems Division (1986) show that the formal inspection process can find between 75% and 90% of all defects at early phases of the software development life cycle. And, we can look forward to a commensurate reduction in maintenance costs: As much as 90% of corrective maintenance is eliminated (Ackerman, 1989).
In fact, formal inspections are the single most important quality improvement technique a development can make, according to experts at IBM, Bell Labs, the Software Engineering Institute and other sources.
As well as these very tangible benefits, a number of less tangible benefits come from the inspection process. Frequent inspections aid in project tracking accuracy and reduce the effort involved in traditional project management. The frequent deep-dives into the system also works to foster better team communication and collaboration on the work product. This, combined with the iterative cycle of inspections works against repeated mistakes, fueling better software quality and team development.
All of these benefits conspire to produce a better product, at a lower cost and within a shorter time than would otherwise be the case.