Peer Reviews

Draft 2 Encyclopedia of Software Engineering

Peer Reviews
Encyclopedia of Software Engineering
Topic Area: Quality

Don O’Neill
Independent Consultant

• Software Inspections and Walkthroughs Distinguished
  
• Software Walkthrough
  
• Software Inspection
  

• Structured Review Process
  
• System of Checklists
  
• Defined Roles of Participants
  
• Forms and Reports
  

• National Software Quality Experiment
  
• Common Problems Revealed
  
• Inspection Lab Operations
  
• Defect Type Ranking
  
• Return on Investment
  

Keywords

Defect density
Defect detection
Defect leakage
Defect prevention
Defined roles
Forms and reports
Measurements
Product checklists
Return on investment
Software inspections
Software walkthroughs
Standard of excellence
Structured Review Process

Abstract

Peer Reviews are considered a best industry practice for detecting software defects early and learning about software artifacts. Peer Reviews are composed of software walkthroughs and software inspections and are integral to software product engineering activities. A collection of coordinated knowledge, skills, and behaviors facilitates the best possible practice of Peer Reviews.

The elements of Peer Reviews include the structured review process, standard of excellence product checklists, defined roles of participants, and the forms and reports. Software inspections are the most rigorous form of Peer Reviews and fully utilize these elements in detecting defects. Software walkthroughs draw selectively upon the elements in assisting the producer to obtain the deepest understanding of an artifact and reaching a consensus among participants.

Measured results reveal that Peer Reviews produce an attractive return on investment obtained through accelerated learning and early defect detection. For best results, Peer Reviews are rolled out within an organization through a defined program of preparing a policy and procedure, training practitioners and managers, defining measurements and populating a database structure, and sustaining the roll out infrastructure.

Origin and Evolution

Peer Reviews began as formal software inspections and now include the less formal software walkthroughs. They provide value by improving reliability, availability, and maintainability [SEI 97].

IBM Corporation originated and adopted software inspections in the early-1970’s and recognized Michael Fagan with an Outstanding Contribution Award for his pioneering work [Fagan 76, 87]. A member of the IBM group, Ron Radice said, “I worked with Fagan in 1972 and actually moderated the first inspection” [SPIN 96]. Software inspections are known to add economic value in detecting and correcting defects early at greatly reduced cost [McGibbon 96]. IBM reported an 83 percent defect detection rate resulting from software inspections practice; AT&T Corp., 92 percent [SEI 89].

Gerald Weinberg and Daniel Freedman gave real thought to the dynamics of the software inspection role players providing deep and interesting insights useful to practitioners [Freedman 1990]. Robert Ebenau provided leadership in the roll out of software inspections at AT&T Corp. and documented his knowledge [Ebenau 94] as did Tom Gilb and Dorothy Graham [Gilb 93].

The Software Engineering Institute (SEI) identified software inspections as an industry practice essential to managing the software process [Humphrey 89] and offered practitioner training [SEI 88, 89]. Peer Reviews are included in the SEI Capability Maturity Model (CMM) for Software as a level 3 key process area [Paulk 95]. The ongoing Capability Maturity Model Integration (CMMI) project spanning software, systems engineering, and integrated product development includes Peer Reviews in its Product Verification process area.

Through December 1999, the SEI CMM assessments revealed that eight percent of level 1, twenty-three percent of level 2, and one hundred percent of level 3 organizations fully satisfied the Peer Reviews key process area criteria [SEI 00].

Context of Use

Peer Reviews are considered a best industry practice for use on software projects. Senior managers consistently ranked Peer Reviews as the significant enabler of software product quality among all key process areas [Johnson/Brodman 96]. These reviews are integral to the software product engineering life cycle activities associated with software requirements and specifications, designs and code, and test plans and procedures [Paulk 95]. The best practices for software management and engineering on the project and the context of use for Peer Reviews are shown in figure 1 Best Software Practices.

Figure 1 Best Software Practices

Scope

Peer Reviews are composed of software walkthroughs and software inspections [Humphrey 89, 95]. Software inspections are the most rigorous form of Peer Reviews. Both software walkthroughs and software inspections are composed of a collection of coordinated knowledge, skills, and behaviors associated with process, standards, roles, and measurement. Peer Reviews are conducted as an integral part of each life cycle activity. See figure 2 Life Cycle Activity and Peer Reviews.

Figure 2 Life Cycle Activity and Peer Reviews



Figure 3 Peer Reviews Scope

Software Inspections and Walkthroughs Distinguished
Peer Reviews are a group activity organized to systematically reason about a software artifact. There are two types of Peer Reviews: software walkthroughs and software inspections. Each of these serves different purposes. Software walkthroughs are an informal review used to confirm the understanding of the producer and validate the approach being taken. Software inspections are a formal review used to verify that the artifact complies with the standard of excellence. In a life cycle activity, the software inspection is the exit criteria or gate that concludes the activity. See figure 3 Peer Reviews Scope.

Software Walkthrough
The software walkthrough is organized to serve the needs of the producer or author of the software artifact in acquiring superior knowledge of all aspects of the software artifact. It is a learning experience. A desirable side effect of the software walkthrough is the forging of a shared vision among the reviewers and consensus among participants on the approaches taken, product and engineering practices applied, completeness and correctness of capabilities and features, and rules of construction for the domain product. Since the software walkthrough caters to the needs of the author, it is the author who initiates the session. Consequently there may be several walkthroughs in each life cycle activity. Software walkthroughs yield open issues and action items. While these issues and action items may be tracked to closure, the only measurement taken is a count of the software walkthroughs held.

Software Inspection
The software inspection is structured to serve the needs of quality management in verifying that the software artifact complies with the standard of excellence for software engineering artifacts. The focus is one of verification, on doing the job right. The software inspection is a formal review held at the conclusion of a life cycle activity and serves as a quality gate with an exit criteria for moving on to subsequent activities.

The software inspection utilizes a structured review process of planning, preparation, entry criteria, conduct, exit criteria, report out, and followup. It ensures that a close and strict examination of the product artifact is conducted according to the standard of excellence criteria spanning completeness, correctness, style, rules of construction, and multiple views. This close and strict examination results in the early detection of defects. The software inspection is led by a moderator and assisted by other role players including recorder, reviewer, reader, and producer. The software inspection is initiated as an exit criteria for each activity in the life cycle. Product and process measurements are recorded during the software inspection session and recorded on specially formatted forms and reports. These issues and defects are tracked to closure.

Elements

The Peer Reviews process is made up of several elements: the structured review process, defined roles of participants, system of checklists, and forms and reports See figure 4 Peer Reviews- Entry, Task, Verification, Exit and figure 5 Elements of Peer Reviews.

Figure 4 Peer Reviews- Entry, Task, Verification, Exit

Figure 5 Elements of Peer Reviews

1. A structured review process is a systematic procedure integrated with the activities of the life cycle model. The process is composed of planning, preparation, entry criteria, conduct, exit criteria, reporting, and followup [Fagan 76], [Gilb 93].
2. A system of checklists governs each step in the structured review process and the review of the product itself, objective by objective. Process checklists are used as a guide for each activity of the structured review process. Product checklists house the strongly preferred indicators that set the standard of excellence for the organization’s software products [O’Neill 88].
3. The role of each participant in the structured review process is defined. The roles include the moderator, producer, reader, reviewer, recorder, manager, and consumer. Each role is characterized by particular skills and behaviors [Freedman/Weinberg 90].
4. Forms and reports provide uniformity in recording issues at all software inspections, reporting the results to management, and building a data base useful in process management. Data collection utilizes three recording instruments: Inspection Record, Inspection Reporting Form, and Report Summary Form. The results of software walkthroughs are recorded as open issues and action items [Ebenau 94a].

Structured Review Process

The activities of the structured review process are organized for software inspections. Software walkthroughs may employ variations for planning, conduct, and followup.

Planning
The structured review process begins early in the project when the manager plans for software inspections of requirements, specifications, designs, code, and test procedures. The schedule for each software inspection is recorded in the project’s Software Development Plan [SDP] or project plan. A trained moderator is assigned to each software inspection, and moderator training is scheduled as necessary. The Software Quality Assurance (SQA) Plan also discusses the use of software walkthroughs and software inspections in terms of their contribution to product verification and validation.

Preparation
Preparation is initiated by the moderator a week before the inspection session. The readiness of the product for inspection is assessed by the moderator and the producer. The moderator obtains the reviewers, recorder, and reader and briefs them on their roles along with the key principles of software inspections.

In assessing product volatility, the moderator ascertains the status of the baseline change activity and the completion of the preceding life cycle activity. The producer conducts a brief overview of the product to be inspected to assist the inspection team in their preparation for the inspection session.

The inspection materials are distributed to team members, the time and place for the inspection session are announced, and reviewers are encouraged to prepare individually for the inspection session. Individual preparation is the mother’s milk of the software inspections process.

Entry Criteria
Entry criteria are checked by the moderator at the start of the inspection session. Before the conduct activity begins, the moderator determines that the software product is ready to be inspected and the inspection team is prepared to inspect it. The moderator again assesses the product volatility indicators. Inspection team members are asked for their preparation effort, and the recorder notes this information. Where the entry criteria are not satisfactorily met, the moderator may reschedule the inspection session.

The moderator script for directing the entry criteria includes:
1. Has the preceding life cycle activity been concluded?
2. Are there any changes to the baseline?
3. Are review participants in place and briefed?
4. Have all participants received all the review materials and checklists?
5. How many minutes of preparation did each participant perform?

Conduct
The inspection session including entry, conduct, and exit is directed by the moderator and attended by the producer, reviewers, recorder, and reader. The manager does not attend. Some key principles govern the inspection session:
1. The inspection is limited to periods of peak concentration (1-2 hr.).
2. The product is reviewed, not the producer.
3. Issues are identified, not proposed solutions.

Each product component is inspected using the strongly preferred indicators found in the appropriate software product checklists. Each inspection team member in turn is asked if there is an issue to be raised for the product component and product checklist now before the group. If so, the issue is stated, discussed, and recorded. The producer may wish to obtain clarification of the issue at the time it is raised, but there is no need for the producer to defend or even explain the approach taken. The producer will have the opportunity to resolve the issue during the followup activity.

The moderator script for directing the conduct activity includes:
1. Are there any issues in completeness?
2. Are there any issues in correctness?
3. Are there any issues in style?
4. Are there any issues in rules of construction?
5. Are there any issues in multiple views?

Exit Criteria
Exit criteria are checked by the moderator at the close of the inspection session. The moderator verifies that all product components have been inspected and that the intended product checklists have been utilized. The recorder verifies that all the metrics have been recorded including preparation effort of each team member, the duration of the inspection session, and the size of the product being inspected as well as the defect type, defect category, defect severity, and defect origin for each issue raised. Finally the moderator asks the producer for any closing comments, permitting the producer to have the last word.

The moderator script for directing the exit criteria includes:
1. Have all product elements been inspected?
2. Have all checklists been processed?
3. Have the inspection results been recorded?
4. Have metrics been collected?
5. Would the recorder read back the issues?
6. What should be the disposition of the inspection?
7. Would the producer like an opportunity to comment?

Reporting
The moderator with the help of the recorder reports the findings of the inspection session to the manager within a week. This report provides a review summary, the preparation effort and conduct time expended, the types of defects detected, and followup recommendations.

Followup
The followup rework on the product is performed by the producer. The followup actions are prepared jointly by the manager and the producer and entered in the project action log. As these followup actions are completed, the action log reflects the closure. Tracking issues to closure is an important indicator of software process maturity.

System of Checklists

Checklists are at the heart of software inspections. They fuel the structured review process and form the standard of excellence expected for the software product. Checklists provide the criteria for evaluating the quality of the product as well as progress within the process. Process and product checklists promote uniformity in the use of software inspections throughout a project and across an organization. Process checklists are used as a guide for each activity of the structured review process to ensure that the Software Inspections Process runs smoothly. Product checklists provide reviewers with the thorough technical focus needed to guide the review of each product component from all viewpoints [SEI 88]. The system of checklists helps to overcome human limitations on information processing by focusing on just one consistent perspective at a time. See figure 6a for an example of a Design and Code Checklist and figure 6b for an example of a Specification Checklist [SEI 88].

Figure 6a Design and Code Checklist




Figure 6b Specification Checklist

Completeness
Completeness is based on traceability among software product artifacts of various types including requirements, specifications, designs, code, and test procedures. Completeness analysis may be assisted by tools that trace the components of a product artifact of one type to the components of another type. Completeness analysis of predecessor and successor artifacts reveals what sections are missing and what fragments may be extra. A by-product of the completeness analysis is a clear view of the relationship of requirements to the code product: straightforward (one to one), simple analysis (many to one) , and complex (one to many).

The moderator script for inquiring about completeness includes:
1. Has traceability been assessed?
2. Have all predecessor requirements been accounted for?
3. Were any product fragments revealed not to have traceability to the predecessor requirements?
4. Was traceability found to be straightforward, simple, or complex?

Correctness
Correctness is based on reasoning about programs through the use of informal verification and correctness questions derived from the prime constructs of structured programming and their composite use in proper programs [Linger 79], [McCabe 94]. Input domain and output range are analyzed for all legal values and all possible values. State data is similarly analyzed. Adherence to project specified disciplined data structures is analyzed. Asynchronous processes and their interaction and communication are analyzed [Prowell 99].

The moderator script for inquiring about correctness includes:
1. Is the function commentary satisfied?
2. Are programs limited to single entry and single exit?
3. Is the loop initialized and terminated properly?
4. Does the input domain span all legal values?
5. Is there systematic exception handling for illegal values?
6. Are disciplined data structures used?

Style
Style is based on project specified style guidance. This guidance is expected to call for block structured templates. Naming conventions and commentary are checked for consistency of use along with alignment, highlighting, and case. More advanced style guidance may call for templates for repeating patterns and semantic correspondence among software product artifacts of various types.

The moderator script for inquiring about style includes:
1. Are style conventions for block structuring followed?
2. Are naming conventions followed?
3. Are style conventions for commentary followed?
4. Are the semantics of the product component traceable to the requirements?
5. Are templates used for repeating patterns?

Rules of Construction
Rules of construction are based on the software application architecture and the specific protocols, templates, and conventions used to carry it out. For example, these include interprocess communication protocols, tasking and concurrent operations, program unit construction, and data representation.

The moderator script for inquiring about rules of construction includes:
1. Are guidelines for program unit construction followed?
2. Is the interprocess communication protocol followed?
3. Are data representation conventions followed?
4. Is the system standard time defined and followed?
5. Are encapsulation, localization, and layering used to achieve object orientation?
6. Is logical independence achieved through event driven and process driven paradigms, late binding, and implicit binding?
7. Is scalability achieved through uniformity, parameterization, and portability?
8. Are fault tolerance, high availability, and security achieved?

Multiple Views
Multiple views are based on the various perspectives and view points required to be reflected in the software product. During execution many factors must operate harmoniously as intended including initialization, timing of processes, memory management, input and output, and finite word effects. In building the software product, packaging considerations must be coordinated including program unit construction, program generation process, and target machine operations. Product construction disciplines of systematic design and structured programming must be followed as well as interactions with the user, operating system, and physical hardware.

The moderator script for inquiring about multiple views includes:
1. Has the logical view of user interface and object orientation considerations been assessed?
2. Has the static view of packaging considerations been assessed including program unit construction, program generation process, and target machine operations?
3. Has the dynamic view of operational considerations been assessed including communications, concurrency, synchronization, and failure recovery?
4. Has the physical view of execution considerations been assessed including timing, memory use, input and output, initialization, and finite word effects?

Defined Roles of Participants

Software Inspections are a reasoning activity performed by practitioners playing the defined roles of moderator, recorder, reviewer, reader, and producer. Some may name these roles facilitator, scribe, inspector, and author. Each role carries with it the specific behaviors, skills, and knowledge needed to achieve the expert practice of Software Inspections [Freedman/Weinberg 90].

Individuals attending the inspection session may take on more than one role. For example, the producer may also be a reviewer. The moderator and recorder roles are demanding ones, and the individual assigned is usually dedicated to the single role. The reader role is not always utilized in software inspections. When the reader is used, one of the reviewers, not the producer, is assigned this role. In software walkthroughs, the producer serves as reader.

Manager
The manager is active in the planning, preparation, reporting, and followup activities. In planning, the manager identifies and schedules all software inspections in the project plan. The manager identifies personnel resource needs in terms of labor hours and allocates them to each inspection. The moderator is assigned by the manager who ensures that only trained moderators are appointed.

The manager generally does not attend the inspection session. Practitioners are wary that managers attending an inspection session might use the results in the personnel performance appraisal of the producer. In addition, reviewers are reluctant to identify defects in the artifacts of their peers in the presence of managers. However, if the manager is an expert in the application and must be present to make a technical contribution, the manager must first check management and organizational behaviors at the door convincingly and attend the inspection as a technical peer.

After the software inspection is conducted, the manager receives the moderator’s report, meets with the producer to plan the followup, and administers the followup oversight.

Moderator
The moderator is the keystone of the Software Inspections Process and is active in the preparation, entry criteria, conduct, exit criteria, and reporting activities. The moderator directs the activities of the software inspection. During the preparation activity, the moderator briefs the inspection team members on their roles in the structured review process, asks the producer to overview the software product to be inspected, distributes the inspection materials, and announces the time and place for the inspection session.

During the inspection session, the moderator directs the entry criteria, conduct, and exit criteria activities and facilitates the interaction among the inspection team members. The moderator intervenes as little as possible and as much as necessary to ensure that an effective and efficient software inspection session takes place.

A skillful moderator recognizes the role specific needs of inspection team members. For example, a producer with a “good catch” on his own product is called upon first. A talkative reviewer with little preparation effort is controlled. Where the moderator has issues to bring up, it is good form to insert these after the other team members have spoken. The moderator collaborates with the recorder in preparing the report for the manager on the findings of the inspection session.

Producer
The producer is active during the preparation, entry criteria, conduct, exit criteria, and followup activities. The producer is responsible for creating the materials to be inspected. The producer attends the inspection as reviewer and is expected to raise issues. From time to time the producer may offer a technical explanation of the product as necessary.

The producer expects criticism of the product and need not offer any defense as issues are raised. It is understood that the producer may be in a protective state of mind with respect to the product being inspected. What is asked of the producer is that the protective state not be exhibited as defensive behavior. Where an issue is surfaced that is not understood by the producer, a dialogue may be needed to obtain clarification.

At the conclusion of the conduct activity, the producer is afforded the opportunity to comment on the inspection session and to acknowledge the value of the issues raised. The producer meets with the manager to plan the rework and performs the followup actions resulting from the inspection.

Recorder
The recorder is active in the preparation, entry criteria, conduct, exit criteria, and reporting activities. The recorder completes the Inspection Record, the Inspection Reporting Form, and the Report Summary Form. The practice of the recorder is “to leave no bits on the floor”. In other words, the recorder is expected to record every issue without exception.

During the entry criteria, the recorder notes the preparation effort of each inspection team member, the start and stop time of the meeting, the project and product name and size, and the life cycle activity for which the inspection is an exit criteria. As issues are raised, the recorder describes each issue and notes defect category, defect severity, defect type, and defect origin. The recorder uses the key word “investigate” in recording issues that may be defects but require additional research following the meeting. At the conclusion, the recorder tabulates the issues by defect type, severity, and category.

The role of the recorder is to be transparent to the inspection session and to record all issues completely and accurately. This requires a high degree of concentration, judgment, and technical knowledge.

Reviewer
Reviewers are active in the preparation, entry criteria, conduct, and exit criteria activities. A reviewer is expected to spend sufficient time preparing and to raise issues and concerns about the software product. Reviewers are asked to refrain from proposing solutions and to direct their comments at the product not the producer. Reviewers accept the discipline imposed by the round robin, checklist structure of the inspection session. In return for accepting these responsibilities and disciplines, each reviewer is assured of an uninterrupted opportunity to raise issues.

Reader
The reader is active in the preparation, entry criteria, conduct, and exit criteria activities. Where necessary, the moderator may ask the reader to read parts of the product aloud so as to focus attention on a particular trouble spot. The reader does this by paraphrasing not by reading line by line. Using the reader for this task helps promote the egoless behavior of the producer. The reader is responsible for bringing to the inspection session and being prepared to navigate any background materials, such as, baseline documentation and style guide.

Forms and Reports

All data collected and reported during the Software Inspections Process is recorded by the recorder. This includes data about the product being inspected and about the inspection process itself. The requirements for data collection are defined and focus on three recording instruments: Inspection Record, Inspections Reporting Form, and Report Summary Form.

Inspection Record
The Inspection Record is initiated during the entry criteria activity when the recorder gathers the preparation effort from each inspection team member. The name of the project and the product component are recorded along with the size of the product to be inspected. The life cycle activity for which this inspection serves as the exit criteria is recorded. The start time for the inspection session is entered at the beginning of the meeting, and the stop time is recorded at the close. Also at the close of the session, the disposition is recorded in terms of acceptance, reinspection, or conditional. See figure 7 Inspection Record.

Figure 7 Inspection Record

Inspection Reporting Form
During the conduct activity as issues are raised, the recorder documents a description of each issue and assigns attributes that characterize the issue. Each issue is assigned a sequence number, and the page and line number are pinpointed. Similar issues that occur a few times are recorded as separate issues. An issue type that occurs an uncountably large number of times is recorded once.

A defect category is assigned as missing, wrong, or extra. A defect severity is assigned as major or minor. The defect origin is noted as the life cycle activity during which this defect was inserted. The defect type is entered [Ebenau 94a]. See figure 8 Inspection Reporting Form.

Figure 8 Inspection Reporting Form

A major defect affects execution; a minor defect does not. In practice, some prefer to extend defect severity to include the extremes of critical and trivial. Other severity gradations used to classify defects, faults, and failures detected in testing and field operations are not used in software inspections. These test and operational execution-based severities often revolve around the criticality of the defect and its impact on sustaining testing or operations. Another dimension of defect severity is related to the effort needed to correct the defect.

The appropriate defect type is assigned as follows:
1. Interface: error in parameter list
2. Data: error in data definition, initial value setting, or use of disciplined data structures
3. Logic: error revealed through informal correctness questions spanning prime constructs of structured programming
4. I/O: error in formatting, commanding, or controlling I/O operations
5. Performance: error in managing or meeting constraints in computer resource allocations and capacities for CPU, memory, or I/O
6. Functionality: error in stating intended function or in satisfying intended function through refinement or elaboration
7. Human Factors: error in externally visible user or enterprise interface or interaction
8. Standards: error in compliance with product standards for construction or integration including programming style guidelines, open systems interfaces, or guidelines for the application domain architecture
9. Documentation: error in guidance documentation
10. Syntax: error in language defined syntax
11. Maintainability: error in uniformity and consistency
12. Other: any other error

Report Summary Form
During the exit criteria, the recorder completes the meeting stop time, verifies the completeness of all recorded results, and completes the Report Summary Form. This form is a frequency count of issues presented as a matrix of defect types by defect severity and defect category. This form serves several purposes. Since it cannot be constructed unless the recorder has completed the Inspection Reporting Form, it serves as an on the spot check of the recorded results. Once completed, weaknesses are highlighted and some opportunities for defect prevention suggest themselves. When the results of numerous inspection sessions are overlayed on the Report Summary Form, these frequency counts divided by the total defects serve as the probability of occurrence for each defect type, defect severity, and defect category. See figure 9 Report Summary Form.

Figure 9 Report Summary Form

Preparation for Expert Use

A collection of coordinated knowledge, skills, and behaviors facilitates the best possible practice of Peer Reviews. As Deming reminded us, there is no substitute for superior knowledge. In conducting Peer Reviews, superior knowledge is sought in the application domain, the computing platform both hardware and operating system, and programming language. In addition, participants must be knowledgeable in the Peer Review process and the standard of excellence expected in the product artifact.

For best results, participants filling certain defined roles must possess particular skills. The moderator needs facilitation, conflict identification, and conflict resolution skills. The recorder needs listening, synthesizing, and recording skills. The reviewer needs code reading skills.

Participants in Peer Reviews are expected to adopt certain behaviors known to contribute to effective and harmonious review sessions. First, the rules of civility apply. For example, one person speaks at a time, and personal attacks are not permitted. Second, since people make mistakes sometimes, it is necessary to decriminalize defects so that they do not remain hidden. Recognizing that, assigning blame for defects is discouraged. Third, participants are encouraged to direct their comments towards the product not the person who authored the artifact being reviewed. Finally, everyone is encouraged to give way to the individual who possesses superior knowledge.

Measurements

While many organizations have adopted software inspections, few have published their results. Those that have published results typically have done so following early successes in the new practice adoption cycle. Organizations with published results have included the Jet Propulsion Laboratory [Kelly 90], Litton Data Systems [Madachy 93], Bull HN Information Systems, Inc. [Weller 93], AT&T Corp. [Ebenau 94b], and Lockheed Martin Corporation [Bourgeois 96]. While all used software inspections and have documented measured results, the particular adaptations are not well aligned, and the results do not lend themselves to systematic comparison.

National Software Quality Experiment
In 1992 the DOD Software Technology Strategy set the objective to reduce software problem rates by a factor of ten by the year 2000. The National Software Quality Experiment is being conducted¹ to benchmark the state of software product quality and to measure progress towards the national objective [O’Neill 97c, 98, 99].

The centerpiece of the experiment is the Software Inspection Lab where data collection procedures, product checklists, and participant behaviors are packaged for operational project use. The uniform application of the experiment and the collection of consistent measurements are guaranteed through rigorous training of each participant.

Approximately three thousand participants from nearly sixty organizations have populated the experiment database with over twelve thousand defects of all types along with pertinent information needed to pinpoint their root causes. These results are highlighted below in the discussion of the common problems, Inspection Lab operations, defect type ranking, and return on investment.

Common Problems Revealed
Analysis of the issues raised in the experiment to date has revealed common problems that reoccur from session to session. Typical organizations which desire to reduce their software problem rates should focus on preventing the following types of defects:
1. Software product source code components are not traced to requirements.
As a result, the software product is not under intellectual control, verification procedures are imprecise, and changes cannot be managed.
2. Software engineering practices for systematic design and structured programming are applied without sufficient rigor and discipline.
As a result, high defect rates are experienced in logic, data, interfaces, and functionality.
3. Software product designs and source code are recorded in an ad hoc style. As a result, the understandability, adaptability, and maintainability of the software product are directly impacted.
4. The rules of construction for the application domain are not clearly stated, understood, and applied.
As a result, common patterns and templates are not exploited in preparation for later reuse.
5. The code and upload development paradigm is becoming predominant in emerging e-commerce applications.
As a result, the enterprise code base services only the short term planning horizon where code rules and heroes flourish, but it mortgages the future where traceable baseline requirements, specification, and design artifacts are necessary foundations.

Inspection Lab Operations
The Inspection Lab is the consistent operation of software inspection sessions as part of the National Software Quality Experiment. These sessions apply the elements of software inspections including the entry, conduct, and exit processes; defined roles of participants; product checklists; and forms and reports. Through 1999, 2669 participants conducted inspection sessions. A total of 903,979 source lines of code have received strict and close examination in the Software Inspection Lab. There have been 161,139 minutes of preparation effort and 61,547 minutes of conduct time expended to detect 12,378 defects. See figure 10 Inspection Lab Operations.

Figure 10 Inspection Lab Operations

Of these 13,042 defects, 2116 were classified as major, and 10,926 as minor. A major defect effects execution; a minor defect does not. It required 12.36 minutes of preparation effort on the average to detect a defect. To detect a major defect required 76.15 minutes of preparation effort on the average. On the average, .811 thousand source lines of code were examined each inspection conduct hour. There were 2.34 major defects detected in each thousand lines, and 12.09 minor defects. There were 4.89 defects detected in inspecting 338.70 lines per session. The preparation effort was 0.65 of conduct effort. The Software Inspection Labs produced a return on investment of 4.41.

Defect Type Ranking
The foremost defect types that accounted for 92.97% of all defects detected include the following. See figure 11 Defect Type Distribution.
Documentation 42.97% error in guidance documentation
Standards 20.68% error in compliance with product standards
Logic 7.13% error revealed through informal correctness questions function
Functionality 6.61% error in stating or meeting intended
Data 4.70% error in data definition, initial value setting, or use
Syntax 4.69% error in language defined syntax compliance
Maintainability 3.89% error in good practice impacting the supportability and evolution of the software product
Performance 2.30% error in managing or meeting constraints in computer resource allocations and capacities for CPU, memory, or I/O

Figure 11 Defect Type Distribution

Return on Investment
Managers are interested in knowing the return on investment to be derived from software process improvement actions. The Software Inspections Process gathers the data needed to determine this [McGibbon 96].

The Return on Investment for Software Inspections is defined as Net Savings divided by Detection Cost, where Net Savings is Cost Avoidance less Cost to Repair and Detection Cost is the cost of preparation effort and the cost of conduct effort. The defined measurements collected in the Software Inspections Lab may be combined in complex ways to form this derived metric.

The model for Return on Investment bases the savings on the cost avoidance associated with detecting and correcting defects earlier rather than later in the product evolution cycle. A Major Defect that leaks from Development to Test may cost as much as ten times to detect and correct. Some defects, undetected in test, continue to leak from Test to Customer Use and may cost an additional ten times to detect and correct. A Minor Defect may cost two to three times to correct later. IBM Rochester, winner of the Malcolm Baldrige Award, reported that defects leaking from code to test cost nine times more to detect and correct, and defects leaking from test to the field cost thirteen times more [Lindner 94]. See figure 12 Defect Detection and Defect Leakage Model.

Figure 12 Defect Detection and Defect Leakage Model

Figure 13 is a graph showing the Return on Investment Measurements for each organization participating in the National Software Quality Experiment. This graph suggests that the Return on Investment for software inspections ranges from 4:1 to 8:1. For every dollar spent on software inspections, the organization can expect to avoid 4-8 dollars on higher rework cost.

Figure 13 Return on Investment Measurements

Peer Reviews Roll Out

Peer Reviews has application throughout the software organization. Consequently a systematic approach is needed to introduce it to all participants. The steps in the defined program for rolling out Peer Reviews within the organization include [O’Neill 97a, 97b]:
1. Assess Peer Reviews practice
2. Obtain management commitment
3. Conduct Software Inspections training for practitioners, managers, and executives
4. Prepare and disseminate Software Inspections Policy and Procedure documents
5. Establish a coordination infrastructure, assign personnel, and formulate a program agenda
6. Establish an inspection-based measurement program and database
7. Set management objectives for planning, training, conducting, and using software inspections and measurements
8. Continue to evolve the organization's process and product checklists

The initial step in the roll out is to conduct an assessment of the Peer Reviews practice. This will identify the strengths and weaknesses in planning, training, conduct, and reporting and use of results. Armed with this assessment, the change agent seeks management commitment and sponsorship for the improvements needed to offset the weaknesses. The following key practice indicators [Paulk 95] are assessed at regular intervals for each project in terms of approach, deployment, and results:

• The project follows a documented organization policy for performing software inspections.
  
• Adequate resources and funding are provided for performing software inspections on each software work product to be reviewed.
  
• Moderators and reviewers receive required training in how to lead software inspections, as well as in objectives, principles, and methods of software inspections.
  
• Software inspections are planned and documented.
  
• Software inspections are performed according to a documented procedure
  
• Data on the conduct of the software inspections are recorded.
  
• Measurements are made and used to determine the status of the software inspections activities.
  
• The software quality assurance group reviews and/or audits the activities and work products for inspections and reports the results.
  

A clear management commitment is needed to approve and fund the training program including the cost of the instructor, training site, labor and burden for student attendance, and scheduling and administration of student enrollment and attendance.

With training underway and inspections being initiated by projects, Peer Reviews policy and procedure documents provide the guidelines for the well defined process being deployed. The policy applies to all software development projects and states that each project involved in software development shall prepare, plan, conduct, and utilize software inspections. The purpose of the procedure is to provide the step by step instructions needed to carry out consistent examinations of work products on the project.

To fully stimulate the Peer Reviews roll out program among project personnel, an infrastructure of coordinators drawn from active projects is convened. These project coordinators meet periodically to share experiences, compare results, and discuss common problems.

As inspection results and measurements are accumulated, a measurement database and the operations for analyzing, reporting, and acting upon these measurements are established. The measurements are recorded in the Software Inspection Lab. These measurements include preparation effort, conduct time, and the artifact size in lines of code or pages inspected and for each defect detected the defect severity, defect category, defect type, and defect origin. Using the measurements, metrics are derived to continually assess the efficiency and effectiveness of the process and its operation. These metrics include:
1. Minutes of preparation effort per defect
2. Minutes of preparation effort per major defect
3. Major defects per thousand lines
4. Minor defects per thousand lines
5. Lines per conduct hour
6. Defects per session
7. Preparation/ conduct effort
8. Lines per session
9. Return on Investment

Future Directions

In reasoning about future trends of Peer Reviews, the topics considered include increasing rate of software problems, improving the practice of defect prevention and prediction, extending the practice of Peer Reviews to systems engineering, understanding the process of experimentation in software development, exploiting technology in automating the Peer Reviews, and adapting to changes in business environment.

Software problem rates may be increasing. The results of the National Software Quality Experiment show no systematic improvement towards fulfilling the national goal of a ten times reduction in software problems. The quality goal is not being met by a wide margin. Instead problem rates are being pushed higher:

• With the emphasis on quicker, better, and cheaper
  
• With the trend towards code and upload practice as the life cycle model
  
• With the preoccupation on improving software process maturity and mastering the management track practices of the Software Engineering Institute’s Capability Maturity Model for Software level 2, an obstacle to many
  
• With the downsizing of middle management and senior technical staff who often held the line on product quality.

While software inspections have been in use for over twenty-five years, defect prevention remains an immature practice. Defect prevention is a CMM level 5 key process area, and some organizations have achieved level 5. As more organizations seek to adopt the practice of defect prevention, its benefits and methods may become more well understood stimulating others to adopt the practice.

Similarly, defect prediction remains an underdeveloped practice. If software defects, faults, and failures can be predicted, perhaps they can be detected, controlled, and prevented. Model based techniques calibrated with defect detection early in the life cycle to predict defect rates in later life cycle activities have been demonstrated [Gaffney 97]. More modest efforts utilizing software inspections data to estimate the number of defects remaining to be found in testing are being applied on the project [Harding 98]. However, there is insufficient defect, fault, and failure data available from the nation’s factory floor [O’Neill 99]. In addition there is insufficient process, method, and tooling to combine defect data obtained through software inspections practice, software fault data obtained through software product test and use, and software failure data obtained through software system operation into predictions of trustworthy software system operation [Wallace 97].

While the benefits and usage of software inspections on code artifacts is well known, there is increasing interest in extending software inspections to all phases of the life cycle including requirements, specifications, design, code, and test artifacts. The CMMI project with the inclusion of Peer Reviews in the Product Verification process area extends Peer Reviews to both systems engineering and software artifacts.

To achieve the best possible practice of software inspections, both managers and technical practitioners are encouraged to decriminalize defects. People make mistakes sometimes, yet software must be bit perfect. When managers and technical participants view with alarm the defects detected in software inspections, it produces a negative impact. On the other hand, when managers genuinely decriminalize defects and use their detection as a means to prevent their recurrence, it produces a positive result. During a software inspection session, the litmus test for decriminalization lies in the reaction of participants when a major defect is detected. Does the group say ‘good catch’ or ‘bummer’?

With the growing recognition that fielding software involves a process of experimentation and with the increasing pressures of competition and demand for innovation, software walkthroughs may experience increasing in usage. Software walkthroughs encourage and support the learning essential to experimentation. In favoring the group interaction needed to achieve consensus, software walkthroughs may contribute to increased innovation in software products.

There is interest in automating software inspections. The value of programming languages with strong typing, robust compilers, static analyzers and traceability tools, and complexity metrics [McCabe 94] is recognized. However, software inspections is a reasoning activity and will remain essentially a human activity. The use of information technology innovations to support the logistics of preparation, scheduling, conduct, and results repository operations are sources for improved industry practice.

Software inspections are being conducted effectively using groupware tools. However, where global software development teams conducting geographically dispersed inspection sessions are using ‘follow the sun’ software development tactics, software inspection participants may be separated by both geography and time zones complicating the logistics of their application [Carmel 99].

Software inspections usage is increasing in e-commerce applications where code and upload is the typical life cycle practice. In an environment of rapid change and frequent releases, there is an absence of robust testing and sometimes even regression testing.

Conclusion

Peer Reviews deliver value to the organization through the close and strict examinations on life cycle product artifacts that detect defects early and promote the deepest possible understanding of the artifact. The organization that sets the standard of excellence for its software engineered products in terms of completion, correctness, style, rules of construction, and multiple views and disciplines its practitioners to meet the standard set is able to reap an attractive return on investment while earning higher customer satisfaction. The measurements taken during software inspections promote an understanding of common problems and reveal opportunities for product and process improvement.

Bibliography

[Bourgeois 96] Bourgeois, Karen V., “Process Insights from a Large-Scale Software Inspections Data Analysis”, CrossTalk, The Journal of Defense Software Engineering, Vol.9 No. 10, October 1996, pp. 17-23.

[Carmel 99] Carmel, Erran, “Global Software Teams: Collaborating Across Borders and Time Zones”, Prentice Hall, 1999, 269 pages.

[Ebenau 94a] Ebenau, Robert G. and Susan H. Strauss, "Software Inspection Process", McGraw-Hill, Inc., 1994, pp. 236-240.

[Ebenau 94b] Ebenau, Robert G., "Predictive Quality Control with Software Inspections", CrossTalk, The Journal of Defense Software Engineering, Vol.7 No. 6, June 1994, pp. 9-16.

[Fagan 76] Fagan, M., "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal, 15, 3, 1976, pp. 182-211.

[Fagan 87/] Fagan, M., "Advances in Software Inspections", IEEE Transactions on Software Engineering, 12, 7, 1987.

[Freedman/Weinberg 90] Freedman, D.P., G.M. Weinberg, "Handbook of Walkthroughs, Inspections, and Technical Reviews", Dorset House Publishing Co., Inc., 1990, pp. 89-161.

[Gaffney 97] Gaffney, John E., “Software Defect Estimation, Prediction, and the CMM”, Metrics ‘97 Conference, 1997.

[Gilb 93] Gilb, Tom and Dorothy Graham, “Software Inspection”, Addison Wesley Longman Limited, 1993, pp. 40-136.

[Harding 98] Harding, John T., “Using Inspection Data to Forecast Test Defects”, CrossTalk, The Journal of Defense Software Engineering, Vol.11 No. 5, May 1998, pp. 19-24.

[Humphrey 89] Humphrey, Watts S., "Managing the Software Process", Addison-Wesley Publishing Company, Inc., 1989, pages 171-190, pp. 463-486.

[Humphrey 95] Humphrey, Watts S., “A Discipline for Software Engineering”, Addison-Wesley Publishing Company, Inc., 1995, page 233.

[Johnson/Brodman 96] Johnson, Donna L. and Judith G. Brodman, “Realities and Rewards of Software Process Improvement”, IEEE Software , Vol. 13 No. 6, November 1996.

[Kelly 90] Kelly, J., J. Sherif, “An Analysis of Defect Densities Found During Software Inspections”, Proceedings of the Fifteenth Annual Software Engineering Workshop, Goddard Space Flight Center, Greenbelt, Maryland, December 1990.

[Lindner 94] Lindner, Richard J. & Tudahl, D. "Software Development at a Baldrige Winner," Proceedings of ELECTRO `94, Boston, Massachusetts, May 12, 1994, pp. 167-180.

[Linger 79] Linger, R.C., H.D. Mills, B.I. Witt, "Structured Programming: Theory and Practice", Addison-Wesley Publishing Company, Inc., 1979, pp. 147-212.

[Madachy 93] Madachy, R., L. Little, S.Fan, “Analysis of a Successful Inspection Program”, Proceedings of the Eighteenth Annual Software Engineering Workshop, Goddard Space Flight Center, Greenbelt, Maryland, December 1993, pp. 176-188.

[McCabe 94] McCabe, Thomas J. and Arthur H. Watson, “Software Complexity” CrossTalk, The Journal of Defense Software Engineering, Vol. 7 No. 12, December 1994, pp. 5-9.

[McGibbon 96] McGibbon, T., “A Business Case for Software Process Improvement”, Rome Laboratory DACS Report, 30 September 1996.

[O’Neill 97a] O'Neill, Don, "Issues in Software Inspection", IEEE Software,
Vol .14 No 1, January 1997, pp. 18-19.

[O’Neill 97b] O’Neill, Don, “Setting Up a Software Inspections Program”, CrossTalk, The Journal of Defense Software Engineering, Vol.10 No. 2, February 1997, pp. 11-13.

[O’Neill 97c] O'Neill, Don, "National Software Quality Experiment: A Lesson in Measurement 1992-1996", Quality Week Conference, San Francisco, May 1997 and Quality Week Europe Conference, Brussels, November 1997, pp. 1-25.

[O’Neill 98] O’Neill, Don, “National Software Quality Experiment: A Lesson in Measurement 1992-1997”, CrossTalk, The Journal of Defense Software Engineering, Vol. 11 No. 12, Web Addition, December 1998.

[O'Neill 99] O’Neill, Don, “National Software Quality Experiment: A Lesson in Measurement 1992-1997”, First Annual International Software Assurance Certification Conference, Chantilly, Virginia, 1 March 1999, pp. 1-14.

[Paulk 95] Paulk, Mark C., “The Capability Maturity Model: Guidelines for Improving the Software Process”, Addison-Wesley Publishing Company, 1995, pp. 270-276.

[Prowell 99] Prowell, Stacy J., Carmen J.Trammell, Richard C. Linger, Jesse H. Poore, “Cleanroom Software Engineering: Technology and Process”, Addison-Wesley Longman, Inc., 1999, page 17, 33-90.

[SEI 88] O'Neill, Don and Albert L. Ingram, "Software Inspections Tutorial", Software Engineering Institute Technical Review 1988, pp. 92-120.

[SEI 89] O’Neill, Don, “Software Inspections Course and Lab”, Software Engineering Institute, 1989.

[SEI 97] O’Neill, Don, “Software Inspections”, Software Technology Guide, Software Engineering Institute, 10 January 1997.

[SEI 00] “Process Maturity Profile of the Software Community 1999 Year End Update”, Software Engineering Institute, March 2000.

[SPIN 96] “Newsletter of the Boston Software Process Improvement Network”, Issue 8, January/February 1996.

[Wallace 97] Wallace, Dolores R., Laura M. Ippolito, and Herbert Hecht, "Error, Fault, and Failure Data Collection and Analysis", Quality Week, San Francisco, May 1997.

[Weller 93] Weller, Edward F., “Lessons From Three Years of Inspection Data”, IEEE Software, September 1993, pp. 38-45.

Author: Don O’Neill

Don O’Neill is a seasoned software engineering manager and technologist currently serving as an independent consultant. Following his twenty-seven year career with IBM’s Federal Systems Division, Mr. O’Neill completed a three year residency at Carnegie Mellon University’s Software Engineering Institute (SEI) under IBM’s Technical Academic Career Program. There he developed a blueprint for charting software engineering evolution in the organization including the training architecture and change management strategy needed to transition skills into practice.

As an independent consultant, Mr. O’Neill conducts defined programs for managing strategic software improvement. These include implementing an organizational Software Inspections Process, implementing Software Risk Management, and conducting Global Software Competitiveness Assessments. Each of these programs includes the necessary practitioner and management training.

In his IBM career, Mr. O’Neill completed assignments in management, technical performance, and marketing in a broad range of applications including space systems, submarine systems, military command and control systems, communications systems, and management decision support systems. He was awarded IBM’s Outstanding Contribution Award three times.

1. Software Development Manager for the Global Positioning Ground Segment (500,000 source lines of code) and a team of 70 software engineers within a $150M fixed price program.
2. Manager of the FSD Software Engineering Department responsible for the origination of division software engineering strategies, the preparation of software management and engineering practices, and the coordination of these practices throughout the division’s software practitioners and managers.
3. Manager of Data Processing for the Trident Submarine Command and Control System Engineering and Integration Project responsible for architecture selections and software development planning (1.2M source lines of code).

Mr. O’Neill served on the Executive Board of the IEEE Software Engineering Technical Committee and as a Distinguished Visitor of the IEEE. He is a founding member of the National Software Council and the Washington DC Software Process Improvement Network (SPIN). He is an active speaker on software engineering topics and has served as the Program Chairman and Program Committee member for several conferences. He has numerous publications to his credit. Mr. O’Neill has a Bachelor of Science degree in mathematics from Dickinson College in Carlisle, Pennsylvania.

Contact Information

Don O’Neill
Independent Consultant
9305 Kobe Way
Montgomery Village, Maryland 20886

email: ONeillDon@aol.com
Phone: (301) 990-0377
Web Site: http://members.aol.com/ONeillDon/index.html


Don O’Neill Photo oneill.gif
http://members.aol.com/ONeillDon/oneill.gif

word count: 10,187
figures: 13 * 200 words = 2,600
total: 12,787

¹ The National Software Quality Experiment is an entrepreneurial activity.

Don O’Neill Consulting, 2000 # Peer Reviews