Taking the Art out of Software Development An In-Depth Review of Cleanroom Software Engineering

by Chaelynne M. Wolak wolakcha@scsi.nova.edu

A paper submitted in fulfillment of the requirements for DISS 725 Research Paper One DISS 725 Spring 2001

School of Computer and Information Sciences Nova Southeastern University April 2001

Introduction
Chief scientist, Caper Jones, of Software Productivity Research in Massachusetts, estimates that 60 percent of the United States software work force is dedicated to fixing software errors that could have been avoided. In addition, he states there are only 47 days in a calendar year dedicated to doing actual development or enhancement of software applications. Most of the software engineering work is wasted due to fixing errors or working on projects that are eventually cancelled (Chapman, 2000). It appears product reliability has fallen to the waste side. The 1980s emphasized quality control, however by the 1990s innovation at top speed has taken top priority. In America, being first to market is the ultimate goal. Forget the errors as those can be fixed later by technical support staffs or customer service staffs (Barron, 2000). However, there is a new era of computing upon us. The current era of desktop computing will soon pass evolving to a new era of globalization. Software globalization will form the new infrastructure of our society. It will require desktop computers to be an essential business tool rather than the occasional productivity enhancer. In essence, this new era will require high dependability and reliability due to software computing now performing life critical functions (Morris, 2000). Therefore, there must be a better method to software development without creating numerous defects. One approach to developing quality software with minimal defects is Cleanroom Software Engineering (CSE). This academic research paper provides an in-depth review of CSE including a description of what it is, the advantages and disadvantages of using it, and real world examples of its use. Lastly, the paper concludes with some thoughtful points on how CSE can be used to develop better commercial-off-the-shelf (COTS) software.

Cleanroom Software Engineering (CSE) Defined

Harlan Mills and his colleagues from IBM developed the CSE methodology in the early 1980s. They were part of IBMs Federal Defense System where software failures could mean millions of dollars and most importantly, human lives (Head, 1994). This software methodology follows the same analogy as cleanroom fabrication of semiconductors. Instead of trying to clean dirt off the semiconductor wafers after production, the object is to prevent the dirt from getting into the production environment in the first place. The reason for this is that defect prevention is more cost effective than defect removal (Deck, 1994). Therefore, in software development, the CSE methodology eliminates or avoids as many defects as possible before software execution using controlled and measurable statistics (Stavely, 1999). CSE focuses on theory-based engineering practices of software development and certification while using the Capability Maturity Model for Software (CMM) as its framework (Oshana, 1998). This framework (i.e. CMM) describes the key elements of an effective software process such as planning, engineering, managing development and maintenance. In addition, CMM establishes a means to measure an organizations software development process by

maturity levels (Oshana & Linger, 1999). Nevertheless, the main point is that CSE encompasses CMM as part of its software development process. CSE spans the entire software life cycle. It has an 80-20 life cycle. This means 80 percent of the time is dedicated to design while 20 percent is writing code. In contrast, the traditional life cycle is 40-20-40 where 40 percent is design, 20 percent is coding, and 40 percent is unit testing. There is absolutely no unit testing as part of the CSE process (Head, 1994). The untested software product is expected to work. If it does not, the defects are analyzed to determine how the software development process should be improved. Then the defective product is discarded. According to Harlan Mills, the most important tool to CSE is the wastebasket (Head, 1994). CSE is based on two main principles. They are the design and testing principles. The design principle consists of mathematical functional verification of the product. Here reviewers of the software product use mathematical functions to argue the correctness of the original design. The testing principle focuses on certifying the reliability of software product (i.e. measuring the quality). This technique uses statistical testing based on expected system usage rather than trying to remove defects (Becker, Deck, & Janzon, 1996). These may be the two fundamental principles to CSE. However, there are four main components to the CSE framework. These components are discussed next. Main Components The four main components to CSE include the following: incremental development process model; formal methods for specification and design; correctness verification of developed code; and statistically based, independent testing (Grillo & Tewari, 1997). Figure one in Appendix A shows the CSE Model. Each of these components is necessary to develop software code that prevents errors from the beginning of development. Step 1: Incremental Development Process Model. In the incremental development process of CSE, each increment is a functional subset of the final software product. These increments are developed and tested in the environment similar to that of the final software application, so inferences about field quality can be drawn from the test results (Deck & Whittaker, 1997). If the increment adheres to pre-established quality standards, then the next incremental development process is started. If not, then development is stopped and the process is fixed (Oshana, 1998). By having many increments (i.e. mini releases) within one software project, the results from one increment can be used to modify the process for subsequent increments. This is a great advantage to any iterative process (i.e. incremental, spiral, evolutionary) because it has the ability to use feedback from early iterations to improve the process. Thus, the chances for success are greater. Therefore, the incremental development process is a technique for managing risk and incorporating risk alleviation into the CSE process (Deck & Whittaker, 1997). Step 2: Formal Methods for Specification and Design.

The Box Structure Method is typically used for specification and design of the CSE process. It is used to bridge the gap between system-level thinking and design-level thinking (Deck, 1996, October 10-11). The key concept of CSE is that a software program is a rule for a mathematical function. Thus, a software program executes transformations based on inputs (i.e. domains) to outputs (i.e. ranges). The transformations are specified as functional mappings (Grillo & Tewari, 1997). The box structure method is used as a stepwise refinement and verification process. It defines three levels of abstractions for defining software programs as rules for functions. The three levels of abstractions are the behavioral view, the finite state machine view, and the procedural view (Oshana, 1998). These levels of abstraction are classified as black, state, and clear boxes respectively. The black box defines usage hierarchy and may be referenced by the state and clear box. The state box view details about the data implementation while the clear shows the procedural view (Deck, 1996, October 29-30). Nevertheless, the box structure method helps to create clear and well-defined specifications. Step 3: Correctness Verification of Developed Code. Correctness verification is believed to be the heart of CSE. This component is primarily responsible for the dramatic improvements in quality (Head, 1994). Here the testing process is completely separated from the development process. Team reviews use mathematical verification techniques to verify the correctness of the software program. Thus, by being able to reason mathematically the effect of each code statement, this proves that the software program can meet its required specifications (Grillo & Tewari, 1997). This step compares to unit testing of traditional software development methods. Unit testing is where the program developer will test their program for the purpose of finding defects. However, in CSE, team reviews replace this. There are two main reasons why team reviews are considered more effective than unit testing. They are more thorough and cost effective (Deck, 1997, May 27-30). Step 4: Statistically Based, Independent Testing. Lastly, the software program is tested based on statistical principles. First all possible software uses are generated into statistical subsets. Performance is measured on these subsets based on the usage model developed. Then the expected operational use is represented in a usage model. Lastly, test cases are randomly generated and executed in the operational environment. Any failures are interpreted according to statistical models (Oshana, 1998). An independent team does the statistical testing. This testing is viewed as a sampling problem in CSE versus coverage testing for debugging in the traditional approach. The team does not assist the development group in getting the software product to an acceptable level of quality, but rather certifies its reliability (Grillo & Tewari, 1997). Summary In summary, CSE is an error-removal software development methodology. It centers around two principles design and testing. In addition, it has four main components to its

process. They are the incremental development process model, formal methods for specification and design, correctness verification of developed code, and statistically based, independent testing. These components differentiate it from most traditional software development models. There are three areas of differentiation. First, the programmer is removed from any testing activity. Second, the programmer must do formal (measurable) verification. Lastly, all testing must be statistical in nature (Glass, 1999). Thus, it would appear CSE has a distinct advantage over all other software development models. However, that may not be the case. Next, a review of the advantages and disadvantages is presented.

Advantages & Disadvantages of CSE

CSE can be difficult to understand. This is especially true due to the unavailability of literature on CSE. However, publications are forthcoming as more organizations embrace it (Deck, 1997, May 27-30). In this section, the advantages and disadvantages of CSE are discussed. Advantages CSE has at least three main advantages. They are improved quality, increased productivity, and improved software maintainability. CSE improves quality by preventing errors in the first place. Thus, there is a reduction in the overall cost of the program (Kelly & Oshana, 1996). For example, from 1987 to 1994, software code developed with CSE only had 2.3 defects per thousand lines of code measured from the first time of execution. In contrast, industry averages using traditional software development methods were estimated at 10 defects per thousand lines of code after unit testing (Stavely, 1999). Another advantage CSE has over traditional software development methods is productivity. Productivity is increased due to the reduced time required to debug and rework software (Kelly & Oshana, 1996). For example, the industry productivity rate for non-comment source statements (NCSS) per engineer month is 120. Conversely, with a trained cleanroom team, the productivity rate is 800 NCSS (Head, 1994). That is an additional 680 NCSS just using CSE! Lastly, software developed with CSE has clear, well-defined specifications with a lesscomplicated design allowing for easier maintenance (Kelly & Oshana, 1996). This is because CSE incorporates team reviews as part of the process and uses the box structure method for creating specifications. In a team review situation, there is not the crutch of relying on unit testing to find defects. In addition, team reviews generally are a more cost-effective means of improving quality and disseminating knowledge (Deck, 1997, May 27-30). In addition, the use of box structures creates a less-complicated design. Box structures permit a user-view black box specification that is written in terms of system usage. Therefore, specifications are detailed, clear, and well-defined creating a comprehensible view of the

software program. This results in the software program being much easier to maintain (Deck, 1994). Despite these advantages, CSE is still not a widely accepted software development practice (Deck, 1997, May 27-30). In fact, there are quite of few critics. Next is a discussion of the disadvantages of CSE.

Disadvantages
There are three reasons why critics dislike CSE. First, there is the belief that CSE is too theoretical, too mathematical, and just plain too radical for software development. Even the correctness proof portion of CSE can be difficult and time consuming especially for non-safetycritical programs (Deck, 1997, May 27-30). CSE does require incremental development, the use of box structures for well-defined specifications, and statistical testing for verification. CSE use requires intensive training. Once trained, an organization can save time and money. Nevertheless, an organization can try to add small parts of the CSE process to their normal software development method. Once confidence is built, additional components of CSE can be added to their original process (Henderson, 1995). Second, unit testing is not part of the CSE process. Denying the programmer access to the compiler is unrealistic and can be counterproductive. This is especially true if the programmer does not even know the language or environment, when the software program is started (Deck, 1997, May 27-30). However, unit testing only builds confidence about the specific paths tested. Thus, only high-frequency errors are found from the users point of view. Unit testing is unlikely to find rare execution errors. Conversely, CSE uses statistical testing which tends to find errors in the order of seriousness and can uncover failures more effectively than randomly through unit testing (Henderson, 1995). Lastly, there is the belief that the software industry is still in its infancy to incorporate such a rigorous process as CSE. The software development paradigm must change from one of art to one of science. In many engineering disciplines, failures are not an option and definitely are not considered the norm. In order to implement CSE successfully, one must not only have formal training but commitment from management (Henderson, 1995). Next, a review of some organizations that have successfully implemented CSE is discussed.

Real World Examples of CSE

Although CSE is not widely accepted, it has been used in many industries. Industries such as aerospace, telecommunications, graphical, and CASE tools have implemented CSE successfully (Deck, 1997, May 27-30). Listed below is a sample of projects that have successfully used CSE. The first application of CSE was to develop the COBOL Structuring Facility (COBOL/SF) product at IBM. There were 80 thousand lines of code (KLOC) of which 52,000 lines were modified or added. This project was split into five increments of four to 19 KLOC. It

was estimated that 1,500 to 3,000 errors were eliminated through team reviews. Productivity was measured to be 740 lines of code per labor month. Typically, the industry averaged only 150 lines of code per labor month (Deck, 1994). In addition, this product had required only a small fraction of its maintenance budget in its operating history (Foreman, 1997). Another successful CSE implementation was with the Software Engineering Laboratory (SEL) of the NASA Goddard Space Flight Center. CSE was applied to a 40 KLOC project. Results of this project included a 69 percent increase in productivity, a 45 percent decrease in error rate, and approximately 60-80 percent decrease in resource usage compared to other SEL projects (Deck, 1994). The US Department of Defense (DoD) Software Technology at the Picatinny Arsenal used CSE for software enhancements for a set of Army weapon systems (Deck, 1994). The program was approximately 90 KLOC. This project had results of a 4.2:1 productivity increase and a 20:1 return on investment (ROI) (Foreman, 1997). One of the largest implementations of CSE was in the telecommunications industry. Ericsson used CSE to develop its OS32 operating system for telecommunication switches. This project had 350 KLOC. Ericsson had seen the following results: a 114 percent productivity increase in testing, a 70 percent increase in design productivity, a 157 percent increase in quality, a 50 percent decrease in failures per KLOC, and a 4.4 percent decrease in resource usage (Deck, 1994). Results like those above cannot be ignored. If history were any indicator of future performance, it would appear that many organizations would benefit from implementing CSE into its normal software development process. A prime example of an industry who could substantial benefit from CSE is those that provide commercial-off-the-shelf (COTS) software.

Thoughtful Points about COTS Software and CSE

It is a myth that software must contain defects (Head, 1994). However, we tend to accept them as part of the normal development process. This self-fulfilling prophecy prevents defectfree software from being introduced to the marketplace (Head, 1994). For example, when the author of this paper thinks of the next operating system from Microsoft, she expects that there will be many errors. Typically, COTS software relies on updates and patches for errors to be fixed. It is readily apparent that COTS software companies would benefit by incorporating CSE as part of their software development process. Although this industry has heard about CSE, they believe it is not well-suited to the COTS software environment. This is because developers must use new, unstable, or poorly documented languages or platforms. In addition, most COTS software is rarely used for lifecritical applications (Deck, 1996, October 29-30). However, as stated before, there is a new era of computing upon us. Software globalization will require reliable software systems.

CSE is not an all or nothing technology. It should be used to complement other software development life cycles (Oshana, 1998). It is recommended that CSE be phased-in with the current software development methodology. This will make adoption much easier (Deck, 1994). For example, Microsoft could use a phased-in version of CSE for their current Windows 2000 operating system. First Microsoft could replace beta testers with team reviews of program code (i.e. verification of correctness). Microsoft, along with other software companies, tends to market error-ridden programs and then uses consumers as their free developer testers. What makes this comical is that as consumers we purchase those software problems. In addition, we spend endless hours explaining the problem or trying to resolve it. Another phased-in approach Microsoft could use is to perform the statistical testing on the marketable version of the Windows 2000 operating software. Introducing reliability measurements is an easy extension after the commitment to statistical testing has been made (Henderson, 1995). This way Microsoft could proactively find errors instead of relying on consumer usage. After finding these errors and correcting them, Microsoft could continue the norm of providing updates or software patches. Now, Microsoft is not the only supplier of COTS software that could benefit from CSE. There are numerous companies. However, Microsoft provides the operating system for which other COTS software is developed. They are the foundation and this foundation should have much better quality than what is currently released. Developing high quality software should be in every companys best interest. CSE has plenty of benefits that outweigh any obstacle or excuse.

Conclusion
If anything is new to CSE, it is the attitude that near zero-defect software is possible (Deck, 1994). This paper provided an in-depth review of CSE including a description of what it is, the advantages and disadvantages of using it, and real world examples of its use. In addition, this paper provided some thoughtful points on how CSE could be phased-in to develop better COTS software. CSE is still thought of as some mythical software development process. However, more companies need to make quality software besides innovation as top priority. Further research is still needed on CSE. More companies need to pilot and publish how CSE was integrated in their current software development process. Once this happens, CSE will not be this unachievable mythical creature. It is sad fact that 60 percent of the U.S. workforce is fixing software errors (Chapman, 2000). Developing high quality software that is within budget and does not require a massive customer service staff should be on any software companys objective list. CSE is no silver

bullet, however it does provide a more scientific and measurable process that takes the art out of software development.

References
Barron, C. A. (2000, December 6). High Tech's Missionaries of Sloppiness. Salon. Retrieved March 12, 2001, from the World Wide Web:http://www.salon.com/tech/feature/2000/12/06/bad_computers/index.html. Becker, S. A., Deck, M., & Janzon, T. (1996, October 29-30). Cleanroom and Organizational Change. Paper presented at the Proc. 14th Pacific Northwest Software Quality Conference, Portland, Oregon. Chapman, G. (2000, November 27). No 'Silver Bullet' for Software's Growing Complexity. LA Times. Retrieved March 12, 2001, from the World Wide Web:http://www.latimes.com/business/columns/dnation/20001127/t000113753.html. Deck, M. (1994, October). Cleanroom Software Engineering: Quality Improvement and Cost Reduction. Paper presented at the Proc. 12th Pacific Northwest Software Quality Conference. Deck, M. (1996, October 10-11). Data Abstraction in the Box Structures Approach. Paper presented at the Proc. 3rd International Conference on Cleanroom Software Engineering Practices, College Park, Maryland. Deck, M. (1996, October 29-30). Cleanroom Review Techniques for Application Development. Paper presented at the International Conference on Software Quality, Ottawa, Ontario, Canada. Deck, M. (1997, May 27-30). Cleanroom Software Engineering Myths and Realities. Paper presented at the Quality Week, San Francisco, CA. Deck, M., & Whittaker, J. A. (1997, May 5-7). Lessons Learned from Fifteen Years of Cleanroom Testing. Paper presented at the Software Testing, Analysis, and Review (STAR), San Jose, CA. Foreman, J. (1997, October 27). Cleanroom Software Engineering: Software Technology Review. Carnegie Mellon Software Engineering Institute. Retrieved March 24, 2001, from the World Wide Web: http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html. Glass, R. L. (1999, February). The Realities of Software Technology Payoffs. Communications of the ACM, 42(2), 74-79. Grillo, P., & Tewari, R. (1997). Implications of Using Cleanroom Software Engineering to Develop Highly Reliable Three-Tier Client/Server Applications. Baylor University's Hankamer School of Business. Retrieved March 10, 2001, from the World Wide Web: http://hsb.baylor.edu/ramsower/ais.ac.97/papers/grillo.htm.

Head, G. E. (1994, June 1). Six-sigma Software Using Cleanroom Software Engineering Techniques. Electric Library: Hewlett-Packard Journal. Retrieved March 11, 2001, from the World Wide Web:http://wwws.elibrary.com/. Henderson, J. (1995, May). Why Isn't Cleanroom the Universal Software Development Methodology? U.S. Air Force's Software Technology Support Center. Retrieved March 11, 2001, from the World Wide Web: http://stsc.hill.af.mil/crosstalk/1995/may/cleanroo.asp. Kelly, D. P., & Oshana, R. S. (1996, November). Integrating Cleanroom Software Methods Into an SEI Level 4-5 Program. U.S. Air Force's Software Technology Support Center. Retrieved March 11, 2001, from the World Wide Web: http://stsc.hill.af.mil/CrossTalk/1996/nov/xt96d11f.asp. Morris, J. H. (2000, April). A High Dependability Computing Consortium. Carnegie Mellon School of Computer Science. Retrieved March 11, 2001, from the World Wide Web:http://www.cs.cmu.edu/~jhm/hdcc.htm. Oshana, R. S. (1998, November-December). Tailoring Cleanroom for Industrial Use. IEEE Software. Retrieved March 26, 2001, from the World Wide Web:http://computer.org/software/so1998/s6046abs.htm. Oshana, R. S., & Linger, R. C. (1999). Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project. IEEE Online: 32nd Hawaii International Conference on System Sciences. Retrieved March 26, 2001, from the World Wide Web: http://computer.org/proceedings/hicss/0001//00017/00017042abs.htm. Stavely, A. M. (1999, March 22-24). High-Quality Software Through Semiformal Specification and Verification. Paper presented at the 12th Conference on Software Engineering Education and Training, New Orleans, Louisiana.

Appendix A CSE Model

Figure 1. CSE Model (Oshana & Linger, 1999)