The Shangri-La of ROI

Sarah A. Sheard and Christopher L. Miller Software Productivity Consortium 2214 Rock Hill Rd., Herndon, VA 20170 sheard@software.org; miller@software.org
Abstract. Many INCOSE members are dismayed that there are no hard numbers to justify implementation of systems engineering process improvement. This paper shows that: 1) There are no hard numbers. 2) There will be no hard numbers in the foreseeable future. 3) If there were hard numbers, there wouldnt be a way to apply them to your situation, and 4) if you did use such numbers, no one would believe you anyway. We have to get past fantasy and find out how to make the case for implementing systems engineering process improvement without these elusive hard numbers. This paper concludes with suggestions on how to build a case for process improvement that does not rely on hard numbers. INTRODUCTION When you introduce systems engineering into a traditionally commercial organization, you will have to justify your ideas on the basis of dollars and cents. (Jackson 1996) The questions. How can I convince my manager to fund systems engineering? How can I convince my company that improving systems engineering processes by working toward complying with the Systems Engineering Capability Maturity Model (SE-CMM) or the Electronic Industries Affiliates Interim Standard (EIA/IS) 731 is going to save them money? Where are the hard return-on-investment (ROI) data showing that for every dollar I invest in process improvement, I will get so many dollars back? Note: This paper considers the implementation of systems engineering to be the same as process improvement of the systems engineering process. The former is equivalent to achieving Level 1 in a process area in any of the systems engineering capability models. WHAT I WANT IS HARD NUMBERS What everyone would like to see is shown in Table 1. There are also many variations on the numbers desired. Many people want hard numbers, meaning numbers that are statistically proven and incontrovertible, that show the benefit of performing risk management. Some want to prove that completing requirements analysis Parameter Program cost savings for each dollar spent doing systems engineering Conditions All EIA/IS 731 technical focus areas at Level 1 Technical and management focus areas at Level 2 All focus areas at Level 3 Level 4/5 organization per (Sheard and Roedler 1999) Before SE assessment After SE assessment After 1 year of improvements amounting to $200 spent per engineer Guaranteed Level 2 Guaranteed Level 3 Guaranteed Level 4 Guaranteed Level 5 Number a:1 b:1 c:1 d:1 x 20% y 15% z 10%

Systems engineering productivity

Required processes to implement to achieve:

Ref.1* Ref.2 Ref.3 Ref.4

* Note, these are fictitious. Do not look at the references to this paper. Such processes do not exist. Table 1. Desired Systems Engineering ROI Data prior to starting design saves the program time and money, or show the reduction in rework that occurs when requirements and high-level design are reviewed by all affected stakeholders. In general, people want proof that improvements to do things right will return enough money to justify the investment. Where can we find such numbers? What kind of research should INCOSE do to obtain such numbers? Let us start by looking at what does exist and why it does not provide what we want. THERE ARE NO HARD NUMBERS Herbsleb report. In 1994, the Software Engineering Institute (SEI) released a report called Benefits of CMM-Based Process Improvement (Herbsleb et al. 1994). In this report, information is collected from thirteen companies who published results from process improvement using the CMM for Software (SW-

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

CMMi) (Paulk 1993). The paper shows returns on investment ranging from 4:1 to 8.8:1 (See Table 2). This means that for every dollar invested in CMMbased process improvement, between $4 and $8.80 was received in productivity gain or other savings. This is very interesting data, and the authors show it whenever we teach a capability model class. But we bracket it in caveats as follows. Applicability. First, this relates to the SW-CMM. Can it apply also to systems engineering process improvement? While there are no specific reasons why not, there is a leap of faith required to believe that it does. Time. Although hundreds to thousands of companies have used the SW-CMM since 1994, updates to this paper have not been released that include more companies or more thorough data. This situation is worrisome because, in the five years since this paper, much has changed in the software development industry, and it is reasonable to believe that studies done now might show different numbers. For example, many companies have already made improvements comparable to those discussed in the Herbsleb report. Is it not possible that early efforts represented low hanging fruit, and additional process improvement would yield diminishing returns? Common Definitions.. To determine return on investment, both return values and investment values must be collected. Unfortunately a common definition of return values has not been accepted across industry (Brodman and Johnson 1996). Therefore, the meaning of any published return-on-investment numbers is inherently unclear. What was counted as return? Cost savings are typically included, but what about softer values, like Category Yearly cost of software process improvement (SPI) activities Years engaged in SPI Cost of SPI per software engineer Productivity gain per year Early detection gain per year (defects discovered pre-test) Yearly reduction in time to market Yearly reduction in post-release defects Business value of investment in SPI (value returned on each $ invested) Range $49K $1,202K 1-9 $490 - $2,004 9% - 67% 6% - 25% 15% - 23% 10% - 94% 4.0 - 8.8

improved morale, reduction in cycle time, and improved predictability? Return parameters are not clearly or consistently defined across the studies that have been publicized Sample size. Certainly, thirteen companies cannot be representative of the thousands performing process improvement, even if the survey were current. Any study that is truly predictive must use random sampling of far more than thirteen companies. Bias. Sampling bias is also inherent in this kind of study. Remember, these were actual companies that published their results in the open literature. Let us do a thought experiment here. Thought experiment #1. Suppose an executive at your company is excited about the promise of process improvement based on the SW-CMM. Suppose the executive is intelligent and implements real improvements that meet the needs of the company. He or she assigns a talented and respected project manager to run the process improvement project, procures knowledge from experienced outsiders as necessary, documents processes that are actually done using up-to-date and efficient tools, and obtains cooperation and buy-in from all affected programs. Fast forward one year A million dollars has been invested, and the company has reached the next level of process maturity. Now suppose the executive leaves the company. Maybe he or she was offered a better job in another company that wanted to duplicate process improvement success, or perhaps the company was acquired and the executive was offered a severance package. Suppose the executives replacement is distinctly not a process improvement person. Instead the new executive is a very product- and bottom-line-focused person. To save costs, he or she eliminates all funding for overhead such as process improvement and incentivizes program managers to think tactically and not strategically. Essentially, the CMM-based process improvement effort has been nullified, and the return on the million-dollar investment is approximately zero. Would your company publish this result in the open literature?

Readers will recognize that nothing in this thought experiment is unreasonable. Numbers published in the open literature are, at best, examples of what could happen in the right circumstances and are not predictive in any sort of general case. THERE ARENT GOING TO BE ANY HARD NUMBERS Data such as productivity, cost to produce systems, process improvement cost, and the like is usually con-

Table 2. Herbsleb report results

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

sidered highly confidential. Everyone would be more than happy to receive reports on industry trends and data on all the other companies, but no one is willing to provide such data. Even if the data was clearly defined, additional confidential data would be required to verify that the data had the same meaning from company to company. It is hard to imagine ever prying such numbers out of a large number of companies in a manner that has any sort of fidelity. Therefore, it may be assumed that when any numbers are made public, the publicity is likely to result from motives other than sharing. Perhaps the company feels that it is ahead of the industry and thus releasing such numbers has marketing value. Other reasons might make one doubt the truth of numbers, but even we are different from the rest of the industry makes one doubt their applicability to the general case. Systems engineering startup. What kind of numbers would help convince program managers to start up a systems engineering effort within a program? The numbers in Table 3 would do. How would you expect to be able to calculate the cost savings? According to (McKinney 1995), systems engineers have been known to estimate risk avoidance at costs exceeding the entire program development cost by several times! Would anyone believe any estimates for these numbers? Definition of systems engineering. Table 3 also assumes that any two people will agree what is meant by systems engineering activities. Is systems engineering the creation of software requirements specifications, or is there more? Does it include attending customer technical exchange meetings and subsystem area program reviews, or does your company consider those activities to be project management? Does systems engineering include development of system test procedures, or is system test considered a separate activity? Category Yearly cost of systems engineering activities Cost of rework avoided because systems engineering was done Costs of risks avoided due to systems engineering Costs saved because systems engineering came up with better designs Other cost savings attributed to systems engineering Return on investment of systems engineering Dollars $x $y $z $q $r $(y+z+q+r)/$x

(Sheard 1996a) and (Sheard 2000a) show that definitions vary greatly within INCOSE, from requirements definition and system design to analysis, technical management, coordination, or even configuration management. Definitions vary even within a single company certainly you cannot expect someone elses numbers to have much meaning for your organization. IF THERE WERE HARD NUMBERS, YOU WOULDNT BE ABLE TO USE THEM Comparison to surveyed companies. For a moment, let us suppose that hard ROI numbers do exist for performing process improvement according to any capability model you want, including systems engineering models. Now we must ask whether those numbers can actually be used as desired. Those who ask for hard ROI numbers want to make a case that process improvement is worth the investment. They want analytical models to tell them how much benefit they will receive from their investment. Any estimation model must be examined to determine whether the numbers can be replicated in a specific situation. Would you be able to find the answers to these questions? How much waste was in the baseline process of the companies surveyed? How much waste is in your baseline process? How much of their way of doing business depended on smart people and how much does yours? Are your people just as smart? Smarter? What did the surveyed companies consider productivity to be, and how does it compare to your definition? How are overtime, overhead, and G&A costs considered in the definition of systems engineering costs? How did the surveyed companies calculate return on investment related to a decrease in time-tomarket? Would such calculations apply as well in your marketplace? Do they follow procurement rules similar to yours, or do you have constraints that would keep you from achieving the same improvements?

Table 3. Systems Engineering ROI

Separability of causes. In addition, few companies perform process improvement in a vacuum. Most companies try multiple ways of remaining competitive. Strategies include tweaking executive compensation, making changes in response to financial monitoring and control of operations, implementing quality initiatives, adjusting employee compensation and benefits to obtain the right workforce, entering new markets, and changing bid and proposal strategies or expenditures. These

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

initiatives tend to occur simultaneously, because companies cannot afford to spend a year tweaking each variable to determine its effect on the company bottom line. The environment does not stay constant that long! Therefore it is difficult to attribute improved bottom line numbers to one cause or the other. Suppose your time-to-market metric decreases. Is this because you implemented improvements leading to Level 3 in the SW-CMM? Or can it be attributed to buying a combined system and software modeling toolset? Or maybe you are now making products that are much closer in content to other products and thus new versions can come out faster? Perhaps the fraction of your company that is building web-based applications is larger, and the fraction serving military customers is smaller. How do you allocate dollar savings among these reasons? And how do you count the cost value of reducing cycle time, anyway? In addition, there are two questions: one on the value of systems engineering at all, and the other on the return on investment of process improvement, including systems engineering process improvement. How could anyone separate these two activities to produce an ROI number for both implementing process improvement and the value of systems engineering? For this reason, this paper assumes systems engineering and process improvement are inseparable when it comes to calculating ROI. Thought experiment #2. For extra insight into the problem of separability of causes, imagine that your companys productivity numbers actually decreased after a year of process improvement. See how many reasons your process improvement sponsor would come up with for why the process improvement initiative actually made positive changes but other causes combined to make it seem otherwise. Now consider how correct it is to attribute ROI to the process improvement initiative. Estimation model nonlinearity. The purpose of having hard numbers is to estimate what results investment might bring. These hard numbers then constitute an estimation model, if you will. The concept of ROI used in this way assumes linearity: invest $100,000 and get $400,000 back, invest $100 and get $400 back. Most engineers recognize the limitations of linear models. (Sheard 1999) estimates a full systems engineering assessment as taking 670 hours, which would translate to over $30,000 for just the time of internal people.1 The cost of external assessors could double that number; remember, that is $60,000 spent just to take the test. No actual improvement has yet been funded! Since your

Trait Use of SE Quality Relative cycle time Requirements-RFP Relative cycle time Design-Production Test time

Program 1 Low Low 2.5 n/a 16

Program 2 Med Adequate 1 3 n/a

Program 3 High High 1 2 10

Table 4. Frantz results first $60,000 has so far yielded nothing in return, your next dollars need to be very well placed in improvements of clear benefit. IF YOU DID USE HARD NUMBERS, NOBODY WOULD BELIEVE YOU ANYWAY How process improvement is done. Thought Experiment #1 provided a hypothetical case in which process improvement was done right (until it was cancelled). In many instances, process improvement is forced from the top down by executives who had no idea what process improvement means or what it will take. These executives may have decided that a certificate stating Level 3 or ISO 9000 registration would be good for business, and took whatever shortcuts they could in order to obtain the desired certificate. The chances of the companies receiving good return on investment from productivity, cycle time, or productivity improvement, though not zero, are quite small, and it is highly unlikely that any such return would be sustained over time. Executives who have a history of taking shortcuts like this should be very suspicious of data promising return on investment, because they have shot magic silver bullets in the past in search of such returns, and the promises were never fulfilled. Frantz paper. A different approach was taken by W. Forrest Frantz at Boeing (Frantz 1995). The paper analyzed three programs with different levels of systems engineering to determine how well the programs were executed. The results are shown in Table 4. Although the paper concentrated on cycle-time reduction, this was due to improved quality of systems engineering work products and reduction of rework during the test phase. The Frantz paper may be the closest the industry will get to hard numbers on the value of systems engineering. The analysis compared three programs that ran in parallel in the same company, using approximately the same definition of systems engineering, with the same type of system. Yet the numbers are almost never quoted, and it is widely stated that hard numbers on the value of systems engineering do not exist. Why is this? It is because no data, no matter how good, will ever

Assuming salaries of $100,000 a year, including burden.

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

be good enough. There will always be both rational and emotional reasons to decide the data does not apply. Rational reasons include the data consistency issues mentioned above. In addition, almost any executive under extreme cost pressure will not believe that spending more money is good, no matter what numbers are presented. (Davenport 1998) describes a case where Mobil Oil engineers devised a sophisticated technique for reducing the amount of steam required for most drilling operations. Hard numbers proved that the technique saved significant money. Nevertheless, even with consultants instituting an intensive communications process, including lessons-learned videos, only 30% of the drilling sites were willing to implement the improvement. And these results were for an improvement that had hard numbers showing clear cost savings and insignificant costs to implement! Imagine the difficulty in making people change for reasons that are less clear. GIVE IT UP It should be clear by now that we should stop wishing hard numbers existed. We should also stop waiting for them in order to make the case within our organization that systems engineering process improvement would be beneficial. Feel free to use the arguments in this paper to prove that a) hard numbers are not available, b) hard numbers will not become available any time in the near future, c) if there were hard numbers you wouldnt be able to apply them to your company, and d) if you did use them, nobody would believe you anyway. SO NOW WHAT? Clearly, the case for process improvement and systems engineering has been made successfully in many instances, because many companies have decided to implement one or both. To the authors knowledge, none of these decisions has been based on hard numbers. In all cases, the executives made a leap of faith that systems engineering process improvement would pay off. What convinced many was a belief that return would follow the investment. This expected return might be monetary savings, decreased time to market, or an increased degree of manageability and predictability. Several ways to help convince managers of expected return when there are no believable hard numbers are justification in retrospect, anecdotes, peer pressure, and discussions about the value of systems engineering. Justification in retrospect using internal measurements. A company that knows its costs because it measures them can make decisions based on meaningful internal hard numbers. Can your company show how much money was spent on systems engineering tasks

last year, and which ones? If you can get a baseline, and then perform some improvements with investment funds based on faith, at least those funds can be justified in retrospect when you show improvements in the measurements. In answer to thought experiment #2, you need to be sure that the numbers you are measuring are atomic enough to show the cost reality even in the face of competing initiativeswhereas systems engineering productivity may be easy to manipulate, the labor hours required to complete a change request are more concrete. Anecdotes. Anecdotes are a valuable way to transfer knowledge effectively. Read the literature, including INCOSE proceedings and INSIGHT of course, but also software process improvement literature including the free U.S. Defense Software magazine, Crosstalk (STSC 2000). Accept anecdotes as maybe all there is. Talk to people who have participated in process improvement initiatives and ask if they are willing to share their stories with you. Ask consultants to share what they know, but be sure to put on your whats in it for them? filter to prevent your company from looking like a nail just because they have a hammer. Peer pressure. Everybody else may be onto something. Realize that the only way to remain competitive in a fast-moving society is to keep improving. Tie the need for process improvement to bids you have lost recently. Find out from the SEI how many of your competitors have achieved Level 3 (or other levels).2 Suggest to your management that competitors will eat your lunch if you dont do achieve the same levels.3 Value of systems engineering. With respect to funding improved systems engineering, first define exactly what you would like to fund. Is it a group, a phase, more time in requirements, more engineers and fewer marketers writing proposals, time for people building different product lines to talk about sharing, IR&D for developing modular products? Use (Sheard 1996a and 1996b) and (Sheard 2000) to help you decide what your com2

No comparable collection agency currently exists for collecting systems engineering assessment results. Even if there were such an agency, the nature of continuous-view capability models makes comparison difficult. Would you list the number of companies that have achieved Level 2 in all process areas? 80% of process areas? An average of Level 2? And would you list SE-CMM results differently from SECAM or EIA/IS 731 results? 3 Despair not. While this will not get you commitment to process improvement (it will only get you commitment to getting the certificate), it has happened in a few cases that in the process of attempting to get a certificate, a company sees the light and actually decides that this bitter pill has some beneficial effects. Its worth a try, at any rate.

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

pany considers systems engineering to be, and determine how it will benefit your company. Then describe how those kinds of systems engineering can contribute value in your organization. If at all possible try to quantify this value. At a minimum, track the costs of systems engineering activities. Insight into the total effort spent on systems engineering activities in relationship to program performance may provide enough evidence to make your case. Probably the most overlooked opportunity for building a case for systems engineering process improvement is identifying existing systems engineering activities within the organization. Many organizations assume that if they have not instituted a formal systems engineering process improvement effort, then they are not doing systems engineering. You need to determine if you are already performing activities that should be considered systems engineering process improvement tasks, and build your case from these based on the following three situations. Not called systems engineering. Some systems engineering is done because you are already under contract to do it. (Or, per (Jackson 1996), you are doing it because youre ISO 9001 registered, or because it is what you need to do to get product out.) For example, creating deliverable documents such as operational concepts, users manuals, or specifications is often designated a systems engineering task. If you are not calling this systems engineering, just start doing so. Then convince people that an entire professional organization exists to help you do this better. Program management. There are other systems engineering activities that you are also doing already, but you call them program management (or project management) (Kasser 1996). You have the choice of calling it whatever you like. In either case, both the Project Management Institute and INCOSE have help for you on how to do these activities better. Better program management helps your company reduce cost and schedule overruns and avoid risks. Not being done. Some systems engineering activities are not getting done. You know you need to do these activities, either instinctually because you have seen problems downstream, or because you learned it from INCOSE or elsewhere. Under-performed activities may include taking time to understand customer needs, identifying and mitigating risks, specifying requirements clearly and rigorously, or coordinating disciplines. Do not insist on calling these activities systems engineering. Bill them as reducing rework or increasing reusability of components or ensuring customer satisfaction. In this way, you avoid triggering any emotional reactions that systems engineering is too expensive. Later, you can point out that all the good work you did in these areas is called systems engineer-

ing by some, and youd like to improve it more using a systems engineering model (and be sure to add) as tailored to your particular situation (Sheard, Lykins, and Armstrong, 1999). REASONS FOR INVESTMENT Much of the decision to implement process improvement, or any other program, ultimately rests on faith in the minds of executives. What does the executive believe is the best way to a) grow the business and b) maintain stockholder value? Although analysis can show what return on this investment ought to be, it always involves considerable assumptions. Whether the executive believes the assumptions determines whether the investment will be made. Change advocates will be expected to make a case to help the executive believe the assumptions. Reasons that have been used in the past are grouped in the following categories: External reasons Resource problems Disconnects Vicious cycles In all cases, defining and improving systems engineering processes can provide insight to help answer the problems described below. External reasons. These reasons relate both to winning bids and to performing on contracts. Our competitors have announced a program to reduce their development costs by 20%. If we dont do the same, well never get another contract. Our customer complains about problems with the delivered product. If we dont improve, and fast, well never win another contract from them. This proposal requires us to show we have mature systems engineering processes. What processes do we have? Whos in charge of processes around here? What can we write? Were tired of playing Guess the requirements with the customer. Well build a prototype that meets spec exactly, and theyll run it past their users and its completely wrong. Why cant they get their requirements right the first time? Resource problems. These reasons relate to getting the right people (and other resources) onto programs. In this job market we cant hire people fast enough to staff all our programs. We have no choice but to become more efficient. We find problems getting larger and more complex, and staffing is more stretched every year. We have to do more with less. Software has repeatedly complained that poor requirements are their biggest problem. But our systems

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

engineers dont know enough about software to do it right the first time, and the software people dont want to deal with anything thats physical or mechanical, or in any way indeterminate. They just want the answer and theyll code from there. Disconnects. When processes are broken, these breakages show up in the form of disconnects. Our customer talked to a developer and he agreed to put in a feature they wanted. But the requirements and test people knew nothing about it! This has got to stop! When I ask people for labor estimates for a proposal, they all come in different formats, the mechanical people include system test and no one else does, the software people estimate low and the analysts estimate high, and I have to apply all sorts of fudge factors to get what I think is a good total number. And even so, it ends up being way off sometimes. This change request is full of errors, yet everyone signed off on it! How can that be? It looks like both the Radar area and the Test area have requested $200K capital expenses for the same test equipment. Dont these people talk to one another? Im tired of frustration, tired of running around without getting anywhere, tired of nobody taking responsibility, tired of things being late and not what I wanted anyway. Vicious cycles. Vicious cycles occur when incentives to continue doing things the current way are strong enough that they prevent sensible change. Management must intervene to reduce the barriers to needed changes. We find a lot of problems in test, when its really expensive to correct them. I sure wish we had found them earlier. But who has time? We have to reserve so much time for test and we have to meet the customers schedule, that we have to start building before we have all the requirements or we would never meet schedule. We have new challenges every year. This year we have a teaming arrangement with multiple contractors. Last year we were bought out and had to figure out how the acquirer did business and try to convince them our way was better (we werent very successful). The year before that, we had to learn how to deal with the customer on integrated product teams, where they saw all our dirty laundry. The year before that, we had to learn how to do evolutionary development. How are we supposed to get any engineering done with all these politics? We know our processes are inefficient. We know it takes too long to get signoff on a change request or close out a failure report. But were all swampedwho has the time to jump over hurdles in the hope that they can get anyone to listen? We need more predictable costs to have half a

chance at pushing back on marketing, who always cuts our development funding and time way below what we can handle. And then they wonder why were late and over budget. GETTING STARTED How much is it worth to an executive to solve these problems? According to (Garcia 1996), change is best marketed as a solution to problems: no problem, no investment. The identified problems also need to be those that someone with funding assumes is his or her responsibility to fix. Phrase problems and opportunities in terms of costs. How much operating cost is wasted when system test takes six weeks when it could take one or two weeks? Can you estimate the cost that can be saved and talk to the person with that budget responsibility? What schedule problems arise when you cannot staff programs? Can you quantify the cost of these slips? When a systems engineering problem becomes apparent to an executive, the first reaction is sometimes to issue an order that it shall be solved. But the obvious solution is not necessarily the right one. A hotel found that getting room service meals to rooms before the food got cold required buying more towels!4 Try to convince the executive that a) determining the correct solution (the best way to estimate, for example) may take some time, and will require input from a number of people and b) obtaining compliance will be easier if the individuals responsible for following the process are involved in developing it. (Show that previous orders were followed initially but that people went back to the old way after the pressure was off.) If you know you need to improve your systems engineering processes, consider hiring a consultant to perform an appraisal or mini-appraisal. Initially at least,5 the purpose of an appraisal is to tell reveal the best places for improvement, according to the people performing the processes. An executive who invests in a formal appraisal is more likely to provide money to address the identified opportunities for improvement. Begin by implementing process improvement in the small, such as on a program, and measure initial and resulting costs. Plan to have data to presentimagine your program manager showing executives how many
4

Measurements on room service delivery times isolated waiting for the freight elevator as the biggest delay. The freight elevator was overworked because the service staff was frequently going from floor to floor to find enough clean towels to make up rooms. 5 Once an appraisal has given the organization a set of ratings, subsequent appraisals tend to acquire the purpose of demonstrating that progress has been made. Still, they too produce findings on the next set of steps to take.

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000

engineering staff they needed before and after processes were improved. Measurements produce convincing data that will justify continued investment in improvement. CONCLUSIONS A case must be made for implementing systems engineering or other types of process improvement without having magical data that proves such activities will guarantee return on investment. The suggestions in this paper are a good starting point, but you must do the work to demonstrate the benefits of the process improvement approach that you are recommending. A meaningful ROI can be computed only after the resources have been committed and systems engineering process improvement projects have been completed within your own organization using your own definitions of return and investment. Then, when the benefits of process improvement are realized by management, process improvement sells itself without reliance on ROI numbers. REFERENCES Brodman, Judith G. and Donna L. Johnson, Return on Investment from Software Process Improvement as Measured by U.S. Industry. Crosstalk, April 1996. Davenport, Thomas, Larry Prusak, etc. Working Knowledge, Harvard Business Press, Boston, 1998. Frantz, W. Forrest. The Impact of Systems Engineering on Quality and Schedule, Empirical Evidence. Proceedings of INCOSE, 1995. Garcia, Suzanne M. How to Survive as a Change Agent in Hostile Territory: Principles of Process Improvement Terrorism. Proceedings of INCOSE, 1996. Herbsleb, J., A. Carleton, J. Rozum, and D. Zubrow. Benefits of CMM-Based Software Process Improvement: Initial Results, CMU/SEI-94-TR-13, ESC-TR-94-013. Software Engineering Institute, 1994. Jackson, Scott. Introducing systems engineering into a traditionally commercial organization. Proceedings of INCOSE, 1996. Kasser, Joe. Systems Engineering: Myth or Reality? Proceedings of INCOSE, 1996. McKinney, Dorothy. The Value That Systems Engineering Can And Should Provide, INSIGHT, INCOSE, Dec. 1995. M. C. Paulk, B. Curtis, M. B. Chrissis, and C. V. Weber, Capability Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-24, February 1993. Sheard, Sarah A. Twelve Systems Engineering Roles, Proceedings of INCOSE, 1996a. Sheard, Sarah A. The Value of Twelve Systems Engi-

neering Roles, Proceedings of INCOSE, 1996b. Sheard, Sarah A., Howard Lykins, and James R. Armstrong. Overcoming Barriers to Systems Engineering Process Improvement, Proceedings of INCOSE, 1999. Sheard, Sarah A. and Garry J. Roedler, Interpreting Continuous-View Capability Models for Higher Levels of Maturity, Systems Engineering vol. 2 No. 1, April 1999. Sheard, Sarah A. Fifty Ways to Save Your Budget, Proceedings of INCOSE, 1999. Sheard, Sarah A. Types of Systems Engineering Implementations, Proceedings of INCOSE, 2000. STSC (Software Technology Support Center), Crosstalk. 2000. Available at http://www.stsc.hill .af.mil/crosstalk/crostalk.asp BIOGRAPHIES Sarah A. Sheard has twenty years experience in systems engineering of hardware and software systems. Currently she is a senior systems engineer and the deputy manager of the systems engineering department at the Software Productivity Consortium. Ms. Sheard chairs INCOSEs Measurement Technical Committee, which encompasses the Measurement and the Capability Assessment working groups. She has participated in the Measurement Working group and is active on the Practical Systems Measurement project. Ms. Sheard also serves as an associate editor of Systems Engineering. Christopher L. Miller is a systems engineer at the Software Productivity Consortium. Mr. Miller has intensive experience in measurement, and process improvement of software and systems engineering processes. He is active in the INCOSE Measurement Working Group and Practical Software and Systems Measurement Technical Working Group. Mr. Miller has a Masters of Engineering Management in Systems Engineering at the George Washington University and is a member of their adjunct faculty.

CMMis a registered trademark of Carnegie Mellon University.

Copyright 2000 Software Productivity Consortium NFP, Inc.

Originally published in Proceedings of the Tenth International Symposium of the International Council on Systems Engineering, Brighton, England, June 2000