Rework: models and metrics

An Experience Report at Thales Airborne Systems

Edmond TONNELLIER +33 (0) 1 34 81 99 54 edmond.tonnellier@fr.thalesgroup.com 2 Avenue Gay Lussac, 78851 Elancourt Cedex, France Olivier TERRIEN +33 (0) 1 34 81 75 21 olivier.terrien@fr.thalesgroup.com 2 Avenue Gay Lussac, 78851 Elancourt Cedex, France
Copyright 2012 by Edmond TONNELLIER and Olivier TERRIEN. Published and used by INCOSE with permission.

Abstract. This experience report illustrates the implementation of metrics about rework in Systems Engineering at Thales Airborne Systems. It explains how models and quantification of rework contribute to reduce this waste in development processes. This paper is based upon our real experiences. Therefore, it could be considered as an industrial contribution to share this method in Systems / Products Development and Engineering. This experience report describes an achieved example of metrics and its positive impacts on Defense & Aeronautics Systems. However, this document proposes a scientific way to implement our results in other companies. Clues are also given to apply this methodology through a Lean Engineering approach. Finally, Lean Enablers, key factors of success and upcoming steps are described to further improve the performance of Systems Engineering.

Figure 1. The INCOSE Symposium logo

1. Context
Thales Airborne Systems is a company within the Thales group.

Figure 2. Thales Group key figures for 2010 Thales Airborne Systems is currently the European leader and the third player worldwide on the market of airborne and naval defense equipment and systems. The company designs, develops and produces solutions at the cutting edge of technology for platforms as varied as fighter aircrafts, drones, surveillance aircrafts as well as ships and submarines. About 20 % of the turnover is dedicated to R&D activities with a large proportion of it for systems engineering. With facilities in many European countries, the company employs numerous highly-qualified experts to design the solutions matching increasingly complex customer requirements as closely as possible. Benefiting from traditional positions as leader in the electronic intelligence and surveillance systems, recognized know-how in radar for fighter aircraft and electronic warfare, Thales Airborne Systems involves its customers from the solution definition phase up to the operational validation of the products. This requires high reactivity to changes in specifications and an ongoing effort to reduce development cycles. In addition, international competition and the effect of the euro/dollar conversion result in major budget constraints (in recurring and non-recurring costs). The complexity of the systems developed within Thales is due to the number of components involved, the numerous technologies used and the volume of the embedded software. Due to the systematic combination of technical and non-technical constraints in new contracts, the technical teams must synchronize more and more precisely, skills from complementary disciplines whose contributors are often based in several countries. Lastly, the accelerated pace of the developments requires detailed definitions as well as optimum reactivity to the events or defects encountered, inevitable consequences of faster changes in requirements and markets. In this context, in 2010 Thales Airborne Systems launched an initiative on models and metrics of rework in its development processes. Resolutely practical and pragmatic, this approach has observed, assessed, limited and then reduced the impact and frequency of the phenomenon in the engineering processes, in particular systems. This document describes the theoretical approach of modelling and quantification as well as the illustration of its deployment within technical teams.

2. Rework Problem
This chapter evokes origins and scope of the initiative about rework led by the authors of this present document.

2.1 Surveys & Benchmarks

As literature on Rework mainly comes from Production (in particular in Lean Manufacturing [1]) and Software Development (for example in Agile Methods [2]), we conducted several benchmarks with R&T contributors of other aeronautical or electronic companies. This first step directed our study on Rework in development processes. For most of the improvement initiatives studied, the starting point for rework initiatives is the sudden awareness of a major shift in costs and delays on one or more projects. However, diversity in consequences and multitude of causes for these shifts lead to confusion on this waste and misunderstandings on notions it covers. Hence our need to specify terms used and to illustrate this waste using a methodology applied to our fields of activity

2.2 Diagnosis
Our numerous interviews conducted with technical leaders and with project managers allowed us to draw up a diagnosis on a phenomenon that is always too important. Here are some reports below: Similarity in effects of this phenomenon, irrespective of the discipline involved (hardware, software and system engineering); Disparity between causes of this phenomenon depending on the entities involved (influence of organizations, difference in profiles, etc) and depending on the disciplines concerned (differences in the life cycles according to the constituents of a complex system); Existence of a visible/ observable part and a hidden / latent part; Collective as well as individual contributions to this phenomenon.

how can we explain this cost shift on our project X?, why has our schedule Y been delayed?, what caused this extra cost at the end of our project Z?. (Extracts of external benchmarks to illustrate the occurrence of rework)

2.3 Stakes
To manage our initiative efficiently, we have determined the in and out of scope of our approach. Here are some details below: Define a behavioral model of a general phenomenon despite local specificities; Define a method to assess the scale of the phenomenon (relative assessment, if possible absolute); Separate the technical causes (usually related to the disciplines and the products) from the non-technical causes (usually related to the organizations, profits of the players, human behaviors); Involve all players to reduce/limit the phenomenon.

2.4 Definition of Rework

The definition of Rework formulated in the work of P. Crosby, Quality is Free [3], answered our diagnosis. Our initiative retained:

Rework: work done to correct defects Defect: failure to conform to requirements

(even if this requirement has not been explicitly specified). These definitions are compatible with usual notions of rework applied in Production. Typical wastes in R&T are identified in our Lean Engineering approach [4].

Examples: Incomplete or misinterpreted requirements at the start of a project resulting in rework in cascade through to subcontractors, Late changes in requirements cause high levels of rework throughout the life cycle of products, Low defined designs result in expensive reworks to meet the customers' true requirement. (Extracts of interviews to illustrate the rework in development of systems)

Counter-examples: 1) Multiple reconfigurations of a test bench due to numerous changes of the type of products tested create additional work. This is not due to a defect but to work organization. ('task-switching' type waste) 2) A new manager forces his team to use a control system more like the one applied in his previous jobs. The extra work involved in updating is not due to a defect but to a change of method or to relearning team work. ('hands-off' type waste). (Extracts of interviews to illustrate of 'false friends' of rework) The rest of the present document summarizes the scientific approach which we followed to answer the problem of the rework within the scope of the previous definition.

Applied to Rework waste elimination, our scientific approach has addressed several specific Lean Enablers for Systems Engineering (LEfSE): Develop a system making imperfections and delays visible to all. Use and communicate failures as opportunities for learning emphasizing process and not people problems. Create mechanisms to capture, communicate, and apply experience-generate learning. Use quick response small Kaizen teams comprised of problem stakeholders for local problems and development of standards.

3. Rework model
This chapter formalizes the behavior of this rework phenomenon: it describes a model and confronts it with observations described in our presented diagnosis.

3.1 Behavioural description

The prefix of the word 're-work' indicates a looped phenomenon. When developing solutions, the relation between 'work' and 'rework' plays a central role in the generation of delays, extra costs and risks introduced in cascade in process downstream activities.

Figure 3. Rework is a looped phenomenon Our model proposed, a loop associating work and rework, corresponds to the defect correction process (from discovery of the problem to its complete and definitive resolution [5]).

3.2 Defect correction process

This behavioral model of rework makes the measurable part of this phenomenon visible: additional costs associated with this particular waste. Without corrections for a found defect, rework would be null and this defect would cost nothing (apart from dissatisfaction). However, the need to bring products into conformity requires correction and therefore generates an extra cost so that the solution complies with requirements.

Figure 4. Rework is an accumulation of impacts To a first approximation, the measurable part of rework corresponds to the accumulation of all impacts of a defect on costs and/or delays of a project (impacts made visible by each step of the defect correction process).

3.3 Deduction of rework

The rework generated by a defect results from the second journey of all phases in a development process. This journey is defined during a causal analysis led at the discovery of a defect.

Figure 5. Rework is looped journey Therefore, the scale of the rework loop depends on the defect discovery phase and on the path to be reproduced to correct the cause (and not only to reduce the effect): The later the defect is detected, the larger the impact on delays and costs of a project: correction within a phase will generate an extra cost of 1 whereas correction of this defect between phases will have an impact of 10 or even 100 on the entire project. The faster the second journey, the lower the disturbance on projects: reactivity of correction processes represents a lever to limit the phenomenon.

3.4 Induction of rework

sub-systems partially validated, choice of immature technologies, solutions not tested are some of the defects heard during our interviews. Introduced throughout a development process, they behave as real rework inductors.

Figure 6. Rework is a risk to downstream activities The rework appears from then on as a risk more than a random phenomenon! (Counter) reactions are thus possible to reduce and/or limit its impact, its probability of occurrence or its non-detection. The sooner the defect is detected, the lower its consequences on the project results. In conclusion, the proposed model supports the fact: The more defined the engineering upstream, the lower the downstream rework.

4. Quantification of Rework
This chapter implements the model through a mathematical formulation.

4.1 Correction process translated into data

Modelling of rework led to the following observation: rework is the accumulation of all the impacts of each step in the defect correction process. Without corrective action, no extra cost therefore no rework. The proportion of rework compared with the total work can then be evaluated by observing the impacts of a defect, impacts resulting in additional costs and/or delays on a project.

Figure 7. Envelope of impacts is a mathematical function Each step of the correction process is associated with an extra cost or a delay. To a first approximation of the envelope of impacts, the correct process CP of a defect Di forms a mathematical function CP = f (Di).

4.2 Mathematical modelling

If additional costs recorded throughout the process of correction of the defect which generated it forms a function, their accumulation corresponds to a surface enveloped by the function CP. In mathematics, a surface leads to an integral:

Advantages: This mathematical model can be easily transpose into an IT code. The formula is an expectation for its implementation in any tool of quantification (any statistical tool, queries in databases, etc). Independent of the measurement units (costs, delays), it can be applied to any organization (only the correction process of an entity is related to the model). Lastly, the model measures CP(Di) and not directly Di. Therefore, it measures effects and not causes of the defect, which allows its use in all levels in a complex system (and in any engineering disciple). Nota: any technical team can map its own CP(Di, t) with classical Lean Engineering

tools (using Value Stream Mapping for example). User cases: To determine the rework of a particular phase of a V cycle means replacing the tstart and tstop in the integral by the dates of the studied phase: the cumulated data comes from a narrower but still homogeneous source or period. Moreover, the value units ( or $ but also hour, day or week) can be changed to correspond to the local organizations. Lastly, a change in the correction process of an entity does not disturb the model since it evaluates the scale of the phenomenon (i.e. extra cost) and not the way of generating it or resolving the detected problem. Limitations: The mathematical model possesses its own limits. If it imposes the availability of inputs (such as dates of events, origins of defects, etc), it does not guarantee their reliability. In addition, the model can only compare entities if their defects share a standardized classification. Lastly, since the model is based on a defect correction process, it does not take into account defects that are not recorded, amount of waste before defect discovery or dependencies between defects.

4.3 Data availability

In aeronautical companies as for other sectors, defects and their resolution need traceability. Defect management tools such as ClearQuest [6] facilitate the recording of events, key parameters or information relative to the process of correction of a defect.

Figure 8. Available and reliable data depend on records in databases Nota: If for every defect Di, the recordings insure the existence of data to quantify the rework, they do not insure their reliability, their accessibility or their compatibility. Therefore the confidence level of the calculated quantification depends on the quality of the sources. Interview of an American expert: In God, I trust. With rework, I need data!

In the literature, the rework is usually quantified by a formula y = f (x1, x2, etc). Our paper proposes a simpler formulation by introducing an integral y = g (x). This mathematical solution reduces the number of variables to be managed. Easy to implement,

this integral formula demonstrates that the rework is measurable!

5. Metrics of Rework
This chapter evokes two approaches to exploit the model and the quantification of the rework: a capitalization to prepare a new project and a permanent engagement of staff to improve project in progress.

5.1 Capitalization of a process

A virtuous circle must oppose the vicious circle of rework. During the preparation of a new project, a relevant source of information shows to be the analysis of the processes of correction of the defects of previous projects. Applied to the available recordings, the team exploits the presented model to assess the rework and to evaluate the performance of the current process of correction. Relevance of the correction process: Having determined the status of available data (reliability of the records, the maturity of information fields, etc.), the team guarantees the representativeness of the sample of data for a statistical analysis.

The model filters the monthly data, according to the dates of creation or closing of the correction processes.

Figure 9. Analysis of volume of defects per month (extract from project 5.A) Root causes of defects: After establishing groups of the most frequent or the most penalizing defects, the team defines categories and parameters which are the scales of rework in the ended projects: they are the inductors of rework.

Figure 10. Analysis of Root-causes (extract from project 5.B) During the new project, when a defect occurs, the team characterizes it and, thanks to the categories and parameters previously identified, determines the risk of potential rework for the project. Charged with costs and delays, the defects are included into the management of the project. They are the indicators of rework. The team assesses the possible disturbance due to new defects and defines its priorities according to its capacities of reaction. In conclusion, the project integrates the phenomenon of rework through its management decisions and allows the team to anticipate possible difficulties.

The model accumulates workloads associated with defects (consumed and estimated) to obtain usual S-shaped curves of project management.

Figure 11. Control indicator of corrected defects (extract from project 5.C)

5.2 Improvement of processes

To favor the engagement of staffs in the reduction of waste generated by defects, it is essential to associate them with improvement workshops launched locally. By using the proposed quantification, the team measures the influence of their improvements before/after the workshop. Process effectiveness: Databases (restricted to a homogeneous perimeter: one product type or a given project) allow any team to assess real performances of its correction process (CP). This team describes key steps of its implemented CP using process maps:

(before) Observations of the ground offer numerous levers of improvement (e.g. how dynamic is the sequence of process steps to solve a problem). A causal analysis identifies root causes of weaknesses in CP and suitable counter-measures are implemented by this local team. A regular check with a quantification tool sustains this change.

(after) Process efficiency Similarly, the model can be used to determine the efficiency of the development process by comparing the total workload and the proportion of rework (for example by comparing on a given project and for a given period of time the hours of work and the hours spent to correct defects). To make positive impacts visible involves the team: either by simulating new actions or by pursuing the improvement workshops.

The model compares periodically global workloads with workloads associated with rework. To make wastes visible involve teams.

Figure 12. Rework vs Work indicator (extract from project 5.D)

5.3 Improvement Workshops

Concrete Lean process improvement workshops have been completed since the beginning of this initiative. Team Managers, SE actors and development teams map their own process of defects and make imperfections visible to all. Therefore, problems are not hidden but seen as opportunities to accelerate the loop to correct defects. Everybody is empowered and team spirits are developed.

Figure 13. A3 format from an improvement workshop training session

6. Feedback 6.1 Contributions for system engineering

The model proposed in this document was deployed during local workshops. The teams involved gave feedback on several contributions of our initiative: Rework is a risk, not a random event! It is not fate! The model describes a behavior and provides keys to observe it. The rework is part of project management The mathematical model allows sharing between entities, avoiding the disadvantage of specific and/or local tools Measures of rework are directly performed by local teams; however, training can be carried out based on a global and common approach Dealing with conformity separates necessary iterations (usual part of the creative cycle) and negative spiral activities (waste such as rework).

6.2 Methodologies
About Systems methodologies (or close), here is some feedback : governance by facts and by use of existing databases a generic methodology of rework, common to every level of complex systems, applicable to any development in progress An engagement of technical teams through local workshops; an effect confirm by individual but also collective efforts an approach making waste visible to obtain concrete and efficient improvement levers; easy to implement

6.3 Deployments
The scientific approach described in the present document as well as the examples to deploy the model and the quantification of rework are applicable in other companies. Here are some clues: implementation via the personal initiative of a local manager convinced by this approach (a bottom-up workshop applied on defects of a local project) implementation via the major processes described in the Company Reference System (a top-down deployment on defect correction processes) implementation via a Lean Engineering approach (identification and elimination of waste in engineering processes [8]). implementation of specific Lean Enablers for Systems Engineering (LEfSE) [9]).

In conclusion, the model and metrics of rework proposed in this document can be applied to all the development processes necessary for Systems Engineering.

Rework is not bad luck but a risk to manage.

References
[1] WOMACK James P. & JONES David T., Lean thinking, 1996. [2] POPPENDIECK Tom & Mary, Lean Software Devpt, Agile toolkit, 2003. [3] CROSBY Philip B., Quality is Free, 1979. [4] TERRIEN Olivier, Incose 2010: The Lean Journey http://www.thalesgroup.com/Lean_Engineering_Incose2010.aspx [5] COOPER Kewell G. & MULLEN Thomas W., The Swords & Plowshoes. The Rework Cycles of Defense Software, 2007 [6] GILLES Harry, Thales, Problem & Change Report Management with ClearQuest Tool, 2007 [7] CROSSTALK, The journal of defense software engineering, V.16/3, 2003. [8] OPPENHEIM Bo, Incose 2010, WG Lean for SE http://cse.lmu.edu/about/graduateeducation/systemsengineering/INCOSE.htm [9] OPPENHEIM Bo, Lean for Systems Engineering; with Lean Enablers for Systems Engineering, 2011

Acknowledgements
To Harry Gilles, Roger Martin and Michel Baujard for their valuable support during this initiative.

Biography
Edmond TONNELLIER System Engineering Expert With many experiences dedicated to the development of complex products, Edmond Tonnellier is currently the System Engineering Expert for Thales Airborne Systems and contributes to the Thales Group System methodology (SysEM Handbook and Corporate Guide - Value Analysis). CMMI SEI DEV and ACQ evaluator, he conducts numerous audits inside and outside the Thales group. In addition to his role as SE reference, he contributes to numerous IS training courses at Thales University AFIS and INCOSE member (CSEP certified, BKCASE reader) Engineer (Post grad degree), 1977, ISEP, France. Olivier TERRIEN Lean Engineering Expert Olivier Terrien is a Thales Lean Expert and is the Thales Airborne System reference for Lean Engineering. He has implemented numerous process improvement workshops based on Lean Manufacturing and/or Lean Engineering approaches (in systems engineering, software development and customer commitment). His background is in the engineering processes (design of microwave components, development of electronic warfare receivers, integration of naval radar systems and airborne electronic warfare suites). He has published more than 350 pages in the worldwide Press. Engineer (Post grad degree), 1997, ESEO, France MBA, 2006, IAE Paris-La Sorbonne, France