Master Thesis

Master Thesis Name: Student Number: Thesis Number: In fulfillment of:
Jaap Kabbedijk 0444332 INF/SCR-09-65 Master of Business Informatics
This thesis research is performed from 07/09/09 until 07/01/10 at Lund University. After this period the thesis research is continued at Utrecht University, resulting in a defense presentation at 15/06/10.
Requirements Engineering Decision Making:

Relationships among Decision Characteristics
Abstract
Time efficiency is crucial for decision making in large scale Market Driven Requirements Engineering. In order to identify what factors influence the decision lead time and outcome, we conducted a retrospective case study at a large product software manufacturer and analyzed seven possible relationships among decision characteristics. A large requirements engineering decision log was used to statistically test all hypotheses and the results were cross-validated using a survey among similar software manufacturers. The validated results show that the number of products affected by a decision has a positive relationship with the time needed to take a decision. Furthermore, a higher number of products implies a longer decision lead time. Results also show that when a change request originates from an important customer, the request is sooner accepted than changes requested internally. For efficient requirements management, our findings support that decision making activities can be carefully refined in large scale requirements engineering processes.
Requirements Engineer Decision Making Relationships among Decision Characteristics
Page | II
Table of Contents
1 2 Introduction .......................................................................................................................................... 1 Theoretical Background ........................................................................................................................ 3 2.1 2.2 2.3 3 Related Work ................................................................................................................................ 3 Requirements Engineering as a Decision Process......................................................................... 5 Research Positioning ..................................................................................................................... 5
Case Company Description ................................................................................................................... 8 3.1 3.2 3.3 3.4 3.5 Product Line Environment ............................................................................................................ 8 Requirements Management Process ............................................................................................ 8 Product Line Scoping................................................................................................................... 10 Release Management ................................................................................................................. 12 Market-Driven Nature................................................................................................................. 12
Research Questions ............................................................................................................................ 14 4.1 4.2 4.3 4.4 Possible Questions on Decision-Making ..................................................................................... 14 Main Question ............................................................................................................................ 16 Sub Questions ............................................................................................................................. 16 Relevance of Research Question ................................................................................................ 16 Academic Relevance ........................................................................................................... 16 Business Relevance ............................................................................................................. 17
4.4.1 4.4.2 5
Research Approach ............................................................................................................................. 18 5.1 5.2 Overview ..................................................................................................................................... 18 Case Study Design ....................................................................................................................... 23 Design choice ...................................................................................................................... 23 Validity ................................................................................................................................ 23
5.2.1 5.2.2 6
Case Log Analysis ................................................................................................................................ 25 6.1 Data Preparation ......................................................................................................................... 25 Data Source ......................................................................................................................... 25 Classification of Data Source ............................................................................................... 26 Data Inconsistencies ........................................................................................................... 28
6.1.1 6.1.2 6.1.3 6.2
Hypotheses ................................................................................................................................. 29 Hypotheses Justification ..................................................................................................... 29 Page | I
6.2.1
Requirements Engineer Decision Making Relationships among Decision Characteristics 6.2.2 6.3 Variable Construction ......................................................................................................... 30
Results ......................................................................................................................................... 34 Effect of the amount of products ....................................................................................... 35 Effect of the Release Heartbeat .......................................................................................... 38 Effect of large customers .................................................................................................... 40 Effect of lead time ............................................................................................................... 43 Result Summary .................................................................................................................. 43 Discussion............................................................................................................................ 44
6.3.1 6.3.2 6.3.3 6.3.4 6.3.5 6.3.6 7
Survey Analysis.................................................................................................................................... 45 7.1 7.2 7.3 7.4 Survey Design .............................................................................................................................. 45 Hypotheses ................................................................................................................................. 47 Results ......................................................................................................................................... 48 Discussion.................................................................................................................................... 48
Cross Validation .................................................................................................................................. 49 8.1 8.2 8.3 Results ......................................................................................................................................... 49 Discussion.................................................................................................................................... 49 Effect Size .................................................................................................................................... 50
9 10 11 12 13 14
Method Fragments ............................................................................................................................. 52 Conclusion ....................................................................................................................................... 56 Future Research .............................................................................................................................. 57 Acknowledgements......................................................................................................................... 58 Bibliography ...................................................................................................................................... A Appendix ...........................................................................................................................................H Research Approach .......................................................................................................................H Survey............................................................................................................................................. J SPSS Output .................................................................................................................................. L Resulting papers............................................................................................................................... U
14.1 14.2 14.3 15
Page | II
Table of Figures
Figure 1 - Content Coherence Overview ....................................................................................................... 2 Figure 2 - Reference framework for Software Product Management van de Weerd et al. (2006) ........... 7 Figure 3 - Change Request Process ............................................................................................................. 11 Figure 4 - Research Approach Overview ..................................................................................................... 18 Figure 5 - Detailed Research Approach....................................................................................................... 19 Figure 6 Possible relations among Decision Characteristics .................................................................... 30 Figure 7 - Number of decisions per amount of lead time ........................................................................... 31 Figure 8 - Histogram: Amount of Products ................................................................................................. 32 Figure 9 - Histogram: Amount of Products and Release ............................................................................. 33 Figure 10 (H1): Box plot Lead Time on Number of Products .................................................................... 35 Figure 11 - (H2a): Bar chart Decision Outcome on Number of Products ..................................................... 36 Figure 12 (H2b): Acceptance rate per Number of Products ...................................................................... 37 Figure 13 (H3): Box plot Lead Time on Release Heartbeat ....................................................................... 38 Figure 14 - (H4a): Bar chart Decision Outcome on Release Heartbeat ........................................................ 39 Figure 15 - (H4b): Acceptance Rate on Release Heartbeat .......................................................................... 40 Figure 16 (H5): Box plot Lead Time on Customer ..................................................................................... 41 Figure 17 - (H6a): Bar chart Decision Outcome on Customer ...................................................................... 42 Figure 18 - (H6b): Acceptance Rate on Customer ........................................................................................ 42 Figure 19 (H7): Box plot Lead Time per Decision Outcome ...................................................................... 43 Figure 20 - Survey Background Questions .................................................................................................. 46 Figure 21 - Survey Example ......................................................................................................................... 47 Figure 22 - Lead time per number of products affected............................................................................. 51 Figure 23 - Percentage of accepted operator related decisions................................................................. 51 Figure 24 - PDD of Decision Making Process .............................................................................................. 52 Figure 25 - Method Alternative: Explicit Number of Products ................................................................... 53 Figure 26 Method Alternative: Different Decision Process last Release Heartbeat ................................ 54 Figure 27 - Method Alternative: Monitoring of Lead Time......................................................................... 55 Figure 28 - Research Approach Data Preparation ........................................................................................H Figure 29 - Research Approach Literature Study ..........................................................................................H Figure 30 - Research Approach Survey .......................................................................................................... I Figure 31 - Research Approach Data Analysis................................................................................................ I Figure 32 - Survey Introduction ..................................................................................................................... J Figure 33 - Survey Ratings .............................................................................................................................. J Figure 34 - Survey Thanks .............................................................................................................................. J Figure 35 - Survey Background ..................................................................................................................... K
Page | III
Table of Used Tables

Table 1 - Requirements Engineering process mapped on Decision-making................................................. 5 Table 2 Research Approach: Activity Table .............................................................................................. 21 Table 3 Research Approach: Concept Table ............................................................................................ 22 Table 4 - Decision log entry example .......................................................................................................... 25 Table 5 - Levels of Measurement ................................................................................................................ 32 Table 6 Overview of Statistical Tests ....................................................................................................... 34 Table 7 - Case Log Analysis Results ............................................................................................................. 44 Table 8 - Survey Analysis Results ................................................................................................................ 48 Table 9 - Cross Validation Results ............................................................................................................... 49 Table 10 - (H1): Spearman Correlation test results ......................................................................................M Table 11 - (H2a): Frequency table K-S test ....................................................................................................M Table 12 - (H2a): K-S test results ................................................................................................................... N Table 13 - (H2b): Spearman correlation test results ..................................................................................... N Table 14 - (H3): Spearman Correlation test results ...................................................................................... O Table 15 - (H4a): Frequency table K-S test .................................................................................................... O Table 16 - (H4a): K-S test results ................................................................................................................... O Table 17 - (H4b): Spearman correlation test results ...................................................................................... P Table 18 - (H5): T-test group statistics........................................................................................................... P Table 19 - (H5): T-test results ........................................................................................................................ P Table 20 - (H6a): R x C table results .............................................................................................................. Q Table 21 - (H6a): Chi-Square test results ....................................................................................................... Q Table 22 - (H7): T-test group statistics.......................................................................................................... Q Table 23 - (H7): T-test results ........................................................................................................................ R Table 24 - Survey Summary .......................................................................................................................... R Table 25 - Survey Products - Leadtime Frequency Table .............................................................................. R Table 26 - Survey Release - Leadtime Frequency Table................................................................................ S Table 27 - Survey Customer - Leadtime Frequency Table ............................................................................ S Table 28 - Survey Products - Decision Frequency Table ............................................................................... S Table 29 - Survey Release - Decision Frequency Table ................................................................................. S Table 30 - Survey Customer - Decision Frequency Table .............................................................................. T Table 31 - Survey Lead Time - Decision Frequency Table ............................................................................. T
Page | IV
1 Introduction
Within the area of large scale requirements engineering a lot of decisions have to be made on what requirements to implement and what requirements to ignore. The moment a high level roadmap is agreed upon, changes on the roadmap become possible. Change requests (CR) are filed and the responsible product manager or decision board has to decide what requests to approve and what request to reject. Since they have to cope with a lot if requests, it is crucial to know what characteristics of a change requests determine the lead time and possibly the decision outcome is of high relevance for a company in several ways. First it is very important for a product manager to know what the consequences are of certain characteristics of a decision. When a product manager is aware of certain consequences he can take adequate actions in order to avoid negative effects. A product manager can by this improve the requirements management process within the company. Different relationships existing within Requirements Engineering Decision Making (REDM) and the consequences of the existence of these relationships on the decision making process will be analyzed in this thesis research. The research question we will try to answer in this research is: How can the Requirements Engineering Decision Making (REDM) process be optimized, knowing the decision characteristics influencing decision lead time and outcome?
The analysis is based on a case study performed at a large manufacturer of mobile communication devices. The manufacturer (from now on called Case Company), operates in a market driven environment, meaning they have to react to possible changes in the market. They apply a software product line approach to the manufacturing of its products. This means the case company uses one base product and creates all new products based on this product. The principle of software product lines is similar to a car manufacturing product line in which all different cars are build on the same chassis. The outline of this research and the sections it contains are displayed in Figure 1, where every box represents a different section in the research document. The boxes containing the bullets are general sections that apply to the entire thesis. The boxes containing other boxes are specific sections of the thesis that apply to only one specific research performed.
Page | 1

Introduction Theoretical Background Case Company Description Research Questions Research Approach Survey Analysis Survey Design Case Log Analysis Data Preparation Cross Validation Results
Hypotheses
Results
Discussion
Discussion
Hypotheses
Results
Discussion
Conclusion Future Research Acknowledgements Bibliography Appendix
Figure 1 - Content Coherence Overview
As can be seen in Figure 1, this research has two main tracks. The first track is about the analysis of the case log data that was provided by the case company. The second track is on the analysis of the survey that was performed. Both tracks come together in a cross validation section. All three analyses contain a results and a discussion section that is relevant for the specific analysis. For a more detailed view of the different sections shown in Figure 1, please see the Table of Contents. Additionally two papers resulted that were submitted to international conferences. Both papers can be found in Section Resulting papers, at the end of this thesis.
Page | 2
2 Theoretical Background
2.1 Related Work
Ensuring the quality of software releases is very important for all product software companies. The quality of a software product is highly dependent on the development process used to create it (Aurum & Wohlin, 2003). Still management methodologies have not improved as much as the software technology itself. Major causes for this for this failure are poor strategic management and related human factors (Rodrigues & Williams, 1997). Despite this, you always want your software product to contain the functionality that it needs to contain. To state this more formal; it is important to include the right requirements in the release definition for your next release (Karlsson, 2003). Knowing this, it is important to find these requirements. Often a company has to deal with a lot of requirements and determining which to include can be quite hard (Karlsson, Dahlstedt, Regnell, Natt och Dag, & Persson, 2007). A decision has to be made about every candidate requirement. The major decision is whether or not to accept a certain requirement request. Making the right decisions is the core of requirements engineering (RE) and therefore requirements engineering is identified as being mainly a decision making process (Aurum & Wohlin, 2003). Even stronger, Evan et al. said For the engineering of computerbased systems, the term of requirements might well be replaced with the term decisions (Evans, Park, & Alberts, 1997). Ashrafi (1998) even puts effort in creating a framework for decision making in software quality management. Similar efforts arent only coming from academics, but also the industry has identified the importance of decisions in requirements engineering (DeGregorio, 1999). The high importance of decisions, or even similarity of requirements engineering with decision processes mean that product managers (PM) often have to make a decision whether to accept or reject a proposed requirement. Most of the time they have to make this decision with a limited amount of information and in a short period of time (Kabbedijk, Brinkkemper, Jansen, & van der Veldt, 2009). This causes an opportunity for bad decisions to happen. In order to make the right decisions, it is important for a PM to also have the right resources to base the decision on (Fogelstrm, Gorschek, & Svahnberg, 2008). These resources can for example be an estimation of the revenues, which can by analyzed to create the most optimal set of requirements based on integer linear programming (Akker, Brinkkemper, Diepen, & Versendaal, 2007) or a feasibility analysis. When analyzing the decisions that were made by a PM, we can see several characteristics of the decisions. We can for example see if a decision is made upstream or downstream, or the time needed to take the decision (Wnuk, Svensson Berntsson, & Regnell, 2009). These characteristics can even be made more visual by using a feature survival chart for analyzing the introduction and lead time of requirements (Wnuk, Regnell, & Karlsson, 2009). Decision making is an important aspect of requirements engineering, which by some researchers is provocatively put in the center of the field (Alenljung & Persson, 2008; Aurum & Wohlin, 2003; Evans, Park, & Alberts, 1997). A number of challenges in the requirements engineering decision-making (REDM) field has been defined (Alenljung & Persson, 2008; Karlsson, Dahlstedt, Regnell, Natt och Dag, & Page | 3
Requirements Engineer Decision Making Relationships among Decision Characteristics Persson, 2007), stressing the need to understand which factors affect requirements engineering decision makers. Furthermore, the need for empirical studies of REDM has been stressed (Aurum & Wohlin, 2003; Ngo-The & Ruhe, 2005; Alenljung & Persson, 2008). The distinction between diagnosis and lookahead ways of supporting decision making is proposed by Pomerol (1998), who focuses on supporting look-ahead decision making. Various techniques, even as advanced as the Constraint Satisfaction Problem Solution Techniques (Egyed & Wile, 2006) have been proposed to automatically reduce the space of choices for ambiguities, for example in the software design decision process. The problem of selecting right requirements to the next project or product release has been described or addressed in a number of studies. Among them, Karlsson and Ryan (1997) promote a cost-value approach to support this activity, later experimentally compared to other prioritization techniques (Karlsson, Regnell, Berander, & Wohlin, 2007). Wohlin and Aurum (2003) investigated the reasons for including features, while Wnuk et al. (2009) investigated the reasons for excluding features from the scope of the project. The investigation of REDM in large scale bespoke development performed by Alenljung and Persson (2008) confirms our viewpoint of a large number of related aspects and dimensions of REDM that have to be considered in order to grasp its full complexity. Among other related papers, Strigini (1996) states lack of objective criteria for guiding making decisions, for example based on statistics about past experience, which results in important decisions often depending on subjective judgment. On the other hand, Khatri (2000) discusses the intuition's role in decision making, while Messerschmitt and Szyperski (Messerschmitt, 2004) discuss the marketplace issues that may affect software project planning and decision making. Finally, Ruhe and Saliu (Saliu & Ruhe, 2005) address decision making process during the software release planning phase. Although Andriole (1998) points out many of the decision made within REDM are taken based on intuition and personal preferences, it is still of high relevance to find the relations within REDM (NgoThe & Ruhe, 2006). Currently it is not clear what the actual cause and effect relationships within the area of requirements engineering decision making are. There is a need for some empirical research (Alenljung & Persson, 2008) to get more insight in these relationships. What we also cannot see, is the quality of the decision made. To ensure future success it is relevant for a company to analyze the quality of the decisions they made about the inclusion or exclusion of requirements. This kind of analysis goes by different names, like post-mortem analysis or retrospective analysis, and is an excellent method for knowledge managements in a company (Birk, Dingsyr, & Stlhane, 2002). A postmortem is defined as a collective learning activity which can be organized for projects either when they end a phase or are terminated. The main motivation is to reflect on what happened in the project in order to improve future practice (Dingsyr, 2005). The performing of such a postmortem analysis can be time consuming for a company, but a solution for this has been offered in the form of a lightweight method involving post-its for doing the analysis (Bjrnson, Wang, & Arisholm, 2009). This method is known as the KJ-method (Scupin, 1997). Suggestions have been made on how to do a retrospective requirements analysis (Karlsson, 2006), but these suggestions were mostly focused on release planning, instead of decision making.
Page | 4
2.2 Requirements Engineering as a Decision Process

Seeing requirements engineering purely as decision making, as stated in the previous section, needs some additional explanation. As mentioned shortly in the introduction Aurum and Wohlin (2003) mapped the requirements engineering process by Macaulay (1996) to traditional decision-making models of Anthony (1965) and Mintzberg (1976). This mapping was done to pinpoint the different roles and responsibilities that overlap between requirements engineering and decision-making. Although the awareness of the similarity between decision-making and RE is rising (Evans, Park, & Alberts, 1997), the area is still not very mature. The mapping proposed by Aurum and Wohlin is summarized in Table 1. Model Macaulay Anthony Mintzberg Classification Problem Recognition Strategic Planning Problem Identification
Problem Analysis Management Control Development Phase
Feasibility Analysis
Detailed Analysis
Requirements Document
Operational Control Selection Phase
Table 1 - Requirements Engineering process mapped on Decision-making
As can be seen in Table 1, the first two phases of the process Macaulay (1996) describes have a high similarity with the phases described by Anthony (1965) and Mintzberg (1976). The first phase of the requirements engineering process of Macaulay called Problem Recognition, is about the identification of the concepts involved in the problem. The content of this phase is similar to the phases described by Anthony and Mintzberg in their decision-making processes. The second phase of the process described by Macaulay, called Problem Analysis, again has a similar content to the phases of Anthony and Mintzberg. The last the phases of Macaulay are not described in as much detail by either Anthony or Mintzberg, but both of them do have a phase that covers the last phases of Macaulay. The mapping showed in Table 1 gives a clear overview on how requirements engineering is similar to decision making. This research focuses on the last phase of the decision making process, described by Anthony (1965) as Operational Control. We will try to make this similarity even more clear this by giving a relevant example of a situation a product manager in our case study company has to face. When a product manager gets a request for a change in the product he first identifies the problem. After this he performs an analysis on the problem and determines what is really required to comply with the request. Finally he makes a feasibility analysis to see whether it is feasible to approve the change request. The whole process done by the product manager is actually a process of decision-making; exactly as can be seen in Table 1.
2.3 Research Positioning

It is important to position the research that is described in this thesis within the appropriate research area. All research takes place within the area or software systems requirements engineering, a discipline that is described by Zave (1997) as follows:
Page | 5
Requirements Engineer Decision Making Relationships among Decision Characteristics Requirements engineering is the branch of software engineering concerned with the real-world goals for functions of and constraints on software systems. It is also concerned with the relationship of these factors to precise specifications of software behavior, and to their evolution over time and across software families. (p. 315) Another, more practical, definition of RE is given by Nuseibeh and Easterbrook (2000), who describe RE as the process of discovering . . . [the purpose of a software system] by identifying stakeholders and their needs, and documenting these in a form that is amenable to analysis, communication and subsequent implementation (p. 37). They also describe five core activities in requirements engineering, which are: Eliciting requirements Modelling and Analysing requirements Communicating requirements Agreeing requirements Evolving requirements
The activity of Evolving requirements is all about managing change (Bohner & Arnold, 1996). This research can be positioned in this last activity of requirements management, because of its focus on change request decision making. The best way to position this research is by referring to the software product management reference framework by van de Weerd et al. (2006). In this framework four major domains in requirements engineering are depicted, each containing several sub functions relevant for this domain. See Figure 2 for an adapted version of this framework. The framework depicted in Figure 2 is literally the framework as designed by van de Weerd et al. (2006), adding a light blue background to the sub functions relevant for this research. The sub functions this research positions itself in are: Requirements Prioritization Requirements Selection Release Definition Scope Change Management
Please see Figure 2 to see how these sub functions relate to other domains and sub functions.
Page | 6
Figure 2 - Reference framework for Software Product Management van de Weerd et al. (2006)
Page | 7
3 Case Company Description

We have performed a case study at a large manufacturer of mobile communication devices from Sweden with approximately 5000 employees. We use the definition of a case study by John Gerring, who defined a case study as the intensive study of a single case where the purpose of that study is at least in part to shed light on a larger class of cases (a population) (Gerring, 2007). The case company description is based on previous work by Wnuk (2008; 2009), because of the use of the same decision logs in both studies.
3.1 Product Line Environment

The case company we analyzed develops a software platform in several consecutive releases. Each platform release is the basis for one or more products that reuse the functionality and qualities of the platform. The first platform release has approximately two years lead-time from start to launch, and is focused on functionality growth and quality enhancements for a product portfolio. The following platform releases are shorter and more focused on adaptation to the different products that will be launched on the different platform releases. In order to be able to start the first release project as much as two years ahead of launch, it is necessary to create some flexibility in the process. To create this flexibility a separate flow of requirements is introduced for requirements which were not known when the release project started. This flow is called a secondary flow as opposed to the primary flow which starts at the beginning of the release project. The secondary flow approach enables the project to start analyzing the functionality that is of highest importance in the primary flow and wait with functionality that appears later. The approach provides a balance between flexibility and stability in the projects. The secondary flow is connected to the product development project that builds on the first platform release. Therefore, many of the product requirements are entered in the secondary flow since they were not analyzed at the start of the project. Also late market requirements can be taken into account in the secondary flow.
3.2 Requirements Management Process

The requirements analysts groups in our case company are called Requirements Teams. They are responsible for eliciting and specifying requirements on system level within one or more technology areas. There are 20 requirements teams with between 10 and 20 members, each serving several parallel platform projects. The requirements teams hand over the requirements to the Design Teams, who design and develop the software for the features. The company uses a stage-gate model with several increments (Cooper, 1990). There are Milestones and Tollgates to control the project progress. In particular, there are four milestones for the requirements management and design before the implementation starts. These milestones are numbered from 1 to 4. After each of these milestones, the project scope is updated and baselined. Baselining is the activity of making the next step in a process the starting point for next steps.
Page | 8
Requirements Engineer Decision Making Relationships among Decision Characteristics The stages related to, and preceding the milestones mentioned before (and numbered accordingly) are as follows: Stage 1 In this stage, high-level features are defined based on each requirements teams roadmap. The features now usually contain a description, their market value and estimations of the amount of time required to implement the feature. The features are reviewed, prioritized, and when approved, the initial scope is decided and baselined per requirements team. The requirements teams are guided by a project directive and the scope is based on initial resource estimates in the primary design team that will receive the features. Stage 2 The features from Stage 1 are refined to requirements which are specified and reviewed. An approved feature usually contains ten or more requirements that can affect various areas of the products. The features are assigned to design teams who will have the responsibility of designing and implementing the assigned features after Stage 2. The design teams also determine an effort estimate per feature. Stage 3 Here the effort estimates are refined and the scope is updated and baselined. This milestone indicates the refinement of system requirements and the beginning of the design process. Stage 4 The requirements management phase and the design phase are finished and the requirements are ready to be implemented. The final scope is decided and adapted to the total available development resources. The secondary flow starts approximately around Stage 2 and is connected to the start of the product projects. The same milestones and milestone criteria are used for the secondary flow, as described for the primary flow. The two flows run in parallel until they are merged when Stage 4 is passed for the secondary flow. All requirements are written in domain-specific natural language, usually containing a small number of sentences with many special terms that require contextual knowledge to be understood. The abstraction level of a requirement can vary from detailed implementation oriented descriptions to highlevel customer-oriented descriptions.
Page | 9
3.3 Product Line Scoping

When a project starts within the case company, first a roadmap determined, based on the long-term roadmaps the different requirements teams. The high-level requirements and additional information that is suitable for the oncoming platform project are used in this high level roadmap. All requirements teams have a project directive to guide the extraction so they know what functionality to focus on for this particular platform project. After this the content of the high level roadmap is used as a basis for creating features. Features are used to describe and decide which new functionality shall be implemented in a platform project. A feature groups requirements that constitute a new functional enhancement to a platform. A feature is thereby on such a level that it is possible to judge the market value of scoping it in to a certain platform, and also to judge the effort of implementing it. The marketvalue and effort estimates are obtained using a cost-value approach based on pair-wise comparisons (Karlsson & Ryan, 1997). This method provides the opportunity to calculate the return on investment of a certain feature and is used for decision making. The scope is decided based on the return on investment in relation to the available development resources within the design teams. All requirements and features are stored in a requirements database. There are approximately 25.000 system-level requirements, which are grouped into several hundred features. Exceptions to this are legacy requirements, which are not related to certain feature. Each requirement is described with a set of attributes such as Id, Name, Status and Source. Each feature has attributes such as Name, Description, Justification, Scope, Market-value and Effort estimate. The content of the database is regularly baselined. The scope is controlled in a document called feature list, also stored in the requirement database. The feature list is updated and baselined weekly after decisions are made in the Change Control Board. The Change Control Board decides whether or not to accept suggestions for adding and removing features from the scope based on the input of the requirements teams and other stakeholders such as product planners and design teams. The implementation of a Change Control Board has been identified as a good way of dealing with decisions in a large-scale requirements engineering environment (Konrad & Gall, 2008). All decisions made by the change control board are stored in a decision log. See Figure 3 for a visual explanation about the task of the change control board.
Page | 10
File Change Request

Submitter
CHANGE REQUEST ID Description Decision Outcome ...
Quality Gateway Analyze Completeness
1..*
Analyze Ambiguity
Analyze Similarity Change Control Board
Is based on
[else] [sufficient quality] Perform Impact Analysis

TG / FG
Decide upon acceptance

CCB
[rejected]
[accepted]
Reject Change Request

CCB
Accept Change Request

CCB
1..* Create Requirement

CCB
REQUIREMENT Id Name Status Source
1..* 1..*
FEATURE Name Description Justification Scope Market Value Effort Estimate
Figure 3 - Change Request Process
The main task of the Change Control Board is summarized in Figure 3. This process comes in place of Stage 2 and Stage 3 of the Requirements management process described in the previous section. A decision is made about all change requests and this decision, including its result is stored in the decision log mentioned before. The process described in Figure 3 is based on the decision making process of Macaulay (1996). For more information on the modeling technique used, please see section 5.1. Page | 11
Requirements Engineer Decision Making Relationships among Decision Characteristics In the change request process, a change request is filed, after which, among others ambiguity and completeness of the request are analyzed. This analysis is based on the Quality Gateway described by Natt och Dag, Regnell, Carlshamre, Andersson and Karlsson (2001). If the request is ambiguous or incomplete, it is sent back to the submitter to ask for a clarification; otherwise the request is put on the CCB agenda in order to perform the Impact Analysis (IA). An IA is performed by the appropriate Technical Groups (TGs) that elicit and specify high-level requirements for a special technical area, and Focus Groups (FGs) that design and develop previously defined functionality. After the IA the request is presented at a CCB meeting and the change request is decided upon. When an analysis performed by a certain group is not clear enough, extra information can be requested before the final decision is made. If the request is accepted the change is implemented, else the submitter gets a rejection notification.
3.4 Release Management

There are several releases in each heartbeat within our case company. Each heartbeat platform has one main product called the mother product, which is the first to use a new platform release. A mother product has one or more daughter products, which reuse platform components and requirements. Sometimes a daughter product is released at the same time as the mother product, but most of the time a daughter product is released after the mother product. The mother product is always released in the first release called R1. Successive releases are named R2, R3, etc. Daughter products are released after the mother products in one of the successive releases. One release can have many products that are released within the certain release.
3.5 Market-Driven Nature

The software produced at our case company is aimed at being offered to an open marketplace rather than to one specific customer. According to Regnell and Brinkkemper (2005), this characteristic makes our case company a Market-Driven company. The type of requirements engineering within our case company is called Market-Driven Requirements Engineering (MDRE), and is aimed at satisfying as many customers as possible. In our case however, there are two different types of customers that need to be served. 1. Consumers Explanation: The consumers are the people that will buy the product and will use it. They are the end-users of the developed product. They are however not the biggest stakeholder in the MDRE process, because of the fact they do not actively participate in requirements engineering process. 2. Operators Explanation: These are the large enterprises that own the mobile communication networks and who determine what products they will provide in specific markets. They play a very big role in the MDRE process, because they can request certain requirements for specific products developed by the case company. Page | 12
Requirements Engineer Decision Making Relationships among Decision Characteristics The highest attention in the MDRE process within our case company goes to the Operators. Besides these two types of customers, our company is also forced to react on other players. The competitors in the market our case company is active in, influence the decisions of the company as well to a large extend. Another situation of MDRE in an environment with large enterprises as stakeholders can be seen in a case study performed by Regnell, Hst, Natt och Dag, Beremark and Hjelm (2001). The task of release planning can be quite hard in MDRE (Yeh, 1992), so there is a need for flexibility in planning product releases. The short lifecycle approach used within the case company helps to achieve this flexibility in release planning for MDRE (Carlshamre & Regnell, 2000).
Page | 13
4 Research Questions
4.1 Possible Questions on Decision-Making
When looking at decision-making in requirements engineering there are a few possible questions that arise. Before the main research question can be formulated, we posed some general question on relationships between decision characteristics were formed based on the data and discussions with an expert within Sony Ericsson Mobile Communication. We listed a few of those questions below starting with the question, followed by the assumptions the question was based on. 1. Does the amount of products involved in a single decision influence the time needed to take the decision? Assumption: The more products affected by a certain decision, the larger the complexity of the decision-making and by that the longer the time needed to make the decision. 2. Does the amount of products influence the acceptance rate? Assumption: The more products involved, the harder it is to find resources to implement the change due to high complexity. Because of this the risk of rejecting can be bigger (especially in the late stages of the project). On the other hand, due to a possible product line setting one small change in the core of the product line may result in a lot of product involved. Because of the product line setting this setting can be easy to decide and to implement. 3. Does the release heartbeat influence the time needed to take a decision? Assumption: If a decision affects a single release then possibly some configuration management activities can be omitted, making statement of compliance much easier and decision lead time shorter. Also, when a decision affects a certain release heartbeat that is early in the product lifecycle, decision-making can be more complex than when a decision affects a certain release heartbeat that is in the end of the product lifecycle. This can lead to a shorter lead time for less complex decisions. 4. Does the release heartbeat influence the acceptance rate? Assumption: This assumption is similar to the assumption stated in question 3. The later the specific release heartbeat affected in the product lifecycle, the higher the change a decision accepted, because of the lower complexity of such a decision. 5. Does the market influence the lead time? Assumption: For certain markets there are specific law regulations that state that certain part of the hardware or software should be for example disabled or have limited or extended functionality. This factor could influence decision lead time in a negative way.
Page | 14
Requirements Engineer Decision Making Relationships among Decision Characteristics 6. Does the market influence the acceptance rate? Assumption: This assumption is similar to the assumption stated in question 5. Several law regulations and market constrains or extensions may force a certain decision outcome. 7. Does the involvement of an important customer influence the lead time? Assumption: Operators can be seen as very important customers. Their requests, if they are big and important, can be prioritized over other requests. This can shorten the amount of time needed to take a decision. 8. Does the involvement of an important customer influence the acceptance rate? Assumption: Some operators may be important customer so that the acceptance rate for them will be rather high. On the other hand they may still send unrealistic requests making the acceptance rate decrease. All these questions are based on possible relationships between the characteristics of a decision and their category. They are all about what characteristics influence the lead time or the acceptance rate. Based on these questions, more high level and practical questions can be formed as well. Four of those questions are stated below. 1. How can possible relations between decision characteristics help long term product management? 2. How can a product manager benefit from the knowledge about these relations for long term decisions? 3. How to recommend decision making strategies for certain markets or customers? 4. What does it mean when a decision took a long time and the outcome was rejected? Is this big waste of time and effort? What are the reasons for this happening and can this be justified? All these questions are meant to give an overview of the possible questions that can be posed about decision-making in requirements engineering.
Page | 15
4.2 Main Question

One main question is formulated based on all questions posed before. This research question is formulated in order to be able to build a theory based on the case study performed. By formulating a research question, research can be conducted more focused and it is a part of making the results testable and empirically valid (Eisenhardt, 1989). The main research question assessed in this thesis research is: How can the Requirements Engineering Decision Making (REDM) process be optimized, knowing the decision characteristics influencing decision lead time and outcome? It is assumed that the time that is needed to take a decision and the outcome of the decision are the two characteristics of a decision that have the highest relevance for a product manager. Because of this it is important to know the characteristics influencing these two aspects.
4.3 Sub Questions

The main research question is split up into several sub questions in order to be able to answer the main question. What decision characteristics influence decision lead time? What decision characteristics influence decision outcome? How can the decision making process be optimized, knowing these characteristics?
Each of these questions is answered before the main question is answered.
4.4 Relevance of Research Question

The relevance of our proposed questions is high for product managers in large software product companies dealing with a lot of decisions every day, but also for the academic world. The relevance on both academic and business level are described in more detail in the following subsections.
4.4.1 Academic Relevance

It is stated by Ngo-The and Ruhe (2006) that based on current decision support in requirements engineering it is important to identify and study further decisions problems in the RE process. In our research we will try to identify these decision problems in our case company. Furthermore Ngo-The and Ruhe also state that more empirical studies have to be performed in order to determine what factors influence decisions. We will try to identify these factors and comply with the proposed research NgoThe and Ruhe sketch in their work. Alenjung and Persson (2008) state in their work, that in order to effectively improve RE decision-making we have to identify the key aspects of RE decision-making. We will try in our paper to identify a few of those aspects and relate them to other aspects in decision making.
Page | 16
Requirements Engineer Decision Making Relationships among Decision Characteristics The decision log used in our research is used previously by Wnuk et al. (2009) and they stated that it is also interesting to further investigate what factors are affecting the lead time of decisions. Among the factors they suggested were number of stakeholders and the complexity of their wishes. This is something we will also try to answer in the form of the number of products affected and the customers that requested the change. Based on this selection of research issues in decision-making in requirements engineering can be concluded that our proposed research is highly relevant for the academic world.
4.4.2 Business Relevance

Knowing what characteristics of a decision determine the lead time and possibly the decision outcome is of high relevance for a company in several ways. First it is very important for a product manager to know what the consequences are of certain characteristics of a decision. When a product manager is aware of certain consequences he can take adequate actions in order to avoid negative effects. A product manager can by this improve the requirements management process within the company. Knowing the propositions within decision making in requirements engineering is also important for a company from a financial point of view. Since decision making is equal to requirements management, the quality of the decisions taken directly influence the quality of the requirements for a release. The higher the quality of the decisions is, the higher the quality of the product is. This means it is of the utter most importance for a company to know what influences the outcome of a decision. This research tries to identify these influences. We state that, based on the previously discussed Relevance of research question, there currently is a need for more research on decision-making in requirement engineering. There is a need to know what factors possibly influence decision-making coming from both the academic world and the business world.
Page | 17
5 Research Approach
5.1 Overview
In this section the research performed will be described in further detail, showing what steps we took and what deliverables all steps generate. We use a meta-modeling technique for modeling the relations between our activities and their deliverables. The use of meta-modeling techniques for doing this was first proposed by Saeki (2003), after which it was adjusted by van de Weerd, Brinkkemper, Souer and Versendaal (2006). They proposed a modeling technique they call a Process-Deliverable Diagram (PDD), which is based on a combination of a UML activity diagram and a UML class diagram (Object Management Group, 2009). On the left all activities are drawn, connected by dotted arrows to the related deliverables on the left. More details on this modeling technique can be found in van de Weerd et al (2006; 2007). An example of the modeling technique can be found in Figure 5.
Data Preparation
FINAL DATA SET
1 Is source for 1..* Literature Study

RESEARCH QUESTION
1..* Leads to 1..* Survey

SURVEY RESULT
Is based on
1 1 Data Analysis
1..*
Is based on
VALIDATED RESULT
1 1..* Give Advise Is based on
ADVISE
Figure 4 - Research Approach Overview
Figure 5 shows the high level overview of the way we approached our research. On the left four open activities are drawn which sub activities are expanded elsewhere. On the right the deliverables are drawn that are related to these activities. The relation between the deliverables is indicated by the small black arrows in the diagram. The diagram also shows all cardinalities between the different deliverables. A detailed overview of Figure 5 can be found in Figure 6.
Page | 18
Data Preparation
Data Analysis
Researcher
ORGANIZED RAW DATASET
1 [No unclear decisions] [Else] Is based on 1..*

UNCLEAR DECISION
Select Unclear Decisions

Researcher
1 1
DISCUSSION ITEM
[Else]
Discuss Unclear Decisions

Researcher & PM
1
DISCUSSED DECISION
Is based on
Is based on
Is based on [No unclear decisions]
Form Final Decisions

Researcher
1
DECISION
1..* 1..*
Form Final Dataset

Researcher
FINAL DATASET
Literature Study
Select Relevant Literature Study Literature Form Research Questions
RELEVANT LITERATURE
1..* Is extracted from 0..*

RESEARCH QUESTION
Uses
Researcher Is based on
0..*
Survey
Create Survey Questions

Researcher
1..*
SURVEY QUESTION
Create Survey
Researcher
[Else]
SURVEY State (published or not)
Validate Survey
Expert
Is based on 1
[Survey in order]
1 Is based on 1
Publish Survey
Researcher
Get Results
Researcher
SURVEY RESULT
QUESTION RESULT Median
Data Analysis
Analyze Survey Results Create Hypotheses

HYPOTHESIS
1..* Is based on 1 Is based on
Test Hypotheses
TEST RESULT
Is based on 1
VALIDATED RESULT
Validate Results
Researcher 1..*
CONCLUSION
1..* Is based on 1..*
Draw Conclusions
Is based on 1..*
Give Advise
ADVISE
0..* 1..*
METHOD FRAGMENT
Figure 5 - Detailed Research Approach
Page | 19
Requirements Engineer Decision Making Relationships among Decision Characteristics Figure 6 shows an expanded overview of the entire research approach. In this figure all connections between the three different main activities can be seen. All activities in Figure 6 are listed and explained in Table 3. All italic concepts refer to a concept in Table 4. Activity Data Preparation Sub-Activity Data Analysis Description In this phase the researcher checks if all decisions in the raw dataset are non-ambiguous. He checks if the affected products are clear and all relevant decision characteristics are filled in. Based on the analysis of the raw dataset, the researcher selects all decisions that are unclear and need to be further discussed with the responsible product manager. The researcher discusses all unclear decisions with the responsible product manager. Together they try to get all decisions complete and non-ambiguous. Based on all discussed items, the researcher forms the final decisions. All decisions together are formed to a final dataset by the researcher. Scientific literature relevant to Requirements Engineering Decision Making and Decision Making in General is selected in order to create a body of literature to base the research questions on. All relevant literature has to be studied by the researcher in order to get the right amount of knowledge for creating the research questions. This activity does not result in a deliverable Based on the relevant literature, the research questions are formed that will be answered during this research. In order to validate the analysis and make the results more generalizable, survey questions have to be created to be included in a survey. A survey has to be created that includes all survey questions and can be send out to similar software producing companies. The survey gets validated by an expert in the field of research methods. If the survey is not right yet, the survey questions have to be revised. After the survey is validated, the survey gets published and is communicated with potential participant through social networks and mailing lists. In order to be able to analyze the survey results, all results from the survey have to be gathered and Page | 20
Form Final Decisions Form Final Dataset Literature Study Select Relevant Literature
Study Literature
Form Research Questions
Survey
Create Survey Questions
Create Survey
Validate Survey
Publish Survey
Get Results
Requirements Engineer Decision Making Relationships among Decision Characteristics organized in a dataset. The survey results are analyzed and the median value for all survey questions is calculated. Hypotheses are created based on the research questions in order to be able to do statistical tests on the final dataset extracted from the decision log. The hypotheses get tested with the appropriate statistical test in order to determine whether they can be accepted or have to be rejected. The results of the statistical analysis are validated using the survey results gathered earlier. Based on the validated results, conclusions are drawn on the relationships existing within Requirements Engineering Decision Making. An advise in the form of method fragments that can be used within the current decision making process is given based on the previously drawn conclusions.
Data Analysis
Test Hypotheses
Validate Results Draw Conclusions
Give Advise
Table 2 Research Approach: Activity Table
All concepts and deliverables used in Figure 6 are further explained in Table 4. Concept Organized Raw Dataset Description This is the unaltered decision log, as received from the case company. The dataset is only organized by the researcher to find the incomplete and ambiguous decisions. This is a decision documented in the raw dataset, which is either incomplete or ambiguous. An unclear decision that is formatted in such a way that it is clear for the responsible product manager what the decision was about and what information is missing, wrong or ambiguous. A discussion item together with the notes taken by the researcher, based on the discussion with the responsible product manager. A non-ambiguous and complete entry in the dataset. This entry can either be integrally copied from the raw dataset, or adapted by the research based on the discussed decision. Dataset that consists of all final decisions. All literature from appropriate research area that can be of help for researching this issue. The research question is formulated in order to be able to build a theory based on the case study performed. By formulating a research question, research can be conducted more focused and it is a part of making the results testable and empirically valid (Eisenhardt, 1989). Objectives of the survey that are be part of the research question (Kitchenham & Pfleeger, 2002). Some are meant as a way of identifying the respondent, while others were meant for analyzing the research question. Question of the latter category had a three point Likert scale for answering Page | 21
Unclear Decision Discussion Item
Discussed Decision Decision
Final Dataset Relevant Literature Research Question
Survey Question
Requirements Engineer Decision Making Relationships among Decision Characteristics the questions (Jacoby & Matell, 1971). A system for collecting information from people to describe, compare or explain their knowledge, attitudes or behavior (Fink, 2003). Collection of all question results in an organized dataset. The results of the analysis of the responses participants gave on the survey question, expressed as a median (Stevens, 1946). A quantifiable specification of the requirement of the case study (Kitchenham & Pickard, 1998). The result of the testing of the hypotheses. The results is either an acceptance or rejection of H1 (Wohlin, Runeson, Host, Ohlsson, Regnell, & Wesslen, 2000). The statistical analysis test result, compared with the survey result. Whenever the survey results disprove the statistical test results, there is a reason for further research. Else the results can be considered validated. Explanation of what the results mean in practice for the case company and product managers in general. Based on the conclusion an advise will be given, containing ways of improving the efficiency of the decision making process. These ways of improvement will be given in the form of method fragments. Coherent pieces of information system development methods (Brinkkemper, 1996).
Survey Survey Result Question Result Hypothesis Test Result
Validated Result
Conclusion Advise
Method Fragment
Table 3 Research Approach: Concept Table
The sub activities of the Data Preparation, Literature Study, Survey and Data Analysis activity can be found in some more details in respectively Figure 24, Figure 25, Figure 26 and Figure 27 in the Appendix.
Page | 22
5.2 Case Study Design 5.2.1 Design choice

The use of case studies has been done in Information systems for a long time (Benbasat, Goldstein, & Mead, 1987). More recently the case study approach has also become popular in software engineering (Hst & Runeson, 2007). Case studies are sometimes criticized because they can be inaccurate or subjective. This criticism is not without reason, but with a solid and adequate case study design most of the critique can be caught off. Our case study consists of one case at our case company. This case is in fact one heartbeat in the development process of the case company. Within our case we will analyze different markets the decisions relate to. These markets can be seen as different units of analysis. This case study design is referenced to as an Embedded single case design by Yin (2003). According to the taxonomy proposed by Lethbridge, Sim and Singer (2005), we used first degree data collection methods like interviews with product managers and a survey, but also second degree data collection methods like the analysis of log files.
5.2.2 Validity
In this section we will tell how we assessed all the validity threats listed by Yin in order to perform a reliable case study research. Yin (2003) lists four types of case study tactics to ensure your case study design is sound. All definitions based on Kidder and Judd (1986). 1. Construct validity Definition: Make sure the right operational measurements are used to measure to theoretical constructs. For construct validity it is very important to use the right sources to measure the theoretical constructs. If we for example want to measure the time needed to take a decision, we need a source for this amount of time. We analyzed a backlog that was used in the decision making process. In this backlog all decisions are stored, including the date they were introduced and the date they were accepted or rejected. Based on these dates we calculated the time needed to take a decision in our case company. To be more general, the backlog we analyzed was an actively used log in or case company. All measurements we used were based on this backlog. This backlog is an archival record, which has as strength that it is stable, exact and quantitative (Yin, 2003, ss. 88-89). Whenever decisions in the backlog werent complete or were ambiguous, we discussed them with the responsible product manager to avoid wrong interpretations. These discussions can be seen as interviews we had with the responsible product manager. These interviews gave us the possibility to really focus on our case study topic. Both data collection methods are applicable to software
Page | 23
Requirements Engineer Decision Making Relationships among Decision Characteristics engineering case studies (Runeson & Hst, 2009). By doing this we used multiple sources of evidence, something Yin described as a tactic to ensure construct validity (Yin, 2003, ss. 34-36). Another tactic to create higher construct validity is to have key informants review the case study report and comment on it (Yin, 2003, s. 159). We used a responsible product manager to review our report, not because of professional courtesy, but to make sure the measurements we use are a good reflection of reality. 2. Internal validity Definition: Check what causal relationships are in place. What characteristics influence something else? Internal validity is all about causality. This means it is important to make sure that every proposition we analyze is indeed a proper proposition. If we for example investigate if X influences Y and find a significant relation, we need to be sure that Y in reality is not influenced by Z. To make sure our research question are internally valid, we based all our research questions on theoretical propositions found in the literature as can be found in the research question section. According to Yin this is the most preferred strategy to ensure internal validity (Yin, 2003, ss. 111-112). 3. External validity Definition: Making sure the findings of one case study are generalizable outside that single case. To make sure the case study we performed is generalizable to other cases as well, we used different separate units of analysis. We analyzed the different markets decisions are relevant for and by this created several small case studies within our main case. By doing this we enlarged the generalizability of our research. 4. Reliability Definition: The ability to follow the same procedure used in the research, by someone else. This can be done in order to check the results or to extend the results. Reliability could also be explained as the traceability of the steps you took in the case study research, the ability of you or other the do the same case study again with the same results. In order to ensure this reliability it is important to have a clear case study protocol and to maintain a case study database (Yin, 2003, ss. 37-39). We created a case study protocol, so all actions taken by us while performing our research can be retraced. Besides this we also stored all artifacts from the case in a case study database, so all evidence conclusions are based on can be retraced as well.
Page | 24
6 Case Log Analysis

6.1 Data Preparation 6.1.1 Data Source
In this report, we analyze the first release (one complete release heartbeat rhythm) of one particular platform project, including both primary flow and secondary flow based on the decision log. The project has just reached the fourth milestone and the final scope of the project is set. Some of the features in the database are already planned for following releases but since the focus was on the first release those were considered as out-scoped in the investigation. The project is in the so called Release Planning phase according to the software product management reference framework by van de Weerd, Brinkkemper, Nieuwenhuis, Versendaal and Bijlsma (2006). The scope is changing drastically and frequently in the investigated project, creating a lot of turbulence. Project members are frustrated about the situation and feel that management is changing the direction of the scope without realizing the effect on the project. There is a need to analyze and understand the scope changes in order to improve the scoping process. In addition, there is a need to visualize the amount of effort that is wasted because of constant scope changes, and illustrate how the scope is reduced because of inaccurate effort estimates and unrealistic stakeholder expectations. Attribute ID Change Request Decision Outcome Comments Description of proposed Change Justification Affected Release Proposition Area Main affected TG Affected Product Affected key Customer Affected FGs Submittal Date RM tool ID Decision Date
Table 4 - Decision log entry example
Description 54 HD resolution for video Accepted This will enlarge our market share in this sector Add HD resolution for recording Requested by a large provider R1.2 Video Video Group All products with a camera Customer X HD Group 09-02-09 10F1 18-02-09
Each change request to the scope of the project is registered in the decision log. An example of an entry in the decision log is shown in Table 2. For reasons of confidentiality we used fictive data. This decision log comprises a number of attributes like: the change submitter and justification, the date that the request has been submitted, the decision date, the products impacted by a change, the release of the platform project impacted by a change, and the markets impacted by a change. For our research we were granted access to an extensive decision log with all data. The decision log of all products planned
Page | 25
Requirements Engineer Decision Making Relationships among Decision Characteristics to be released in 2008 containing 1439 change requests was used as input for the performed content analysis.
6.1.2 Classification of Data Source

All change requests submitted regarding possible requirements for future releases of products within the case company can be classified in 6 possible dimensions. A certain change request always belongs to at least one of these dimensions, but they could also belong to more than one. These dimensions should not be confused with the decision characteristics. For example: a certain change request can be on the implementation of a certain feature in all Walkman phones. This change request has to be classified in the Application Planning dimension. The characteristics of the decision however can be for example that it will be released in R3 and it will affect five products. All dimension and characteristics described in this part are dimension and characteristics that can be found in the organized raw dataset as described in Figure 6. The dimensions most of the time were not explicitly mentioned as an attribute of the change request, but were described in the Affected Product attribute as can be seen in Table 2. A list of all 6 change request dimensions and a short explanation about their meaning and the amount of products they affect can be found below: 1. Platform (Software Product Line) Explanation: These decisions affect all products, since they affect the whole platform all products are based on. In this case a decision may be either platform related or non-platform related. A platform decision can be considered as a decision affecting all products released in the scope of our analysis. Products affected: All products. 2. Products Explanation: These decisions affect only a select number of products. They are directly related to these products and no specific other characteristic. Products affected: Equal to the amount of products mentioned. 3. Market Explanation: These decisions affect the products in a certain market the case company operates in, for example the Chinese market. Products affected: Equal to the amount of products released in a certain market. 4. Operator
Page | 26
Requirements Engineer Decision Making Relationships among Decision Characteristics Explanation: These decisions affect all products that are offered by a certain operator. This often goes in combination with market. For example, all phones of T-Mobile in the US. Products affected: Equal to the amount of products offered by a certain operator. 5. Release Heartbeat Explanation: These decisions affect all products that are planned to be released in a certain release within the project. Products affected: Equal to the amount of products released in a certain release. 6. Application Planning Explanation: Application planning is the part of the case company responsible for a long term roadmap definitions for important functionality that defines the brand of the companys products. This affects all products within a fixed product group. These groups can for example be all Walkman or Cybershot products. Products affected: Equal to the amount of products belonging to a certain product group. All these dimensions together describe how decisions within the case company are classified. After a change request is filed and the attributes (including the decision outcome) that are shown in Table 2 are filled in, the change request becomes a decision. Besides the dimensions of the change requests, all decisions also have certain characteristics, namely: 1. Lead time for the decision to be made Explanation: For all decisions the date is recorded on which the change request came in and the date of when the decision about the request was made. Based on these two dates, a lead time can be calculated for all decisions. The lead time is indicated by the number of days needed to take the decision. 2. Decision Outcome Explanation: It is logged for all decisions whether the change request is accepted or rejected. The CCB makes this decision and the decision outcome is either accepted or rejected. Other decision outcomes did exist in the raw data file, but these were recoded together with the responsible product manager to accepted or rejected. 3. Justification of why a certain change has been issued Explanation: For each change request there is also a short explanation on why the request has been issued. We did not actively used this explanation in our research, but it could clarify some unclear or incomplete decisions (see also section Data inconsistencies). 4. Main and other affected areas by a change
Page | 27
Requirements Engineer Decision Making Relationships among Decision Characteristics Explanation: For each decision the areas are listed that are affected by this change. We do not use this characteristic in our research, since it does not have a relationship with one of our research questions. 5. The release heartbeat affected Explanation: This characteristic differs from the dimension release. Every change request has affects a certain release in the development process. The affected release is most of the time strongly related with the moment a change request is introduced. 6. Involvement of a customer Explanation: Since the case company operates in a market driven environment (as described in section 3.5) in which customers play a big role, the involvement of a customer is one of the characteristics of a decision. 7. Number of products affected Explanation: The number of products affected by a certain decision is an important decision characteristic. This number often depends on the dimensions a change request has. Note that there are some overlapping items between the charge request dimensions and the decision characteristics. This overlap exists because of the similarity between the dimensions and the characteristics.
6.1.3 Data Inconsistencies

Since we used the real decision logs used within the case company, there were some inconsistencies within our data. We systematically removed all consistencies to create a reliable data source. All changes of the original data were documented and discussed with the relevant product manager. The first type of inconsistencies we encountered is different ways of spelling words or describing a certain thing. These inconsistencies are easy to solve, because when the spelling errors are resolved, the data can be analyzed again. The second type of inconsistencies came into play when the raw dataset was converted to an analyzable dataset. There was a need for a dataset consisting out of quantitative information, while the source was a decision log with qualitative information. Entries could for example be all products with an XYZ chip. These entries were translated to the amount of products together with a responsible product manager. The collection, coding and analysis of the data happened at the same time, like described by Glaser and Straus (1967). This way of working gives the researcher a head start in analysis but, more importantly, allows researchers to take advantage of flexible data collection(Eisenhardt, 1989, p. 539). In the process of transforming the qualitative dataset to a quantitative dataset, some assumptions have been made. Change requests affecting the entire platform, so actually all products, were considered as Page | 28
Requirements Engineer Decision Making Relationships among Decision Characteristics requests affecting only one product, since the entire platform is actually only one product. More on this can be found in section Case Log Analysis. The variable Market could not be used in the analysis, since the dataset was not clear on the different markets affected by the requests. The markets that could be identified were too small to be of significance in our research. Because of this, the mentioning and analysis of the variable Market is omitted.
6.2 Hypotheses 6.2.1 Hypotheses Justification

The first question (H1) "Is the lead time related to the number of products affected" is based on Hogarth (1975), who created a function on the relationship between the decision time and the task complexity. Hogarth states that the amount of time needed to take a decision is an increasing function of the task complexity, till a certain point. After some point the costs of errors due to the task complexity becomes lower than the cost of time. This is the tilting point at which the amount of time becomes a decreasing function of the task complexity. The question whether (H2) "The decision outcome is related to the number of products affected by a decision" is also based on the work of Hogarth (1975). Since there is a tilting point in the relationship curve, there is a certain level of complexity, after which the decision maker decides the costs of errors due to a wrong decision are lower than the costs of spending any more time an making the decision. From this point it is logical to assume wrong decisions could be made and a relationship between the decision outcome and the number of products affected by a decision could exist. Question three (H3) concerning the relationship between the release and the decision time and four (H4) concerning the release and the decision outcome are partly based on the work of Saliu and Ruhe (2005) and the work of Bagnall et al. (2001), in which they suggest a relationship between decision outcomes and release planning. No explicit relationship between decision lead time or decision outcome and the release in a release cycle is claimed, but we expect to find such a relationship in our data. The paper by Hallowell (1996), suggests a relationship among customer satisfaction, loyalty and profitability. The relationship examined by Hallowell was a business-to-consumer (B2C) relationship, while we analyzed a business-to-business (B2B) relationship. The paper by Patterson et al. (1997) however also suggests relationships among several determinants of B2B customer satisfaction, so we believe it is reasonable to assume there is the possibility there is also a (H5) relationship between the fact a request is filed by an important customer and the decision outcome. The case company benefits of keeping the customer satisfied and could, because of this, sooner accept requests of this customer than internal requests. The same reasoning goes for (H6) the relationship between a request filed by an important customer and the decision lead time. The last hypothesis formulated (H7) is based on the work of Zur et al. (1981), in which they claim a relationship between the time pressure people experience and the risks of their choice behavior. The relationships between all characteristics are summarized in Figure 6. Page | 29
Number of Products
Lead Time
Customer Involvement
Decision Outcome
Release
Figure 6 Possible relations among Decision Characteristics
As can be seen in Figure 6, Number of Products, Customer Involvement and Release all have a possible relationship with Lead Time and Decision Outcome. Lead Time has a possible relationship with Decision Outcome itself as well. All these relationships will be analyzed statistically in the next section.
6.2.2 Variable Construction

To analyze the relationships between the different decision characteristics, first some characteristics of the data have to be analyzed. Based on the decision characteristics, five variables were constructed for each decision. The concepts and measures are defined, like Verner, Sampson, Tosic, Bakar and Kitchenham (2009) suggested in order to be able to do a thorough analysis of our data. These variables can be found below. In order to test the different relationships among these variables, it is very important to know the different level of measurement of each variable. These levels of measurements can be found in Table 5. 1. Lead Time: the duration between the moment a request was filed to the moment the decision was made by the CCB. The lead time is measured in week days and not working days, so there could by a small difference in days between two decisions who took the same number of working days to be taken, due to the weekend
Page | 30
Figure 7 - Number of decisions per amount of lead time
2. 3.
4.
5.
Figure 7 gives an indication of how the lead time is distributed. About half of the decisions are made the same day they are requested (686 decisions, 48%), but the 753 requests that are left can take up to 143 days before a decision is made. Number of Products Affected: a number between one and fourteen indicating the number of different products for which the requirements would change if the request was accepted. Release Heartbeat: a variable strongly related to the release method used within the case company. The product line platform of the case company is released in a heartbeat rhythm of one base release and four sequential releases. The release heartbeat variable indicates the specific number of the release affected by the change request. The higher the variable, the later the release is in the release heartbeat rhythm of the case company. Customer: a nominal variable used to indicate whether a request is filed by an important external customer or is a request coming from inside the company. External customers in this case are very large partners of the case company who also help to bring the developed products to the market. In order to be able to perform a selection of statistical tests, this variable can be considered a dummy variable as well. Decision Outcome: again a variable of nominal level of measurement, but this variable indicates whether or not a change request is accepted by the CCB. This variable as well can be considered a dummy variable, since it value can only be zero (0) or one (1).
Page | 31
Requirements Engineer Decision Making Relationships among Decision Characteristics Beside a nominal variable, this variable can also be used as a ratio variable, by calculating the acceptance rate. This can be done using a formula like discussed before in section Decision Characteristics: total amount of request for a certain lead time = 100% amount of accepted request Level of Measurement Ratio Ordinal Ordinal Nominal / Dummy Nominal / Dummy
All levels of measurements of the different variables are listed in Table 5. Variable Lead Time Number of affected Products Release Heartbeat Customer Decision Outcome
Table 5 - Levels of Measurement
The distribution of all ordinal and ratio variables is also very important to know, since the choice for a specific test is dependent on the distribution of the data. To be able to perform most of the test a Gaussian distribution is required, which can be recognized by its bell-shaped form. When showing the distribution of the lead time of each decision we get a distribution as can be seen in Figure 8.
Figure 8 - Histogram: Amount of Products
As can be seen in Figure 8, the left image does not show a nice Gaussian distribution. In order to transform this distribution to a Gaussian distribution we calculated the log10 function of all data, which resulted on the distribution as can be seen on the right. This distribution looks more like a Gaussian distribution, but this analysis is only visually based. For ratio variables the goodness-of-fit for normality can be determined by the DAgostino-Pearson test for normality (1973). It can be tested using the SPSS macro described by DeCarlo (1997). For all tests performed within this research we formulated a null hypothesis and an alternative hypothesis in order to facilitate in valid test results (Wohlin, Runeson, Host, Ohlsson, Regnell, & Wesslen, 2000). Page | 32
Requirements Engineer Decision Making Relationships among Decision Characteristics With the D'Agostino-Pearson test we can test the following hypotheses (H0):
0 0 : The sample is derived from a normally distributed population. 0 1 : The sample is not derived from a normally distributed population.
When testing the kurtosis and skewness (DeCarlo, 1997) of the distribution we found a result of 2 (1,N = 753) = 35.3, p < .01, which is below the critical value of 67.4 as can be found in the 2 distribution table. This means we cannot reject H0, so we can conclude that the log10-function of the variable "Lead Time" is distributed normally and we can use parametric tests on this variable. Since the other variables we want to analyze are either of ordinal or nominal level of measurement, we have to use non-parametric tests in order to analyze their influences and relationships. After creating a histogram of both the variable Amount of Products and Release, can be seen in Figure 9 that none of them describe a nice Gaussian curve. Because of this violation of the Gaussian distribution, nonparametric statistical tests have to be applied when analyzing these variables.
Figure 9 - Histogram: Amount of Products and Release
Since different combinations of characteristics are tested, different tests have to be applied. The different combinations and related test are shown in Table 6. Hx H1 Relation Number of Products Lead Time Level of Measurement Ordinal Ratio Statistics Test Spearmans Rank-Order Correlation Coefficient (1904).
Page | 33
Requirements Engineer Decision Making Relationships among Decision Characteristics Hx H2a H2b H3 H4a H4b H5 H6a H6b H7 Lead Time Decision Outcome Relation Level of Measurement Number of Products Decision Ordinal Nominal Outcome Ordinal Ratio Statistics Test Independent Sample KolmogorovSmirnov Test (1939) Spearmans Rank-Order Correlation Coefficient (1904). Spearmans Rank-Order Correlation Coefficient (1904). Independent Sample KolmogorovSmirnov Test (1939) Spearmans Rank-Order Correlation Coefficient (1904). Independent Sample T-Test Chi-Square Test for r x c Tables Independent Sample T-Test Independent Sample T-Test.
Release Heartbeat Lead time
Ordinal Ratio
Release Heartbeat Decision Outcome
Ordinal Nominal
Ordinal Ratio
Customer Lead Time Customer Decision Outcome
Nominal Ratio Nominal Nominal Nominal Ratio Ratio Nominal
Table 6 Overview of Statistical Tests
The determination of the tests in Table 6 is based on The Handbook of Parametric and Nonparametric Statistical Procedures (Sheskin, 2004). Since our research is partly of exploratory nature, we posed seven different relationship questions in order to determine the relationship among different decision characteristics in requirements engineering decision making. According to Easterbrook et al. (2007), in general all of these question are of the form "Are characteristic X and Y related?". The type of relationship questions that are posed are probabilistic relations. This means that if the value of one concept changes, the value of another concept will probably change as well. For example: If A is higher, then it is likely that B is higher (Dul & Hak, 2007).
6.3 Results
In this section the effect of all different decision characteristics will be analyzed and visualized in different box plots (McGill, Tukey, & Larsen, 1978) and bar charts. The different possible relationships that are tested can also be found in Table 6. All SPSS output can be found in appendix SPSS Output.
Page | 34
6.3.1 Effect of the amount of products

Based on Hogarth et al. (1975), who state that the time needed to take a decision is highly dependent on the task complexity , we suspect a relationship between the number of products affected by a decision (e.g. the decision complexity) and the lead time. We have two major hypotheses about the relationship of the number of products a decision affects with other variables. The first hypothesis (H1) on the relationship with the lead time of a decision can be described as:
1 0 : The correlation between the number of products affected by a decision and the lead time
1 1 : The correlation between the number of products affected by a decision and the lead time
needed to take the decision is 0.
needed to take the decision is not 0.
Since we will analyze a possible relation between a variable of ratio level of measurement and one of ordinal level of measurement, we used the non-parametric Spearman's Rank-Order Correlation Coefficient (Spearman, 1904) to assess the correlation size. We found (752) = .222, p < .05 after performing the test, which is higher than the listed critical value of .197 at a two-tailed level of significance of .05. This means we can reject H0 and accept the hypothesis that the correlation between the number of affected products and the lead time is not 0. To state this more general: when the number of products affected by a decision goes up, the lead time needed to take the decision goes up as well. Since the correlation coefficient is rather low and the number of products is clearly not the only variable influencing the lead time, we will identify other variables as well in this paper. See Figure 10 for the average amount of lead time per number of affected products, together with their quartile values.
Figure 10 (H ): Box plot Lead Time on Number of Products
Page | 35
Requirements Engineer Decision Making Relationships among Decision Characteristics The second hypothesis (H2a) we tested related to the number of products affected by a decision can be described as:
2 0 : The number of products affected by a decision is not different for the different decision 2 1 : The number of products affected by a decision is different for the different decision
outcomes. outcomes.
Because the relationship between a variable of ordinal level and a variable of nominal level is tested, we use the Kolmogorov-Smirnov test for two independent samples (Smirnov, 1939). We found a result of Z = .545, p < 0.01, which is higher than the reported critical value listed for Kolmogorov-Smirnov's Z at this level of significance. This means we can reject H0 and accept our alternative hypothesis. We can conclude that there is a high likelihood the two groups are derived from different populations. More precise we could say that our data indicates that rejected decisions have a lower number of products they affect. We found both a significant relation between the numbers of products affected by a decision with the lead time needed to take a decision and the decision outcome. The more products involved in a decision, the longer the lead time and the higher the change a decision is accepted. A bar chart showing the differences in decision outcome per number of products can be found in Figure 11.
Figure 11 - (H ): Bar chart Decision Outcome on Number of Products
2a
Besides the fact whether a request was accepted or rejected, also an acceptance rate is calculated which indicates the percentage of accepted requests per number of products. To analyze a possible Page | 36
Requirements Engineer Decision Making Relationships among Decision Characteristics relationship between this acceptance rate and the number of products, in other words a correlation between these two variables we performed a Spearmans Rank-Order Correlation test on the following hypothesis (H2b):
2 0 : The correlation between the number of products affected by a decision and the
2 1 : The correlation between the number of products affected by a decision and the
acceptance rate per number of products is 0.
acceptance rate per number of products is not 0.
The test showed a (1439) = .008, p < .05 after performing the test, which is lower than the listed critical value of .197 at a two-tailed level of significance of .05. This means we can not reject H0, what means there is a no provable positive correlation between the number of products and the acceptance rate. See Figure 12 for a visualization of the relationship between the two tested variables. The different rates of acceptance per number of products are respectively 65.1%, 54.7%, 65.2%, 65%, 61.3%, 75%, 75% and 75%. The last three values are all 75% but with a different N (respectively 16, 44 and 31), so the fact they have all the same acceptance rate is purely coincidental. All groups with an N 5 where removed from the visualization and statistical analysis. This has been done in order to be able to perform valid statistical tests and not to clutter the visualization by irrelevant scores.
Figure 12 (H ): Acceptance rate per Number of Products
2b
The first hypothesis tested on the possible relationship between the number of products and the decision outcome (H2a) showed a significant result. The second hypothesis (H2b) however did not show this significant relation. We can conclude that there probably is a relationship between the number of products affected by a request and the decision outcome. When a request affects a high number of products the likelihood of acceptance is significantly higher than the likelihood of acceptance when a request affects only a small number of products. However this relationship can not be proven by Page | 37
Requirements Engineer Decision Making Relationships among Decision Characteristics analyzing the acceptance rate. The reason for this inconsistency could be the fact the acceptance rate is a constant number per number of products that does not have a standard deviation.
6.3.2 Effect of the Release Heartbeat

On the effect of a certain release we again have two main hypotheses. The first hypothesis about the effect of a certain release on the lead time of a decision (H3) is as follows:
3 0 : The correlation between the release and the lead time needed to take the decision is 0. 3 1 : The correlation between the release and the lead time needed to take the decision is not 0.
To test this hypothesis about the correlation between a variable of ordinal level and a variable of ratio level, we used Spearman's Rank-Order Correlation Coefficient. The result of this test is (752) = .180, p < .05, what is below the critical value of = .197$ for an = .05 two-tailed level of significance as can be found in the table for critical values. This means we cannot reject H0 and we cannot conclude there is any correlation between the release and the lead time needed to take a decision. This does not mean there is no relation between the release and the lead time, but it does however mean that if there is any relation we cannot prove it based on our data. Figure 13 shows several box plots of the lead time on different moments in the release heartbeat.
Figure 13 (H ): Box plot Lead Time on Release Heartbeat
We also tested the relation between the release a decision affects and the decision outcome. We stated the following hypothesis (H4a):
4 0 : The release a decision affects is not different for the different decision outcomes. 4 1 : The release a decision affects is different for the different decision outcomes.
Page | 38
Requirements Engineer Decision Making Relationships among Decision Characteristics We used the Kolmogorov-Smirnov test for two independent samples for this analysis, which resulted in a score of Z = 2.566, p < 0.01. This result is well above the documented critical value of KolmogorovSmirnov's Z, what means we can reject H0 and accept the alternative hypothesis H1. Based on the results we can state that there is likelihood that if a decision affects a release late in the release cycle, the changes of acceptance are higher. See Figure 14 for the different number of accepted and rejected requests per release heartbeat.
Figure 14 - (H ): Bar chart Decision Outcome on Release Heartbeat
4a
Besides the absolute number of accepted or rejected decisions, also the relationship between de release heartbeat and the acceptance rate is examined as shown in Figure 15. The corresponding hypothesis (H4b) is:
4 0 : The correlation between the release heartbeat and the acceptance rate per specific 4 1 : The correlation between the release heartbeat and the acceptance rate per specific
release heartbeat is 0.
release heartbeat is not 0.
After performing a Spearmans Rank-Order Correlation test, we found a (752) = .969, p < .05, which is far above the critical value of = .197$ for an = .05 two-tailed level of significance as can be found in the table of critical values. This means we can reject H0 and accept the alternative hypothesis H1. This correlation is visualized in Figure 15, in which you can see the positive trend in acceptance rate, when the release heartbeat increases.
Page | 39
Figure 15 - (H ): Acceptance Rate on Release Heartbeat
4b
Both hypotheses showed a significant relationship between the change on acceptance and the release heartbeat. This means there is a high probability that request are sooner accepted when they affect releases that take place late in the release heartbeat.
6.3.3 Effect of large customers

To test the effect a large customer can have on decision making we created two types of decisions. The first type are decisions that are requested from somewhere within the company, while the second type are decisions that are requested by large customers of the case company. We first tested the difference of lead time needed to take decisions when large customers are involved in comparison with decision where they are not involved. In order to test this, we did an independent sample t-test on the following hypothesis (H5):
5 0 : The average lead time needed to take a decision is not different when a large customer is 5 1 : The average lead time to take a decision is different when a large customer is involved.
involved.
The result of the t-test (t(752) = .586,p = .558) did not allow us to reject H0, therefore we can state that based on our data there is no significant difference between the lead time needed to take decision when the decision is requested by a large customer, compared with a decision that is requested from within the company. We also tested if there was any effect on the decision outcome caused by large customers. In order to test this we had to perform a 2-test for r*c tables, because we tested the relation between two variables of nominal level of measurement. A visualization showing the box plots of lead time on customer involvement can be found in Figure 16.
Page | 40
Figure 16 (H ): Box plot Lead Time on Customer
Our next hypothesis (H6a) on the relationship between the decision outcome and the involvement of an important customer, as shown in Figure 17 is:
6 0 : The frequencies in the contingency table between the decision outcome and involvement
6 1 : The frequencies in the contingency table between the decision outcome and involvement
of a large customer do not differ from the normal expected frequencies. of a large customer differ from the normal expected frequencies.
The result of this test is with 2 (1,N = 1439)=7.032, p < .01 above the listed critical value. This means we can reject H0 and accept our alternative hypothesis. Since the value of 2 is rather low, we can state that the change of a positive decision outcome is with large likelihood a little higher when a decision is requested by a large customer.
Page | 41
Figure 17 - (H ): Bar chart Decision Outcome on Customer
6a
When the acceptance rate is calculated for the requests that are, and the requests that are not requested by an important customer a visualization can be drawn like Figure 18.
Figure 18 - (H ): Acceptance Rate on Customer
6b
In Figure 18 an acceptance rate of 62.9% can be seen for requests that are not filed by an important customer. A rate of 70.3% is accepted for requests that are filed by an important customer. This means 7.4% more request are approved if they are filed by an important customer, than when they are not. No Page | 42
Requirements Engineer Decision Making Relationships among Decision Characteristics statistical test could be performed on the relationship between the acceptance rate and the customer variable, since the acceptance rate is a variable having no standard deviation because of the fact it is constructed as an average within the groups of customer and not customer related.
6.3.4 Effect of lead time

The final relation we examined is whether the lead time influences the acceptance rate. In order to test this we stated the following hypothesis (H7):
7 0 : The average lead time needed to take a decision does not differ per decision outcome. 7 1 : The average lead time to take a decision does differ per decision outcome.
The average lead time for accepted and rejected decisions is respectively = 1.12 and = .98. The result of the t-test (t(752) = 3.940,p < 0.01) indicated a significant differences between the average lead time for both decision outcomes. This means we can accept H1 and reject the null-hypothesis. Based on these results we can state that the average lead time needed to reject a decision is significantly longer than the lead time needed to accept a decision. This relationship is visualized in Figure 19.
Figure 19 (H ): Box plot Lead Time per Decision Outcome
6.3.5 Result Summary

All results from the previous sections are summarized in Table 7, in which the results of the test are shown, together with the level of significance. Hx H
1
Test Result Significant
Level of Significance = .222 > .197
Page | 43
Requirements Engineer Decision Making Relationships among Decision Characteristics Hx H2a H

2b
Test Result Significant Not Significant Not Significant Significant Significant Not Significant Significant Not applicable Significant
Level of Significance Z = .545 > .440 = .008 < .197 = .180 < .197 Z = 2.566 > .440 = .969 > .197 p = .558 > 0.05 2 = 7.032 > 2.710 Not applicable p = 0.01 < 0.05
H3 H
4a
H4b H
5
H6a H6b H7
Table 7 - Case Log Analysis Results
6.3.6 Discussion
The analysis showed a significant relationship between the number of products affected by a decision and both the decision outcome and lead time. This means we can state with a high certainty that there is a relationship between the decision complexity and the outcome and time needed to take a decision in our case company, like suggested in the literature (Hogarth, 1975). Other research (Bagnall, RaywardSmith, & Whittley, 2001; Saliu & Ruhe, 2005) also suggested a possible relationship between release planning and decision quality. This relationship could only partly be found in our case data, since there proved to be no significant relationship between the release a decision affects and the lead time. We did find a significant relationship between the release and the decision outcome, meaning decisions either get accepted or rejected more when the case company is later in their release cycle. The fact a request is filed by an important customers has no relationship with the decision lead time. It does however have a relationship with the decision outcome. Request filed by an important customer or more easily accepted than request coming from inside the company. The final result of our statistical analysis shows a significant relationship between the lead time and the decision outcome. This means that the decision outcome could be influenced by the time needed to take a decision. This implication could be of high relevance because it could mean that more wrong decisions are made in decision procedures that take a long time. In order to generalize the results of our case study and make conclusions like the concern stated above, the results previously discussed need to be compared with the results from the survey conducted among similar software producing companies, as described in the next section.
Page | 44
7 Survey Analysis
7.1 Survey Design
To validate our results, we referenced the results we found by performing statistical analysis on the decision log with experiences from other similar companies. To measure these experiences we created a questionnaire based on principles described by Kitchenham and Pfleeger (2002). The questionnaire contained a part to identify the context and background of the participant, followed by a part asking about their experiences considering possible relations in requirements engineering decision-making. The questions identifying the context and background of the participants are based on the facets identified by Paech et al. (2005). They suggest including, among others, questions on the region, role and experience of the participants. The questions that were asked can be found in Figure 19. The questions concerning the possible relationships within requirements engineering decision making were all structured by using a three-point Likert scale for effectively measuring the experiences of the respondents (Jacoby & Matell, 1971). We asked all respondents to state whether a certain characteristic influenced the decision lead time in a positive, neutral or negative way. The same question was asked about the influence of decision characteristics on the decision outcome. For example, some questions from the survey were: 1. "Please indicate how the number of products affected by the decision influences the time needed to take the decision"
2. "Please indicate how the fact an important customer is involved influences the time needed to take the decision"
3. "Please indicate how the number of products affected by the decision influences the decision outcome" The answer categories for example 1 and 2 were: "This makes the time to decide shorter", "No influence" or "This makes the time to decide longer. For example 3 the answer categories were: This increases the probability of rejection, No influence, or This decreases the probability of rejection. In Figure 20 a screenshot of a part of the survey can be found as an example. The entire survey can be found in the Appendix in the Survey section. A summary of the survey results can be found in the Appendix as well.
Page | 45
Figure 20 - Survey Background Questions
Page | 46
Figure 21 - Survey Example
7.2 Hypotheses
In the survey analysis we again posed relationship questions (Easterbrook, Singer, Storey, & Damian, 2007) in order to identify the relationships between the number of products affected by a decision, the specific release heartbeat and the involvement of an important customer with the lead time and the decision outcome. We examine seven hypotheses with the results of our conducted survey. All hypotheses are based on the same scientific foundations as described in section 6.2.1. The hypotheses are as follows: H 1. The number of products affected by a change request has a relationship with the lead time needed to take a decision. H 2. The number of products affected by a change request has a relationship with the decision outcome. H 3. The specific release heartbeat a change request affects has a relationship with the lead time needed to take a decision H 4. The specific release heartbeat a change requests affects has a relationship with the decision outcome. H 5. The involvement of an important customer has a relationship with the lead time needed to take a decision. H 6. The involvement of an important customer has a relationship with the decision outcome. H 7. The lead time needed to take a decision has a relationship with the decision outcome. Page | 47
7.3 Results
In order to be able to test our hypotheses, we recoded the possible survey answers to quantitative analyzable results. We rated the first answer as a score of -1, the second answer as 0 and the last answer as +1. This allowed us to determine how strong a certain decision characteristic influenced the decision lead time or outcome. Since a variable of ordinal level of measurement is created by coding the answer options like described above, we use the median and percentile scores as ways of assessing the survey results (Stevens, 1946). In this case this means when a median is negative, at least half of the sample has identified a negative relationship. When the median is positive, at least half of the sample has identified a positive relationship. A frequency table can be used to further analyze the results. Table 8 shows the median values for all hypotheses tested within the survey research. HX H1 H2 H3 H4 H5 H6 H7 Median 1 -1 -1 -1 -1 1 0
Table 8 - Survey Analysis Results
7.4 Discussion
As can be seen in Table 8, only H 1 and H 6 have a positive relationship according to the survey respondents. H 7 has a neutral relationship, while all other hypotheses have a negative relationship. This means that in general most respondents did not think there is a relationship between the certain change request characteristics and the decision lead time or outcome. They did think there is a positive relationship between the number of products affected in a change request and the lead time needed to take the decision. They also think there is a positive relationship between the fact an important customer is involved and the decision outcome. Remarkable is the high amount of negative relations, meaning most products managers think the lead time and the acceptance rate become lower if change request characteristics, like the specific release heartbeat, become higher as well. The amount of overlap between the results of the CCB decision log and the survey analysis will be discussed in the next section.
Page | 48
8 Cross Validation
8.1 Results
Table 7 shows a list of all hypotheses together with their survey result median. Column "Level of Significance" supplies all test results, together with their critical values. Hx Results Decision Log Analysis (N=1439) Test Result Significant Significant Not Significant Not Significant Significant Significant Not Significant Significant Not applicable Significant Results Survey Analysis (N=46) Median 1 -1 -1 -1 -1 -1 -1 1 1 0
Level of Significance = .222 > .197 Z = .545 > .440 = .008 < .197 = .180 < .197 Z = 2.566 > .440 = .969 > .197 p = .558 > 0.05 = 7.032 > 2.710 Not applicable p = 0.01 < 0.05
2
H H H
2a
2b 3
H H H
4a
4b 5
H H H
6a
6b 7
Table 9 - Cross Validation Results
The results of the survey do not always correspond with the results from the case study, but there are some hypotheses that do have corresponding results. We will focus on those hypotheses in the next section.
8.2 Discussion
We state the results of the statistical analysis can be validated if the result of the appropriate test is significant and the survey shows a score below or above zero (a score of zero would indicate no relationship). Based on Table 7 hypotheses H1, H2, H4 and H6 seem to satisfy these constraints. On H1 we statistically proved that if the number of products affected by a decision rises, the decision lead time rises as well. The results of the survey show a positive relationship as well, so it is highly probable that the number of products affected by a decision has a positive relationship with the decision lead time. In other words, the bigger or more complex a decision is, the longer it can take to make a decision. H2 showed a relation between the number of products affected in a decision and the decision outcome. We could see that decisions are sooner accepted when they have a large number of products they affect, than when they affect a lower number. The survey however indicated an opposite result and shows a different relationship. The results of the survey disprove the statistical case results. This conflict Page | 49
Requirements Engineer Decision Making Relationships among Decision Characteristics in results could be caused by the fact that our case company uses a development method in which all products released are based on a certain platform and platform decision, although affecting all products, could be rather easy to take because they actually only affect one product. For H4 we see the same situation with the statistical analysis showing an opposite relation in comparison to the survey results as for H2. Requests affecting products being released late in the release cycle are more likely to be accepted in our case company than requests that are in the beginning of the release cycle. The survey showed that most respondents experience this relationship to be the other way around. We suspect this contrast in results to be caused by our case company getting more requests from external customers instead of internal request the later a release takes place in the release cycle. Customers use the products released in the beginning of the release cycle and come up with requests for future releases. This effect could explain the contrasting results between the case company and survey analysis. The explanation of the difference within the H4-results stated something about a possible effect of external customers with the decision outcome. The survey results for H6 do correspond with the statistical analysis results of our case company and indeed show a relationship between the fact a change is requested by an important external customer and the decision outcome. Both indicate that a request of an external customer is sooner accepted than a request from an internal department. H3, H5 and H7 are either not significant in the statistical analysis, or have a median score of zero in the survey results. This does not necessarily mean the hypotheses are wrong, it only means they could not be proved using our data. The survey shows a negative relationship between the release a decision affects products in and the time needed to take a decision for H3. In this case the respondents of the survey indicate they experience a shorter time needed to take a decision if a request affects products that are scheduled in a release taking place late in the release cycle. The analysis of our decision log did not show any significant relationship, meaning we could not validate the result. Same goes for H5 on which the survey shows a negative relationship, while the statistical analysis of the case data shows no significant relationship whatsoever. H7 does show a significant relation in the statistical analysis, but here the survey results showed a neutral relationship. Because of these conflicting results we cannot make a strong conclusion on these hypotheses.
8.3 Effect Size

Since our validated results showed a relationship between the number of products affected by a decision and the lead time (H1) and the fact an important customer is involved and the decision outcome (H6), we investigated these relationships in some more depth.
Page | 50
Figure 22 - Lead time per number of products affected
Figure 21 shows an increase of the average lead time related to the number of products. If we compare the average lead time for 1 product with the highest lead time (for 7 products), the lead time becomes about five times longer. If we look at a more realistic comparison of lead time between the lead time for 1 product (least number) and 13 products (highest number) we can still see an increase of 130% in average lead time. There appears to be no clear function to predict the amount of time needed to take a decision when the number of products is known, but there is a clear positive trend to be seen. The low amount of time needed when a decision affects 6 or 11 products is due to different product groups within the software product line approach. A certain change request can affect for example all products containing a certain function, making decisions about this reasonably easy. As can be seen by the percentages described above, the growth of lead time when the number of product rises is substantial.
Figure 23 - Percentage of accepted operator related decisions
A significant difference between the number of accepted decisions on important external customers and internal request was found. This difference is displayed in Figure 22, where we can see an 11% difference between the two groups. Expecting a higher acceptance rate on internal requests, because of an expected higher accuracy of internal requests, this relationship is remarkable. Page | 51
9 Method Fragments
In this section several suggestions will be given to improve the decision making process in place within our case company. The current decision making process applied within our case company and based on the general process model of Saaty (2008) can be described in a PDD as shown in Figure 23.
File Change Request

Submitter
CHANGE REQUEST ID Description Decision Outcome ...
1..*
Analyze Ambiguity
Analyze Similarity Change Control Board
Is based on

TG / FG

CCB
[rejected]
[accepted]

CCB

CCB

CCB
1..* 1..*
Figure 24 - PDD of Decision Making Process
Page | 52
The statistical analysis of the decision logs and the survey research indicated several change request characteristics negatively influencing the decision lead time or influencing the decision outcome. From the results of testing H1 and H2 we can see that if a change request will affect a large number of products, the lead time will be longer and the request will sooner be accepted. This could mean that the CCB has problems deciding on large change request and in the end just accept the request. In order to counteract this effect we suggest a method fragment to be included in the PDD shown in Figure 24. A method fragment is a coherent piece of information system development methods (Brinkkemper, 1996).
File Change Request

Submitter
CHANGE REQUEST ID Description Decision Outcome Number of products affected ..
1..*
Analyze Ambiguity
Analyze Similarity Determine Number of Products affected Change Control Board Is based on

TG / FG

CCB
[rejected]
[accepted]

CCB

CCB

CCB
1..* 1..*
Figure 25 - Method Alternative: Explicit Number of Products
Page | 53
Requirements Engineer Decision Making Relationships among Decision Characteristics We suggest a method fragment as shown in Figure 25, in which the quality gateway is extended with one additional activity, in order to make the number of products affected by a change request more explicit. All red elements in Figure 25 are additional or changed activities in comparison to the original PDD. An extra activity called Determine Number of Products Affected is implemented in order to create the property Numbers of Products Affected in the Change Request. By implementing this fragment, the CCB will be more aware of the fact a change request affects a lot of products, since the number of products is no longer implicit in the description. By being more aware of the number of products affected by a change request can help in identifying the large requests easier and by this decide sooner on acceptance or rejection. The test of H4 showed a positive relationship between the specific release heartbeat and the acceptance rate. This means change requests are sooner accepted if they affect a release that is late in the release cycle. It is possible that this increase in acceptance rate is due to the fact the CCB experiences a certain time creep and just accepts requests.
[rejected]
[accepted]

Developer & CCB

Developer & CCB

Developer & CCB
1..* 1..*
Figure 26 Method Alternative: Different Decision Process last Release Heartbeat
In Figure 26 an alternative method fragment is suggested in which one or more relevant developers are involved in the decision making process in order to make a more accurate and realistic decision on whether to accept or reject a change request in the last release heartbeat, when time is short. By involving developers, decisions taken have a higher probability of being realistic in this last stage of the development process, because of the hands-on experience of the developers.
Page | 54
Perform Impact Analysis

TG / FG
Monitor Lead Time

CCB
[LT > 50] [LT 50] Decide upon acceptance

CCB
[rejected]
[accepted]

CCB

CCB

CCB
1..* 1..*
Figure 27 - Method Alternative: Monitoring of Lead Time
The testing of H7 showed that a longer lead time before a decision is made, results in a lower change of acceptance. This means that requests that take a long time are just rejected in the end. This rejection can sometimes take up to 150 days before finally being decided upon. This causes an unnecessary clutter of the decision making process and should be tried to be kept to a minimum. In order to achieve this minimum and alternative method is suggested as shown in Figure 27, in which to lead time build up before a decision is taken is monitored. By monitoring this lead time, the CCB is more aware of decisions taking a long time and are they able to reject decision that take too much time. The decision point for rejecting a request is set at 50 days, since a raise in rejection can be seen from this point on. All alternative method fragments show in this section are suggestions for a more effective decision making process. The fragments are based on the current decision making process and should still be validated in future research.
Page | 55
10 Conclusion
Based on our case study, statistical analysis and survey results, we can conclude there is a relationship between the number of products affected in a decision and the time needed to take a decision. This relationship may seem trivial, but it is very important for a product manager to know exactly what decision characteristics cause a long decision lead time. We showed it is wise for a product manager not to have too large or complex decisions, since both the case study and survey analysis indicated that this will cause a longer decision lead time. In a market driven environment, this certainly is an undesirable effect. Especially not since the lead time could become up to 400% longer if a decision affects multiple products. We also showed another relationship of which the desirability of its effect is disputable. Change requests done by an external customer are more likely to be accepted than internal request. Requests filed by an important customer have an 11% higher change to be accepted than other requests. This effect has two sides; on one hand it can be positive since Hallowell (1996) and Kabbedijk et al. (2009) both stated the importance of listening to customers in order to get their satisfaction and loyalty. On the other hand this effect could also mean that change requests filed by an important customer are only accepted to satisfy this customer and not because it is a useful change to the product. Product management processes can be adapted when being aware of the supported relationships. For example the change process of Figure 3 can be refined by asking for additional details on external customers in order to reduce the lead time as suggested in the previous section. Related to the differences observed between the statistical analysis results and the survey results, it should be noticed that product managers are not always aware of certain relationships among decision characteristics. The statistical analysis of the decision logs could have identified relationships that project managers are not familiar with. For instance the rejection of H2 (more affected products means a higher acceptance change) could teach product managers not to discourage the submission of change requests with many products involved. Furthermore product managers could be helped by implementing the method fragments shown in section 9, in order to counteract negative relationships existing among change request characteristics and achieve a more efficient decision making process in market driven product line requirements engineering.
Page | 56
11 Future Research
The purpose of this research was to identify important relationships within Requirements Engineering Decision Making. We have investigated several relationships among decision characteristics, based on existing literature and tried to validate those relationships. We found significant statistical analysis results for most of the relationships and survey results confirm some of them. Other relationships were not confirmed by the survey results or relationships were indicated by the survey results that were not found in the statistical analysis. Our research made clear that certain relationships can be found among decision characteristics, but we could not yet prove or disprove all of them. Future research is needed to go more in depth on the possible relationships among REDM characteristics. Several relationships could be proven by us, but all of these relationships, including the relationships we could not prove or disprove, need further research in order to really validate them. Within the relationships proven by statistical analysis of the CCB decision log and the survey analysis, more research is needed. Additional decision logs should be analyzed within the same case company or other similar case studies at other companies should be performed in order to examine the relationships into more depth. Additionally, it would be very helpful if a function could be formulated to estimate the lead time or the chance on a certain decision outcome. Such a function could help the CCB and product managers in general to select alternative method fragments when they are necessary. Finally, other decision characteristics, such as the number of stakeholders involved, could also be of relevance for the decision lead time or outcome. These characteristics have not yet been taken into account in this research, but could be very interesting for future research. The alternative method fragments are based on the current decision making process and should still be validated in future research. This could be done be really implementing the fragments and measuring the lead time and decision outcome for future release within the case company.
Page | 57
12 Acknowledgements
I would first of all like to thanks prof. dr. Sjaak Brinkkemper for inspiring and supervising me throughout the entire process of doing my research. I would also like to thank dr. Inge van de Weerd for acting as my second supervisor and dr. Ronald Batenburg for reviewing and discussing the statistical methods used in this research. During my stay at Lund University I was supervised by prof. Bjrn Regnell and drs. Krzysztof Wnuk. I would like to thank both for their support and opportunities they gave me and specifically Krzysztof for the personal help he gave me not only on doing research, but also on getting started in Lund and at Lund University. Last I would like to thank prof. Per Runeson for helping me out with my survey research and Thomas Olssen for all the help and time he spent helping me interpret the case company data.
Page | 58
13 Bibliography
Akker, M. v., Brinkkemper, S., Diepen, G., & Versendaal, J. (2007). Software product release planning through optimization and what-if analysis. Information and Software Technology , 50 (2), 101-111. Alenljung, B., & Persson, A. (2008). Portraying the practice of decision-making in requirements engineering: a case of large scale bespoke development. Requirements Engineering , 13 (4), 257-279. Andriole, S. (1998). The politics of requirements management. IEEE Software , 15 (6), 82-84. Anthony, R. (1965). Planning and Control Systems: A Framework for Analysis. Boston: Harvard University. Ashrafi, N. (1998). A decision making framework for software total quality management. International Journal of Technology Management , 16 (4), 532-543. Aurum, A., & Wohlin, C. (2003). The fundamental nature of requirements engineering activities as a decision-making process. Information and Software Technology , 45 (14), 945-954. Bagnall, A., Rayward-Smith, V., & Whittley, I. (2001). The next release problem. Information and Software Technology , 43 (14), 883-890. Ball, T., & Eick, S. G. (1996). Software Visualization in the Large. IEEE Computer , 29 (4), 33-43. Benbasat, I., Goldstein, D. K., & Mead, M. (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly , 11 (3), 369-386. Birk, A., Dingsyr, T., & Stlhane, T. (2002). Postmortem: Never Leave a Project without It. IEEE Software , 19 (3), 43-45. Bjrnson, F. O., Wang, A. I., & Arisholm, E. (2009). Improving the effectiveness of root cause analysis in post mortem analysis: A controlled experiment. Information and Software Technology , 51 (1), 150-161. Bohner, S. A., & Arnold, R. S. (1996). Software change impact analysis. Wiley: IEEE Computer Society Press. Brinkkemper, S. (1996). Method engineering: engineering of information systems development methods and tools. Information and Software Technology , 38 (4), 275-280. Carlshamre, P., & Regnell, B. (2000). Requirements lifecycle management and release planning in market-driven requirements engineering processes. 11th International Workshop on Database and Expert Systems Applications (pp. 961-965). London, UK: IEEE Computer Society. Chen, C. (2005). Top 10 unsolved information visualization problems. IEEE Computer Graphics and Applications , 25 (4), 12-16. Page | A
Requirements Engineer Decision Making Relationships among Decision Characteristics Cooper, R. G. (1990). Stage-Gate Systems: A New Tool for Managing New Products. Business Horizons , 33 (3), 44-53. D'agostino, R., & Pearson, E. (1973). Tests for departure from normality. Empirical results for the distributions of b^2 and sqrt(b^1). Biometrika , 60 (3), 613-622. DeCarlo, L. T. (1997). On the meaning and use of kurtosis. Psychological Methods , 2 (3), 292-307. DeGregorio, G. (1999). Enterprise-wide requirements and decision management. 9th International Symposium of the International Council on System Engineering. Brighton, UK. Dingsyr, T. (2005). Postmortem reviews: purpose and approaches in software engineering. Information and Software Technology , 47 (5), 293-303. Dul, J., & Hak, T. (2007). Case study methodology in business research. Oxford, UK: Butterworth Heinemann. Easterbrook, S., Singer, J., Storey, M. A., & Damian, D. (2007). Selecting empirical methods for software engineering research. In F. Shull, J. Singer, & D. I. Sjoberg, Guide to Advanced Empirical Software Engineering (pp. 285-311). London: Springer. Egyed, A., & Wile, D. S. (2006). Support for Managing Design-Time Decisions. IEEE Transactions on Software Engineering , 32 (5), 299-314. Eisenhardt, K. M. (1989). Building theories from case study research. Academy of management review , 14 (4), 532-550. Elmasri, R., Weeldreyer, J., & Hevner, A. (1985). The category concept: An extension to the entityrelationship model. Data Knowledge Engineering , 1 (1), 75-116. Evans, R., Park, S., & Alberts, H. (1997). Decisions not requirements: decision-centered engineering of computer-based systems. IEEE International Conference on the Engineering of Computer-Based Systems (pp. 435-442). Monterey, USA: IEEE Computer Society. Fink, A. (2003). The Survey Handbook. London: Sage Publications. Fogelstrm, N. D., Gorschek, T., & Svahnberg, M. (2008). Needs Oriented Framework for Producing Requirements Decision Material - NORM. 2nd International Workshop on Software Product Management, (pp. 9-17). Barcelona, Spain. Gerring, J. (2007). Case Study Research: Principles and Practices. New York: Cambridge University Press. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine Publishing.
Page | B
Requirements Engineer Decision Making Relationships among Decision Characteristics Gotel, O. C., & Marchese, F. T. (2007). On Requirements Visualization. 2nd International Workshop on Requirements Visualization (pp. 80-90). New Delhi, India: IEEE Computer Society. Haber, R. B., & McNabb, D. A. (1990). Visualization ldioms: A Conceptual Model for Scientific Visualization Systems. In G. M. Nielson, B. Shriver, & L. J. Rosenblum, Visualization in Scientific Computing (pp. 74-93). Los Alamitos, CA: IEEE Computer Society. Hallowell, R. (1996). The relationships of customer satisfaction, customer loyalty, and profitability: an empirical study. International Journal of Service Industry Management , 7 (4), 27-42. Hogarth, R. M. (1975). Decision time as a function of task complexity. In D. Wendt, & C. Vlek, Utility, Probability, and Human Decision Making (pp. 321-338). Dordrecht, Holland: D. Reidel Publishing Company. Hst, M., & Runeson, P. (2007). Checklists for Software Engineering Case Study Research. 1st International Symposium on Empirical Software Engineering and Measurement (pp. 479-481). Madrid, Spain: IEEE Computer Society. Jacoby, J., & Matell, M. S. (1971). Three-point Likert scales are good enough. Journal of Marketing Research , 8 (4), 495-500. Johnson, C., & Sanderson, A. (2003). Visualization viewpoints. IEEE Transactions on Computer Graphics and Applications , 23 (5), 6-10. Kabbedijk, J., Brinkkemper, S., Jansen, S., & van der Veldt, B. (2009). Customer Involvement in Requirements Management: Lessons from Mass Market Software Development. 17th International Requirements Engineering Conference, (pp. 281-286). Atlanta, USA. Karlsson, J., & Ryan, K. (1997). Cost-Value Approach for Prioritizing Requirements. IEEE Software , 14 (5), 67-74. Karlsson, L. (2003). Improving Requirements Selection Quality in Market-Driven Software Development. Lic. Thesis, Lund University, Department of Communication Systems. Karlsson, L. (2006). Requirements Prioritisation and Retrospective Analysis for Release Planning Process Improvement. Phd Thesis, Lund University, Department of Communication Systems. Karlsson, L., Dahlstedt, A. G., Regnell, B., Natt och Dag, J., & Persson, A. (2007). Requirements engineering challenges in market-driven software development An interview study with practitioners. Information and Software Technology (49), 588604. Karlsson, L., T, T., Regnell, B., Berander, P., & Wohlin, C. (2007). Pair-wise comparisons versus planning game partitioning-experiments on requirements prioritisation techniques. Empirical Software Engineering , 13, 3--33. Khatri, N. (2000). The Role of Intuition in Strategic Decision Making. Human Relations , 53, 57--86. Page | C
Requirements Engineer Decision Making Relationships among Decision Characteristics Kidder, L., & Judd, C. M. (1986). Research Methods in Social Relations (5th ed.). New York: Holt, Rinehart & Winston. Kitchenham, B. A., & Pfleeger, S. L. (2002). Principles of survey research part 2: designing a survey. ACM SIGSOFT Software Engineering Notes , 27, 18--20. Kitchenham, B. A., & Pfleeger, S. L. (2002). Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Software Engineering Notes , 27, 24. Kitchenham, B. A., & Pickard, L. M. (1998). Evaluating software engineering methods and tools: part 9: quantitative case study methodology. ACM SIGSOFT Software Engineering Notes , 23, 26. Konrad, S., & Gall, M. (2008). Requirements engineering in the development of large-scale systems., (pp. 217--222). Lethbridge, T. C., Sim, S. E., & Singer, J. (2005). Studying software engineers: Data collection techniques for software field studies. Empirical Software Engineering , 10 (3), 311-341. Macaulay, L. (1996). Requirements engineering. London: Springer-Verlag Series on Applied Computing. McGill, R., Tukey, J. W., & Larsen, W. A. (1978). Variations of box plots. American Statistician , 32, 12--16. Messerschmitt, D. G. (2004). Marketplace issues in software planning and design. IEEE Software , 21, 62-70. Mintzberg, H. ,. (1976). The Structure of "Unstructured" Decision Processes. Administrative Science Quarterly , 246-275. Natt och Dag, J., Regnell, B., Carlshamre, P., Andersson, M., & Karlsson, J. (2001). Evaluating automated support for requirements similarity analysis in market-driven development. Ngo-The, A., & Ruhe, G. (2006). Decision Support in Requirements Engineering. In A. Aurum, & C. Wohlin, Engineering and Managing Software Requirements (pp. 267-286). Springer Berlin Heidelberg. Ngo-The, A., & Ruhe, G. (2005). Engineering and Managing Software Requirements. Springer. Nuseibeh, B., & Easterbrook, S. (2000). Requirements engineering: a roadmap., (pp. 35--46). Object Management Group. (2009, 02 02). OMG Unified Modeling LanguageTM (OMG UML), Superstructure. Opgeroepen op 11 19, 2009, van OMG: http://www.omg.org/spec/UML/2.2/Superstructure Paech, B., Koenig, T., Borner, L., & Aurum, A. (2005). An Analysis of Empirical Requirements Engineering Survey Data. Engineering and Managing Software Requirements , 427--452.
Page | D
Requirements Engineer Decision Making Relationships among Decision Characteristics Patterson, P. G., Johnson, L. W., & Spreng, R. A. (1997). Modeling the determinants of customer satisfaction for business-to-business professional services. Journal of the Academy of Marketing Science , 25, 4--17. Pearson, K. (1900). X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series 5 , 50, 157--175. Pomerol, J. C. (1998). Scenario Development and Practical Decision Making under Uncertainty: Application to Requirements Engineering. Requirements Engineering , 3, 3--4. Regnell, B., & Brinkkemper, S. (2005). Market-driven requirements engineering for software products. Engineering and managing software requirements , 287--308. Regnell, B., Host, M., Natt och Dag, J., Beremark, P., & Hjelm, T. (2001). An Industrial Case Study on Distributed Prioritisation in Market-Driven Requirements Engineering for Packaged Software. Requirements Engineering , 6, 51--62. Robertson, P. K. (1991). A Methodology for Choosing Data Representations. IEEE Computer Graphics and Applications , 11, 56-67. Rodrigues, A., & Williams, T. (1997). System dynamics in software project management: towards the development of a formal integrated framework. European Journal of Information Systems , 6, 51--66. Runeson, P., & Hst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering , 14, 131-164. Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences , 1, 83--98. Saeki, M. (2003). Embedding metrics into information systems development methods: an application of method engineering technique. Conference on Advanced Information Systems Engineering (pp. 374389). Springer-Verlag Berlin Heidelberg. Saliu, O., & Ruhe, G. (2005). Supporting software release planning decisions for evolving systems., (pp. 14--26). Scupin, R. (1997). The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology. Human Organization , 56 (2), 233 - 237. Sheskin, D. (2004). Handbook of parametric and nonparametric statistical procedures. CRC Pr I Llc. Siegel, S., & Castellan, N. J. (1988). Nonparametric statisics for the behavioral sciences (2nd Edition ed.). New York: McGraw-Hill.
Page | E
Requirements Engineer Decision Making Relationships among Decision Characteristics Smirnov, N. (1939). On the estimation of the discrepancy between empirical curves of distribution for two independent samples. Bull. Math. Univ. Moscou , 2, 3--14. Spearman, C. (1904). General Intelligence, Objectively Determined and Measured. The American Journal of Psychology , 15, 201--292. Stevens, S. S. (1946). On the theory of scales of measurement. Science , 103, 677--680. Strigini, L. (1996). Limiting the Dangers of Intuitive Decision Making. IEEE Software , 13, 101--103. Verner, J. M., Sampson, J., Tosic, V., Bakar, N. A., & Kitchenham, B. A. (2009). Guidelines for industriallybased multiple case studies in software engineering. Third International Conference on Research Challenges in Information Science, (pp. 313-324). van der Weerd, I., Brinkkemper, S., & Versendaal, J. (2007). Concepts for Incremental Method Evolution: Empirical Exploraion and Validation in Requirements Management. International Conference on Advance Information Systems (pp. 469-484). Springer-Verlag Berlin Heidelberg. van der Weerd, I., Brinkkemper, S., Nieuwenhuis, R., Versendaal, J., & Bijlsma, L. (2006). Towards a Reference Framework for Software Product Management. International Requirements Engineering Coference (pp. 319-322). IEEE publishing. van der Weerd, I., Brinkkemper, S., Souer, J., & Versendaal, J. (2006). A Situational Implementation Mehod for Web-based Content Management System-applications: Mehod Engineering and Validation in Practice. Software Process Improvement Practices , 11, 521-538. Wnuk, K., Regnell, B., & Karlsson, L. (2009). Feature Transition Charts for Visualization of Cross-Project Scope Evolution in Large-Scale Requirements Engineering for Product Lines. Second International Workshop on Requirements Engineering Visualization. Atlanta. Wnuk, K., Regnell, B., & Karlsson, L. (2008). Visualization of Feature Survival in Platform-Based Embedded Systems Development for Improved Understanding of Scope Dynamics. First International Workshop on Requirements Engineering Visualization (pp. 41-50). Barcelona: IEEE Computer Society. Wnuk, K., Svensson Berntsson, R., & Regnell, B. (2009). Investigating Upstream versus Downstream Decision-Making in Software Product Management. International Workshop on Software Product Management, (pp. 12-15). Atlanta. Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell, B., & Wesslen, A. (2000). Experimentation in software engineering: an introduction. London: Kluwer Academic Publishers. Yeh, A. (1992). Requirements engineering support technique (REQUEST): a marketdriven requirements management process., (pp. 211--223). Yin, R. K. (2003). Case Study Research: Design and Methods. Sage Publications: California. Page | F
Requirements Engineer Decision Making Relationships among Decision Characteristics Zave, P. (1997). Classification of research efforts in requirements engineering. ACM Computing Surveys (CSUR) , 29, 321. Zur, H. B., & Breznitz, S. J. (1981). The effect of time pressure on risky choice behavior. Acta Psychologica , 47, 89--104.
Page | G
14 Appendix
14.1 Research Approach
Data Preparation
Data Analysis
Researcher
ORGANIZED RAW DATASET
[No unclear decisions]
[No unclear decisions] [Else] Is based on 1..*

UNCLEAR DECISION
[Else]

Researcher
1 1
DISCUSSION ITEM
[Else]

Researcher & PM
1
DISCUSSED DECISION
Is based on
Is based on
Is based on [No unclear decisions]
Form Final Decisions

Researcher
1
DECISION
1..* 1..*
Form Final Dataset

Researcher
FINAL DATASET
Figure 28 - Research Approach Data Preparation

Literature Study
Select Relevant Literature Study Literature Form Research Questions
RELEVANT LITERATURE
1..* Is extracted from 0..*

RESEARCH QUESTION
Researcher
0..*
Figure 29 - Research Approach Literature Study
Page | H

Survey
Create Survey Question

Researcher
SURVEY QUESTION
Create Survey
Researcher
[Else]
SURVEY State (published or not)
Validate Survey
Expert
Is based on 1
[Survey in order]
1 Is based on 1
Publish Survey
Researcher
Get Results
Researcher
SURVEY RESULT
QUESTION RESULT Median
Figure 30 - Research Approach Survey

Data Analysis

HYPOTHESIS
1..* Is based on 1
Test Hypotheses
TEST RESULT
Is based on 1
VALIDATED RESULT
Validate Results
Researcher 1..*
CONCLUSION
1..* Is based on 1..*
Draw Conclusion
Is based on 1..*
Give Advise
ADVISE
Figure 31 - Research Approach Data Analysis
Page | I
14.2 Survey
Figure 32 - Survey Introduction
Figure 33 - Survey Ratings
Figure 34 - Survey Thanks
Page | J
Figure 35 - Survey Background
Page | K
14.3 SPSS Output1

Dagostino-Pearson test of normality on the Lead Time:
Run MATRIX procedure: Number of observations: 1439 Number of variables: 1 Measures and tests of skew: g1 sqrt(b1) LeadTime 3,4216 3,4180 Measures and tests of kurtosis: g2 b2-3 LeadTime 16,1280 16,0679
z(b1) 27,3005
p-value ,0000
z(b2) 18,2769
p-value ,0000
Omnibus tests of normality (both chisq, 2 df): D'Agostino & Pearson K sq Jarque & Bera LM test K sq p-value LM p-value LeadTime 1079,3625 ,0000 18281,8084 ,0000 ------ END MATRIX -----
Dagostino-Pearson test of normality on the Log10(Lead Time):

Run MATRIX procedure: Number of observations: 753 Number of variables: 1 Measures and tests of skew: g1 sqrt(b1) LogLeadT ,0055 ,0054 Measures and tests of kurtosis: g2 b2-3 LogLeadT -,6833 -,6868
z(b1) ,0616
p-value ,9509
z(b2) -5,9403
p-value ,0000
Omnibus tests of normality (both chisq, 2 df): D'Agostino & Pearson K sq Jarque & Bera LM test K sq p-value LM p-value LogLeadT 35,2905 ,0000 14,8018 ,0006 ------ END MATRIX -----
SPSS output was generated using SPSS 16.0.1 and PASW Statistics 18.0.0 (SPSS successor)
Page | L
Spearman Correlation H1
Correlations AmOfProdWithout Plat Kendall's tau_b AmOfProdWithoutPlat Correlation Coefficient Sig. (1-tailed) N LogLeadTime Correlation Coefficient Sig. (1-tailed) N Spearman's rho AmOfProdWithoutPlat Correlation Coefficient Sig. (1-tailed) N LogLeadTime Correlation Coefficient Sig. (1-tailed) N **. Correlation is significant at the 0.01 level (1-tailed). 1 Table 10 - (H ): Spearman Correlation test results ,222 ,000 754
**
LogLeadTime 1,000 ,173 . ,000 1439 754 1,000 . 754 1,000 ,222 . ,000 1439 754 1,000 . 754
** **
,173 ,000
**
754
Two-Sample Kolmogorov-Smirnov Test and Spearman Correlation H2

Frequencies Decision AmOfProdWithoutPlat Rejected Accepted Total Table 11 - (H ): Frequency table K-S test Test Statistics
a 2a
N 503 936 1439
AmOfProdWithout Plat Most Extreme Differences Absolute Positive ,030 ,003
Page | M

Negative Kolmogorov-Smirnov Z Asymp. Sig. (2-tailed) a. Grouping Variable: Decision 2a Table 12 - (H ): K-S test results ,545 ,928 -,030
Correlations Number of Products Kendall's tau_b Number of Products Correlation Coefficient Sig. (2-tailed) N Acceptance Rate Correlation Coefficient Sig. (2-tailed) N Spearman's rho Number of Products Correlation Coefficient Sig. (2-tailed) N Acceptance Rate Correlation Coefficient Sig. (2-tailed) N 2b Table 13 - (H ): Spearman correlation test results . 1439 ,008 ,772 . 1439 1439 . 1439 -,017 ,485 . 1439 1,000 1439 ,008 ,772 1439 1,000 1,000 Acceptance Rate -,017 ,485 1439 1,000
Spearman Correlations H3
Correlations LogLeadTime Kendall's tau_b LogLeadTime Correlation Coefficient Sig. (1-tailed) N Release Correlation Coefficient Sig. (1-tailed) N Spearman's rho LogLeadTime Correlation Coefficient ,139 ,000 754 1,000 ,180
** **
Release
**
1,000 ,139 . ,000 754
754 1,000 . 1439
Page | N

Sig. (1-tailed) N Release Correlation Coefficient Sig. (1-tailed) N **. Correlation is significant at the 0.01 level (1-tailed). 3 Table 14 - (H ): Spearman Correlation test results ,180 ,000 754
**
. ,000 754 754 1,000 . 1439
Two-Sample Kolmogorov-Smirnov Test and Spearman Correlation H4

Frequencies Decision Release Rejected Accepted Total
4a
N 503 936 1439
Table 15 - (H ): Frequency table K-S test Test Statistics

a
Release Most Extreme Differences Absolute Positive Negative Kolmogorov-Smirnov Z Asymp. Sig. (2-tailed) a. Grouping Variable: Decision 4a Table 16 - (H ): K-S test results ,000 ,142 ,000 -,142 2,566
Correlations Release Heartbeat Kendall's tau_b Release Heartbeat Correlation Coefficient Sig. (2-tailed) N . 1439 1,000 Acceptance Rate ,921
**
,000 1439
Page | O

Acceptance Rate Correlation Coefficient Sig. (2-tailed) N Spearman's rho Release Heartbeat Correlation Coefficient Sig. (2-tailed) N Acceptance Rate Correlation Coefficient Sig. (2-tailed) N **. Correlation is significant at the 0.01 level (2-tailed). 4b Table 17 - (H ): Spearman correlation test results . 1439 ,969
**
,921
**
1,000
,000 . 1439 1,000 1439 ,969

**
,000 1439 1,000
,000 . 1439 1439
T-Test Results H5
Group Statistics Operator LogLeadTime Not operator related Operator related Table 18 - (H ): T-test group statistics
5
N 497
Mean
Std. Deviation
Std. Error Mean ,01926 ,03003
1,0192 ,42927 ,48149
257 ,9991
Independent Samples Test Levene's Test t-test for Equality of Means 95% Confidence Sig. (2F LogLeadTime Equal variances assumed Equal variances not assumed Table 19 - (H ): T-test results
5
Mean
Std. Error
Interval Upper
Sig.
t ,586
df
tailed) Difference Difference Lower ,02017 ,03440
8,117 ,005
752 ,558
-,04736 ,08770
,565 468,807 ,572
,02017
,03568
-,04994 ,09028
Results H6
Operator * Decision Crosstabulation Count Decision
Page | P
.
Operator Not operator related Operator related Total Table 20 - (H6 ): R x C table results
a
Rejected 380 123 503
Accepted 645 291 936
Total 1025 414 1439
Chi-Square Tests Asymp. Sig. (2Value Pearson Chi-Square Continuity Correction Likelihood Ratio Fisher's Exact Test Linear-by-Linear Association N of Valid Cases 7,027 1439 1 ,008
b
Exact Sig. (2sided)
Exact Sig. (1sided)
df
a
sided) 1 ,008 1 ,010 1 ,008 ,009
7,032
6,711 7,145
,005
a. 0 cells (,0%) have expected count less than 5. The minimum expected count is 144,71. b. Computed only for a 2x2 table 6a Table 21 - (H ): Chi-Square test results
T-Test Results H7
Group Statistics Decision LogLeadTime Rejected Accepted Table 22 - (H ): T-test group statistics
7
N 185
Mean
Std. Deviation
Std. Error Mean ,02996 ,01904
1,1239 ,40750 ,45427
569 ,9761
Independent Samples Test Levene's Test t-test for Equality of Means 95% Confidence Sig. (2F Sig. t df Mean Std. Error Interval Upper
tailed) Difference Difference Lower
Page | Q

LogLeadTime Equal variances assumed Equal variances not assumed Table 23 - (H ): T-test results
7
8,736 ,003
3,940
752 ,000
,14783
,03752
,07418
,22148
4,164 344,508 ,000
,14783
,03550
,07801
,21766
Survey Results
Statistics Relationship Relationship Relationship Relationship Relationship Relationship Relationship ProductsLead Time N Valid Missing Mean Mode Std. Deviation Minimum Maximum Table 24 - Survey Summary 39 7 ,69 1 ,655 -1 1 ReleaseLead Time 39 7 -,33 -1 ,772 -1 1 CustomerLead Time 39 7 -,49 -1 ,721 -1 1 ProductDecision 38 8 -,37 -1 ,714 -1 1 ReleaseDecision 38 8 -,74 -1 ,446 -1 0 CustomerDecision 38 8 ,71 1 ,654 -1 1 Lead TimeDecision 38 8 -,05 0 ,613 -1 1
Relationship Products-Lead Time Cumulative Frequency Valid Shorter No Influence Longer Total Missing System 4 4 31 39 7 Percent 8,7 8,7 67,4 84,8 15,2 Valid Percent 10,3 10,3 79,5 100,0 Percent 10,3 20,5 100,0
Total 46 100,0 Table 25 - Survey Products - Leadtime Frequency Table
Relationship Release-Lead Time Cumulative Frequency Valid Shorter No Influence Longer 20 12 7 Percent 43,5 26,1 15,2 Valid Percent 51,3 30,8 17,9 Percent 51,3 82,1 100,0
Page | R

Total Missing System 39 7 84,8 15,2 100,0
Total 46 100,0 Table 26 - Survey Release - Leadtime Frequency Table
Relationship Customer-Lead Time Cumulative Frequency Valid Shorter No Influence Longer Total Missing System 24 10 5 39 7 Percent 52,2 21,7 10,9 84,8 15,2 Valid Percent 61,5 25,6 12,8 100,0 Percent 61,5 87,2 100,0
Total 46 100,0 Table 27 - Survey Customer - Leadtime Frequency Table
Relationship Product-Decision Cumulative Frequency Valid IncrRej No Influence DecrRej Total Missing System 19 14 5 38 8 Percent 41,3 30,4 10,9 82,6 17,4 Valid Percent 50,0 36,8 13,2 100,0 Percent 50,0 86,8 100,0
Total 46 100,0 Table 28 - Survey Products - Decision Frequency Table
Relationship Release-Decision Cumulative Frequency Valid IncrRej No Influence Total Missing System 28 10 38 8 Percent 60,9 21,7 82,6 17,4 Valid Percent 73,7 26,3 100,0 Percent 73,7 100,0
Total 46 100,0 Table 29 - Survey Release - Decision Frequency Table
Page | S
Relationship Customer-Decision Cumulative Frequency Valid IncrRej No Influence DecrRej Total Missing System 4 3 31 38 8 Percent 8,7 6,5 67,4 82,6 17,4 Valid Percent 10,5 7,9 81,6 100,0 Percent 10,5 18,4 100,0
Total 46 100,0 Table 30 - Survey Customer - Decision Frequency Table
Relationship Lead Time-Decision Cumulative Frequency Valid IncrRej No Influence DecrRej Total Missing System 8 24 6 38 8 Percent 17,4 52,2 13,0 82,6 17,4 Valid Percent 21,1 63,2 15,8 100,0 Percent 21,1 84,2 100,0
Total 46 100,0 Table 31 - Survey Lead Time - Decision Frequency Table
Page | T
15 Resulting papers
From this thesis research two papers resulted that were submitted to international conferences. One paper submitted to the: 18th International Conference on Requirements Engineering called What Decision Characteristics Influence Decision Making in Market Driven Large Scale Requirements Engineering? was rejected. The paper send to the first International Workshop on Product Line Requirements Engineering and Quality called What Decision Characteristics Influence Decision Making in Market Driven Large Scale Software Product Line Development? is currently still pending. Both papers are attached in their original layout on the following pages.
Page | U
Page | V
Page | W
Page | X
Page | Y
Page | Z
Page | AA
Page | BB
Page | CC
Page | DD
Page | EE
Page | FF
Page | GG
Page | HH
Page | II
Page | JJ
Page | KK
Page | LL
Page | MM
Page | NN
Page | OO
Page | PP
Page | QQ

Master Thesis

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Master Thesis

Uploaded by

Copyright:

Available Formats

Master Thesis Name: Student Number: Thesis Number: In fulfillment of:

Jaap Kabbedijk 0444332 INF/SCR-09-65 Master of Business Informatics

Requirements Engineering Decision Making:

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

6.1.1 6.1.2 6.1.3 6.2

Hypotheses ................................................................................................................................. 29 Hypotheses Justification ..................................................................................................... 29 Page | I

6.3.1 6.3.2 6.3.3 6.3.4 6.3.5 6.3.6 7

14.1 14.2 14.3 15

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

Table of Used Tables

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

Conclusion Future Research Acknowledgements Bibliography Appendix

Figure 1 - Content Coherence Overview

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

2.2 Requirements Engineering as a Decision Process

Problem Analysis Management Control Development Phase

Operational Control Selection Phase

Table 1 - Requirements Engineering process mapped on Decision-making

2.3 Research Positioning

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

3 Case Company Description

3.1 Product Line Environment

3.2 Requirements Management Process

Requirements Engineer Decision Making Relationships among Decision Characteristics

3.3 Product Line Scoping

Requirements Engineer Decision Making Relationships among Decision Characteristics

File Change Request

CHANGE REQUEST ID Description Decision Outcome ...

Quality Gateway Analyze Completeness

Analyze Similarity Change Control Board

[else] [sufficient quality] Perform Impact Analysis

Decide upon acceptance

Reject Change Request

Accept Change Request

1..* Create Requirement

REQUIREMENT Id Name Status Source

FEATURE Name Description Justification Scope Market Value Effort Estimate

Figure 3 - Change Request Process

3.4 Release Management

3.5 Market-Driven Nature

Requirements Engineer Decision Making Relationships among Decision Characteristics

Requirements Engineer Decision Making Relationships among Decision Characteristics

4.2 Main Question

4.3 Sub Questions

Each of these questions is answered before the main question is answered.

4.4 Relevance of Research Question

4.4.1 Academic Relevance

4.4.2 Business Relevance

Requirements Engineer Decision Making Relationships among Decision Characteristics

FINAL DATA SET

1 Is source for 1..* Literature Study

1..* Leads to 1..* Survey

1 1..* Give Advise Is based on

Figure 4 - Research Approach Overview

Requirements Engineer Decision Making Relationships among Decision Characteristics

ORGANIZED RAW DATASET

1 [No unclear decisions] [Else] Is based on 1..*

Select Unclear Decisions