CSTM05 Ver4.0

STQC IT Services , Bangalore
TEST MANAGEMENT
CSTM Ver 4.0
Module # 5 Test Management
Contents
The Test Management - Basic Principles The Test Process a. Set Test Objectives b. Develop Test Plan c. Execute Tests d. Summarize and Report Results Test Estimation & Forecasting Test Tracking- Use of S curves Defect Tracking People Issues in software testing
CSTM Ver 4.0
The Test Management Challenge

How do you Determine the current quality of your application under test ? Define, measure, and track quality goals? Coordinate efforts of all those involved in the testing effort? Track dependencies and relationships among test assets? Plan, execute, and evaluate your testing iteratively?
CSTM Ver 4.0
Why Test Management?

100% testing is not possible The domain of possible inputs is too large to test. There are too many possible paths through the program to test. The user interface issues are too complex to completely test. Limited time (reduced cycle time) and resources.
CSTM Ver 4.0
Inputs and Activities of Test Planning and Design

System specifications Requirements Change requests Risk list Iteration plan Design specifications Use-case model
Agree Mission Identify Test Team Identify Targets of Test Define Assessment and Traceability Needs Define Test Approach Identify Test Ideas Define Test Details Define Test Environment Configurations Obtain Testability Commitment
Test Plan
Test Approach
Test Case
Test Automation Architecture
CSTM Ver 4.0
Test Manager
Roles
Test Manager
Responsible for
Activitie s
Prepare test plan
Artifacts
Test Plan
CSTM Ver 4.0 Module # 5 Test Management 6
The Test Process

1. Set Test Objectives 2. Develop Test Plan 3. Execute Tests 4. Summarize and Report Results
CSTM Ver 4.0
Step 1 - Set Test Objectives Define Test Objectives and Test Strategy
A test objective is what the test is to validate. eg Performance, functionality etc Test Strategy is an approach depending on size, criticality etc
CSTM Ver 4.0
Determine Test Strategy

What Type of Projects?
How Critical is the system to the Organization? What are the Tradeoffs?
What Type of Software? What Type of Technical Environment? What is the Projects Scope?
Determin e Testing Strategy
Who Will Conduct Testing? What are the Critical Success Factors?
CSTM Ver 4.0
When Will Testing Occur?

9
So little time, so much to test ..

test time will always be limited use RISK to determine: what to test first what to test most what not to test (this time) use RISK to allocate the time available for testing by prioritising testing ...
CSTM Ver 4.0
10
Risk Analysis
Risk based test strategy will be fruitful if all risks are identified at an early stage. Risk Exposure = Probability of failure * Impact
CSTM Ver 4.0
11
Risk Structure
RISK
Impact ( Cost of Failure) Probability of failure
Usage Frequency
Consequence
Quality (Defect Density / Failure Rate)

12
CSTM Ver 4.0
Probability of failure can be due to:

Complex functions involved. Completely new functions, which the testing team is not aware of. Functions for which certain tools or techniques were employed for the first time. Developers lack domain experience. Insufficient Quality Assurance. Impact of No of people involved, Turnover, Time pressure ,geographical distribution etc
CSTM Ver 4.0
13
Risk Assessment
CSTM Ver 4.0
14
Risk based Testing
1. Test for major features or functionalities that would have major set-back on customer, when it is not properly tested. 2. Test for major features or functionalities , the failure of which would result in losing credibility and market share, or missing a market window or in incurring failure costs 3. Record any features that are not tested and the reasons for not testing.
Module # 5 Test Management 15
CSTM Ver 4.0
Risk based Testing contd

4. Check for result of missing important faults or of over-testing/ under testing/ ineffective testing 5. Outlining software risks help the testers prioritize what to test and allows them to concentrate on those areas that are likely to fail or have a large impact on the customer if they do fail.
CSTM Ver 4.0
16
Challenges of Risk Analysis

Consensus on risk Avoiding priority inflation Making rational decisions about risk Investing the time Categorization is not quantification
CSTM Ver 4.0
17
Methods of Risk Analysis

Informal - Judgment and Intuition Formal - Identifying and weighing risk attributes
CSTM Ver 4.0
18
STQC Identifying the Risks to System Quality IT Services , Bangalore
Informal
Start with the Classic Quality risk categories Functionality, states and Transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc. Set priority for testing each quality risk with key stakeholders OR Apply Risk Lists
CSTM Ver 4.0
ISO 9126 Start with six main quality Characteristics Functionality, Reliability, Usability, Efficiency, Maintainability, Portability (FRUEMP), then decompose into key sub characteristics for your system Set priority for testing each sub-characteristic with key stakeholders
FMEA Start with categories, characteristics, or subsystems Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number (RPN) Stakeholders use RPN to guide appropriate depth and breadth for testing
19
Informal :RISK LISTS

Possible ranking criteria (all risk based) test where a failure would be most severe, most visible, most likely ask the customer to prioritise the requirements what is most critical to the customers business areas changed most often areas with most problems in the past most complex areas, or technically critical
CSTM Ver 4.0
20
FMEA- Failure Mode and Effect analysis

Technique for understanding and prioritizing quality risks in: System functions Attributes Components Interfaces Studied in terms of: Severity Potential cause of failure Priority Detection methods Likelihood or vulnerability
Potential cause of failure - Possible factors that may trigger the failure like operating system, operator error or normal use Detection Methods Existing methods or procedure that can detect the problems Reviews, customer testing etc
CSTM Ver 4.0
22
Severity
Scale Description IEEE 1044 class Urgent High Class Number
1 2 3 4 5
CSTM Ver 4.0
Loss of Data, hardware damage or safety issue Loss of functionality without a workaround Loss of functionality with a work around Partial loss of functionality Cosmetic or trivial
IM110 IM120
Medium IM130 Low None IM140 IM150

23
Priority
Scale Description
1 2 3 4 5
Complete loss of system value Unacceptable loss of system value Possibly reduction in system value Acceptable reduction in system value Negligible reduction in system value
IEEE Class Number 1044 class Priceless IM310 High Medium Low None IM320 IM330 IM340 IM350
CSTM Ver 4.0
24
Likelihood
Scale Description IEEE 1044 class Urgent High
Class Number
1 2 3 4 5
CSTM Ver 4.0
Certain too affect all users Likely to impact some users Possible impact on some users Limited impact to few users Unimaginable in actual usage
IM210 IM220
Medium IM230 Low None IM240 IM250

25
FMEA
Feature GIS Failure Mode Functio n limits Effect Critical Sev erity 2 Potential Cause Memory size Protocol impleme ntation Priority Detection Method 3 Customer trials FAT Likeli hood 2 RPN 12
Limited Y Drawin g ability Positio Y n not determi ned
GPS Driver Interface failure
Min RPN = 1 (high Risk) Max RPN=125(min risk)
CSTM Ver 4.0
26
FMEA
Test Type Load Failure Mode Scaling of system to peak concurrent users Scaling of system to peak concurrent users Effect System fails at 20 concurrent users System fails at 200 concurrent users Severity 1 1
Priority
Likeliho od 3
RPN 3
Load
Volume
Scaling of System fails system to to handle handle large documents volumes of data larger than 100kB
Min RPN = 1 (high Risk) Max RPN=125(min risk)

FMEA and Degree of testing

Software testing expanse: Extensive Testing- Breadth and Depth Balanced Testing- Breadth Opportunity Testing - Light weight Testing Anomaly reporting only
CSTM Ver 4.0
28
eg of RPN driven Testing

Degree of Testing assigned based on FMEA RPN 1-10 Extensive testing RPN 11-30 Balanced testing RPN 31-70 Opportunity Testing RPN > 70 No testing, only anomaly reporting
CSTM Ver 4.0
29
Breakdown of Risk Based Testing

Critical test cases Important test cases Low yield test cases 10 - 15% 15 - 25% 60 - 75%
Most important 15% of test cases find over 75% of the defects.
CSTM Ver 4.0
30
Test Effort Estimation
CSTM Ver 4.0
31
When will Testing Finish ?

A good estimate 1. Accurately predicts and guides the projects future 2. Realistic: All tasks included, accurately estimated, risks understood and mitigated 3. Actionable: Clear ownership by committed individual contributors, assigned resources, known dependencies
Developing a Work-Breakdown-Structure The major stages of a testing subproject 1. Planning 2. Staffing (if applicable) 3. Test environment acquisition and configuration 4. Test development 5. Test execution Break down to discrete tasks within each stage Tasks should be short in duration (e.g., a few days) Assign Dependencies Analyze Critical Paths Estimate time taken for each task
CSTM Ver 4.0
Testing WBS- an example

S No. 1 1a 1b 1c 2 2a 2b 2c 3 3a 3b 3c 4 4a 4b 4c 4d . . Task Name Planning Analyze Risks Prepare Estimate Write Test plan Test Development Write Functional tests Write Usability tests Write reliability tests .. Test Environment Configuration Windows 2000 Linux .. System test Execution Entry Criteria System Test Entry Meeting Test pass 1 Test Pass 2 Acceptance Test pass System Test exit Meeting Exit Criteria Met Start Date
End date

Pre task
CSTM Ver 4.0
34
Methods of Test Effort Estimation Informal

Tapping the teams wisdom
Formal
Test Point Analysis Use Case Point Analysis
CSTM Ver 4.0
35
Tapping the Teams Wisdom for Estimation

Delphic Oracle Technique Gather estimates from each team member Ask high and low estimators to explain estimates Repeat twice, then use average

Three-Point Technique Have team estimate bestcase, worst-case, and expected-case
Use expected-case Variances between best and Worst useful to gauge estimate accuracy
CSTM Ver 4.0
36
Test Point Analysis

1.
An extension of function point analysis to high-level (system and Acceptance) testing
2. Uses ISO 9126 quality characteristics analysis 3. Test points are calculated based on.. Size (function points adjusted for complexity, interfaces, and uniformity) Test strategy (which quality characteristics or risks are to be tested and to what extent) Productivity (skills of the test team, influenced by the project, process, Technology, and organization)
CSTM Ver 4.0
37
Dynamic Quality Characteristics

Consider dynamic quality characteristics with weights totalling100%: Functionality Usability . Give a rating for each characteristic on a scale :
0 (quality requirements not important) 3 (quality requirements relatively unimportant but for purpose of testing to be considered) 4 (quality requirements of normal importance) 5 (quality requirements very important) 6 (quality requirements extremely important)
Dynamic quality characteristics (Qd)

Depending on the rating of the characteristics suitable degree of rigor is introduced into testing Eg If characteristics are functionality(0.75), Security(0.05),usability(0.10) and efficiency(0.10) Qd=(.75Qf+.05Qs+.1Qu+.1Qe)/4
CSTM Ver 4.0
39
Function dependent factors (Df)

1. User Importance (Ue) with following weights:
3- Low importance of function 6- Normal importance of function 12 High importance of function
2. Usage Intensity (Ui) with the following weights:

2- Low used a few times a week 4- Normal used a number of times a day 12- High used continuously throughout the day
CSTM Ver 4.0
40

3. Interfacing (I) with following weights: 2- Low degree of interfacing with function 4- Normal degree of interfacing with this function 8- High degree of interfacing with this function 4. Complexity (C) with the following weights: 3- Function has no more than 5 conditions 6- Function contains between 6 and 11 conditions 12- Function contains more than 11 conditions
CSTM Ver 4.0
41

5. Uniformity (U): 0.6 for cases where test specifications can be largely reused because of similar or clone functions 1.0 if function is unique in terms of processing or a new data set
Method of calculation: Df=((Ue+Ui+I+C)/20)*U

Dynamic Test Points

TPf = FPf *Df * Qd TPf Number of test points assigned to function FPf- number of function points assigned to the function Df Function dependent factors Qd- Quality characteristics factor
CSTM Ver 4.0
43
Static Test Points

Determine if any quality characteristic has been statically tested by using a checklist for this function For each quality characteristic give a value of 16 Qs= 16* Qi char i=1,n
CSTM Ver 4.0
44
Calculate Test Points

Dynamic quality Characteristics Qd=(.75Qf+.05Qs+.1Qu+.1Qe)/4 Function-dependent factors Df=((Ue+Ui+I+C)/20)*U
Uniformity 0.6 or 1
Importance of functionality, security, usability, efficiency, 0,3,4,5, or6.
Dynamic test points TPf=Qd Df FPf

Function point count for function under consideration
Complexity, 3,6, or 12.
Interfacing,2,4, or 6. Usage intensity, 2,4,or 12 Usage importance 3,6, or 12
Static test points (Qs=SumQi) 16 per statically tested quality characteristic. Total test points Total for all statically tested quality TP=Sum(TPf)+(FPQs)/500 characteristics, 0-96. Function point count for entire system,minimum 500.
Primary Test Hours

Productivity factor Number of test hours required per test point Measure of experience, Knowledge and skill sets of test team In practice between 0.7 to 2.0(P) Environmental factors 1. Test Tools
Rating of 1,2 or 4 depending on maturity in usage of testing tools from the highest to none
Environmental factors contd..

Development Testing Reflects quality of product and less testing may be required Rating of 2,4,8 depending on availability of development test plan and test cases Development Environment Degree to which development environment has prevented error injection Rating of 2,4 or 8 dependent on rigorous procedures followed in development cycle to an immature process
Environmental Factors contd..

Test Basis Reflection of system documentation available. Rating of 3,6 and 12 dependent on quality of system documentation available from reviewed documentation as per Standards to availability of some documentation. Test Environment Extent to which the test infrastructure has been used before. Ratings of 1,2 and 4 reflect a well tried out environment to a newly equipped environment
CSTM Ver 4.0
48
Environmental factors contd..

Testware:
This reflects as to what extent the existing tests and other stubs and drivers can be reused. The ratings are 1,2 and 4 starting with test ware readily available to none available. E=Tt+ Dt +Tb +De +Te +Tw /21
Primary Test Hours PT=TP*P*E

Calculate Test Hours

Productivity factor P
Based on historical data, 0.7-2.0.
Environmental factors E=Tt+Dt+Tb+De+Te+Tw /21
Test ware, 1, 2, or 4
Primary test hours PT=P*E*TP

Total test points from previous step
Test environment, 1, 2 or 4. Development environment, 3, 4, or 8. Test basis, 3, 6, or 12 Development testing 2, 4, or 8. Test tools, 1, 2, or 4.
Total test hours TH=PT+MO
Total test hours Management overhead factor Breakdownn: MO=PT*(Ts+Mt)/100 10%preparation (plan, design) 40% specification Management (test dev) 45% execution tools, 2,4, or 8 (run, report) 5% completion (final report) Test team size, 3, 6 or 12.
CSTM Ver 4.0
Use Case based Test effort Estimation

Based on Use case definitions Depends on Complexity of Actors and Use cases Accounts also for technical complexity factors Refer article
CSTM Ver 4.0
51
Step 2 - Develop Test Plan

Test Planning is the process to proactively decide what the important issues are in testing and how to best deal with them. The artifact produced as a result of this activity is a Test Plan.
CSTM Ver 4.0
52
The Test Plan

Outlines the goals and objectives of testing within the scope of the iteration or project. Identifies software and hardware items being targeted. Outlines the tests to be performed. Identifies approach to be taken (strategy). Estimates the required resources. Outlines the schedule and milestones. Identifies key deliverables.
Test Plan
CSTM Ver 4.0
Test Plan When to begin?

Begin developing test plan at the same time as the requirements document and project plans are being developed. Acceptance test plans can be started as soon as the requirement definition process has begun.
CSTM Ver 4.0
54
How Much Time Should be Spent on Testing?

Many organizations report spend one third to one-half of a projects time in test related activities.
33%
Development Testing
67%
CSTM Ver 4.0
55
Planning Time Guidelines

The total test time, can be allocated each of the following activities:
Test planning Test Specification Test execution Test Completion
5%
10%
Planning Specification Execution Completion
45%
40%
CSTM Ver 4.0
56
Test Plan (IEEE 610.12-1990)

A document describing the scope, approach, resources and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task and any risks requiring contingency planning.
CSTM Ver 4.0
57
Test Plan Outline (IEEE 829:1998)

1.
2. 3. 4. 5. 6. 7. 8.
CSTM Ver 4.0
Test Plan Identifier Introduction Test items Features to be tested Features not to be tested Approach Item pass / fail criteria Suspension criteria and resumption requirements
Test Plan Outline (IEEE 829:1998)

(contd.)
9. Test deliverables 10.Testing tasks 11.Environmental needs. 12.Responsibilities 13.Staffing and training needs 14.Schedule 15.Risks & Contingencies 16.Approvals
Test Plan Identifier
Specify unique identifier assigned to this test plan.
CSTM Ver 4.0
60
Introduction
Objectives Background Scope Reference
CSTM Ver 4.0
61
Test Items
Identify the items to be tested with version. Supply reference to the test item documents like requirements specification, design specifications, user guide, operators guide, installation guide
Features to be tested
Identify all the software features to be tested. Identify test design specification associated with each feature.
Features not to be tested

Identify all the features that will not be tested and the reasons
CSTM Ver 4.0
63
Approach
Describe overall approach to testing that will ensure the adequacy of testing. Approach should be designed in details to identify the major testing tasks and estimation of time requirements. Comprehensiveness Constraints
CSTM Ver 4.0
64
Item pass / fail criteria

Specify the criteria to be used to determine whether each test item has passed or failed. Suspension criteria and resumption requirements . Specify the criteria to be used to suspend all or portion of the testing activity on the test items.
CSTM Ver 4.0
Specify the testing activities that must be repeated when testing is resumed.
Test Deliverables
Identify the deliverable documents which include
CSTM Ver 4.0
test plan test design specifications test case specifications test procedure specifications test logs test incident reports test summary reports
Testing Tasks
Identify the set of tasks necessary to prepare for and perform testing Identify all the inter-task dependencies and any skill requirements.
CSTM Ver 4.0
67
Environmental Needs
Specify the facilities required for testing in terms of hardware, software, communication lines. Identify any special tools required.
CSTM Ver 4.0
68
Test environments are the facilities built up around the system under test to assist in the testing process.The test environment includes: the system or components under test test documentation, describing the tests to be performed and recording results test harnesses and test stubs, used to replace parts of the system that are not yet available or cannot be used to control the system for the necessary tests test oracles, used to predict or check outputs of test cases test data, including data files that are used as inputs in test cases Module # 5 Test Management 69
Test Environments
CSTM Ver 4.0
Responsibilities
Identify the groups responsible for managing, designing, preparing, executing, witnessing, checking and resolving. The groups may include developers, testers, user representatives etc.
CSTM Ver 4.0
70
Staffing and Training needs
Specify test staffing needs and skill set. Identify training options for providing necessary skills.
CSTM Ver 4.0
71
Schedule
Include test milestones. Estimate time required to do each testing. Specify schedule for each testing task and test milestones. Specify the testing resource usage periods.
CSTM Ver 4.0
72
Scheduling Test Activities

Key scheduling objectives Provide predictability to the project manager Identify opportunities to pull in or protect the project schedule Be fair to your staff Maximize productivity
Scheduling Test Activities cont

1. 2. 3. 4. Define the tasks to be undertaken Identify dependencies Identify resources required Identify critical paths through the schedule 5. Analyze risks in schedule and propose contingencies
CSTM Ver 4.0
74
Understanding Dependencies
Assign Dependencies 1. Identify tasks with no predecessors 2. Identify tasks dependent only on previously dependent tasks 3. Repeat step 2 until all dependencies have been identified
Analyze Critical paths A set of dependent tasks where delay in any task delays the project 1. What affects phase entry and exit criteria 2. During test execution how many test passes, releases and cycles are there? 3. External dependencies??
Risks & Contingencies

Identify the high-risk assumptions of the test plan. Specify contingency plans for each.
CSTM Ver 4.0
76
Approvals
Specify the names and titles of all persons who must approve this plan.
CSTM Ver 4.0
77
Step 3 - Execute Tests

Select test tools - A test tool is any vehicle that assists in testing. - May be manual or automated Develop test cases Execute tests
CSTM Ver 4.0
78
The Test Toolbox

Data Modeling Boundary Value Analysis Capture/Playback Cause-Effect Graphing Change Control Trackers Checklists Checkpoint Review Code Comparison Compiler-based Analysis Complexity-based Analysis
Compliance Checkers Control Flow Analysis Correctness Proofs Coverage Based Analysis Data Dictionary Decision Tables Defect Trackers Desk Checking Equivalence Partitioning Error Guessing
79
CSTM Ver 4.0
Types of Testing
Static and Dynamic Testing Functional (Black-Box) Testing Structural (White-Box)Testing Module (Unit) Testing Integration Testing Interface Testing System Testing Acceptance Testing Performance, Load and Stress Testing Usability Testing Reliability Testing Security Testing Recovery Testing Installation Testing Configuration Testing Compatibility/Conversion Testing Serviceability Testing Alpha and Beta Testing Regression Testing Documentation Testing Mutation Testing
CSTM Ver 4.0
80
Test Case Execution: Sample based

Execute a small baseline group of randomly selected tests Calculate pass % (P) Calculate size of forecasting subset group Execute randomly selected tests Forecast by extrapolation
CSTM Ver 4.0 Module # 5 Test Management
N = [Z2P*(1-P)]/E**2
Z is value based on confidence level (65%, 95%, 99%) representing 1W, 2W, 3W. E margin of error allowable.
% fail E
81
Step 4 - Summarize and Report Test Results

Occurs throughout the testing life cycle Tracks testing progress Keeps management informed of testing progress
CSTM Ver 4.0
82
Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings and Recommendations Task 4 - Finalize Test Report Task 5 - Test Report Quality Control
Test Defect Report Test Defect List Test Defect Log
Test Progress Tracking
CSTM Ver 4.0
84
Test Tracking
S Curve Plan, attempted, actual X axis representing time and y axis representing number of test cases or test points Data is cumulative having a steep planned test ramp up Planned progress over time in terms of number of test cases to be completed successfully week by week Number of test cases attempted by week Number of test cases completed successfully by week
Testing Progress S curve example

160
T C s / F a i l u r e s
140
120
100
80
60
40
20
0 Attempted Successful Planned
Days
Test Progress tracking Plan, attempted, successful

Project Plan to Date 232 1175 708 %attempted of Plan %successful Plan not of plan attempted
Project A Project B Project C
6.47 66.98 87.29
6.47 65.62 84.46
217 388 90
CSTM Ver 4.0
87
Defect Reporting & Tracking
CSTM Ver 4.0
88
Why Defect Tracker?

Facilitates clear communication about defects Using a database allows automatic bug numbering Provides analysis and reporting options Allows development team to fix problems based on importance to project and management Aids management of defects in the lifecycle from initial report to disposition Allows capture of history and future reference Brings in new information of the product/project to testers, developers and managers Every defect report that is closed brings you nearer to a quality product. Each report that is not closed prepares you for technical support.
CSTM Ver 4.0
Defect tracking system

Simple defect tracking system running on a single system Browser based web enabled defect tracking system on the Corporate Intranet or on Internet
CSTM Ver 4.0
90
How to write a good defect report?

Be a systematic tester- test thoughtfully and as per a plan. Dont be ad hoc in testing Try to reproduce the failure at least 3 times. If problem is intermittent then report that. Try to isolate the defect by changing data, configuration etc Look for other places in the software where similar defects would be found Write lucid, simple defect reports that bring out the problem in a clear manner. This will help in defect review meetings Leave out any unnecessary details or conjectures Express yourself impartially making no judgments or sarcasm Have a peer review before submitting a defect report
Elements of a defect tracker DB

Project Name/ID Version No/Build no Feature tested Phase in which detected Defect ID Tester name Date opened Defect details State Steps to reproduce Attachments if any Severity Resolution priority Date closed
92
CSTM Ver 4.0
Defect States
Defects can be in different states as follows: Review when a tester enters the defect for first time it is kept in the dB for review Reject when reviewer rejects the defect report for want of details it is sent back to tester Open- If tester has fully characterized the defect the reviewer opens the report making it visible to the world Assigned- The appropriate project team member is assigned the defect for correction
CSTM Ver 4.0
93
Defect Tracking
Test- Once development provides a fix it enters the test state Reopened- If fix fails the confirmation testing the tester reopens the defect report and if it fails the regression test the tester opens a new defect report. Closed- If the fix passes the confirmation test the tester closes the defect report Deferred- If the assigned team member decides that the problem is real but chooses to defer the fix then it is in the deferred state
CSTM Ver 4.0
94
Defect report Lifecycle
Review
Open
assigned
Test
Closed
Reopened Rejected Deferred
CSTM Ver 4.0
95
Further analysis using Defect reports.

Defect in relation to: Subsystem, configuration Quality Root cause Life of defect Defect removal models in terms of injection and detection
CSTM Ver 4.0
96
CSTM Ver 4.0
97
Software Testing: Metrics

Product Metrics Process Metrics Progress Metrics
CSTM Ver 4.0
98
Product Metrics: Functionality Metrics

Functional Adequacy= 1-A/B where A - Number of functions in which problems are detected in testing and B- Number of functions tested Functional implementation correctness = 1-A/B where A- Number of missing functions and BNumber of functions in SRS Functional Implementation Coverage = 1-A/B where A- Number of incorrectly implemented or missing functions detected in Testing and B- number of functions described in SRS
CSTM Ver 4.0
99
Product Metrics: Reliability

Failure Density against test cases = A/B where A- number of detected failures and BNumber of performed test cases Failure Resolution-A/B where A- number of resolved failures and B total number of actually detected failures Fault Removal= A/B where A- number of corrected faults and B- Total number of detected faults during testing
Product Metrics: Reliability MTBF= T1/A or T2/A where A - number of actually detected failure occurrences and T1operation time and T2- sum of intervals between successive failure occurrences. Test Coverage=A/B where A- Number of actually performed test cases representing operation scenario and B- Number of test cases to be performed.
Product Metrics: Time behavior Response Time =T (time of gaining the resulttime of command entry finish) Average Response time Worst Case Response Time Throughput = A/T where A- Number of completed tasks and T observation time period Average Throughput
Product Metrics :coverage Statement Coverage Decision Coverage Condition Coverage Multiple condition Coverage
Such coverage metrics are specified for critical applications both in terms of what metric is to be calculated and their level of compliance.
CSTM Ver 4.0
103
Process Metric
Defect Removal Effectiveness= Defects removed in a Development Phase/Defects latent in the product eg Number of defects detected in System Test (20) Defects found after 6 months in operation(5) DRE=20/25*100= 80%
CSTM Ver 4.0
104
Progress Metrics
Effort Variance %= (actual effort-base lined estimate) /base lined estimate*100 Anomaly find rate Plot of defects found say weekly from beginning of testing to release (generally a bell shaped curve) Anomaly fix rate Anomalies should be fixed as soon as they arrive. The anomaly fix rate can be plotted by week
Progress Metrics contd..

Outstanding anomalies rate- Total anomalies found total anomalies fixed by week Priority Outstanding Rate- Urgent and High priority anomalies found- urgent and high anomalies fixed by week Anomaly classification trends- Plot of different categories of anomalies by priority by week
CSTM Ver 4.0
106
People Issues in Software Testing
CSTM Ver 4.0
107
Test Manager characteristics

Have an overall perspective of the application under test. Broaden vision by understanding the entire functionality Knowledge of the system helps a Manager to suggest valid improvements to the system, direct the performance of meaningful tests. Leadership capabilities by extending help to test team, and defend the defects found by them Build a team with an attitude to crack the system Address both the urgent and the immediate:
urgent matters require immediate attention eg inability to install Important matters require focused and deliberate attention eg improvement to process
A diplomatic interface between the test team and the Project :

While defects mean good news to testers for everyone else its bad news!
Roles and responsibilities

Test Manager? Indicate unit, integration or system testing Conduct SRS review, design review code review QA- defect detection and prevention Director of Quality/Chief Quality Officer? Corporation wide management of product and service quality Sufficient authority and standing to make processes for the entire organization
Model 1
Development Manager Test manager Developer Developer
Difficult to preserve notion of independence Difficult to get test resources Development manager has overall responsibility and forced to choose between develops & testers
CSTM Ver 4.0
110
Model 2
Project manager Test manager Development manager
Test Manager still not independent Project Manager is less involved in in creation of tests but his interests are still with development Development and test organizations have different budget heads and eases the problem of resources to an extent Still expect to participate in all panics and rushed schedules
CSTM Ver 4.0
Model 3
Executive management Test manager Project Manager Project Manager
Test Team is truly independent Management heeds test status reports with an open mind Budget and staffing problems are minimized Still pressures of release will remain but are controllable
Felder Silverman learning Styles Model
People have different learning styles as per FA model: Sensory/intuitive- those who derive information from external senses while the latter derives it from intuition Sensory Testers pay attention to detail prefer problems with well defined standard solutions Sensory Testers are generally good at regression testing The intuitive tester is good at abstraction, and often innovative and imaginative Intuitive learners are bored by repetition They are to be preferred for exploratory testing or trying new techniques
CSTM Ver 4.0
113
Visual/Verbal
Visual testers have a strong preference for the visual and base their tests on UML diagrams, Sequence diagrams and other visual forms. The application generally runs like a movie in their heads and they are therefore good at scenario based tests and test automation Verbal testers prefer textual representations from where they draw their tests
Inductive/Deductive
Inductive testers gather specifics like techniques, defect history, change history and generalize them to the application under test Deductive testers keep a collection of traditional techniques and determine how to apply to the specific domain. Deductive testers gain familiarity with classification methods and then do their classification while inductive testers find new methods of categorizing them.
CSTM Ver 4.0
115
Active/Reflective
Active testes generally do hands-on testing and rapidly perform many test cases. She prefers to work in a group Reflective testers is apt to do fewer tests as a thought process precedes each test They make up for lack of speed by doing good tests that others are not likely to find. They generally like to work alone
Sequential/Global
A sequential tester starts of at a faster pace and goes step by step in the plan. He will be able to explain his tests with clarity A global tester starts of late and may require some hand holing in understanding the application However after the initial phase he will be able to quickly create detailed complex tests.
Learning Styles & the Test manager

The Test manager needs to be able to discern the styles of his team This aids in developing lessons that will enhance their plus points and aid in ridding their weaknesses Test managers will be able to determine the gaps in their teams and see what types of skills they need to balance their team
Summary
CSTM Ver 4.0
119
The Top 10 Testing Problems

10. Lack of skill/ training 9. Developers vs. Testers mentality 8. Lack of test tools 7. Lack of management understanding/support of testing 6. Lack of customer and user involvement 5. Lack of enough time for testing 4. Over-reliance on independent testers 3. Rapid change 2. Blame on testers for delay/ Quality problems 1. Lack of management responsibility
CSTM Ver 4.0
Best Practices for Control/Test Management Processes

Risk
analysis Project customization Defect profiles
Dashboards
Level 5 Preventive Management
Root cause analysis Statistical analysis

Defect
Level 4 - Statistical Process Control
database Defect reporting Defect analysis Level 3 - Defect Management analyzers Walkthroughs Inspections Acceptance test Unit test Integration test System test Ver 4.0
Code
Level 2 - Verification
Level 1 - Validation
CSTM
What do we achieve
Repeated success in testing activity over subsequent projects, based on metrics collected and lessons learnt. Early detection of errors, at lower cost. Focussed testing effort (as a result of risk assessment and prioritization) More comprehensive testing by using automation.
Satisfied customer.

CSTM05 Ver4.0

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CSTM05 Ver4.0

Uploaded by

Copyright:

Available Formats

STQC IT Services , Bangalore

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

The Test Management Challenge

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Why Test Management?

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Inputs and Activities of Test Planning and Design

Test Automation Architecture

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

STQC IT Services , Bangalore

The Test Process

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Determine Test Strategy

Determin e Testing Strategy

When Will Testing Occur?

Module # 5 Test Management

STQC IT Services , Bangalore

So little time, so much to test ..

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Quality (Defect Density / Failure Rate)

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Probability of failure can be due to:

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

CSTM Ver 4.0

Module # 5 Test Management

Risk based Testing

STQC IT Services , Bangalore

CSTM Ver 4.0

STQC IT Services , Bangalore

Risk based Testing contd

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore

Challenges of Risk Analysis

CSTM Ver 4.0

Module # 5 Test Management

STQC IT Services , Bangalore