performance of teams, including

outsourced testing
Managing testing

From the authors of:

1
 2

What you will learn:

ƒ In advance of testing
ƒ Role of Reviews
ƒ Simulation
ƒ Testing Scope
ƒ Software
ƒ Hardware
ƒ Testing outsourcing
ƒ Risks and mitigation
 3

Technical Reviews
ƒ Specification reviews
ƒ Code reviews
ƒ Walk through
ƒ Test plan reviews
ƒ Test case reviews
 4

Test Deliverables and

Metrics
ƒ Integration into development process

ƒ Comprehensive to the development process

ƒ Project phases
ƒ Test Evaluation Master Plan

ƒ Design for testability

 5

Testing Preparation
ƒ Understand REAL demands on a Product (PD&D article)
ƒ www.pddnet.com/scripts/ShowPR~RID~23252.asp
ƒ www.pddnet.com/Scripts/ShowPR~RID~23304.asp

ƒ Human expertise

ƒ Tools

ƒ Signal (features)

ƒ Noise (external influences)

ƒ Communications
 6

Simulation and test

(Evaluation)
ƒ Support or refute design or concept direction
ƒ Reduces reliance or hardware and software
availability (dependence)
 7

Technical Reviews
ƒ Specification reviews
ƒ Code reviews
ƒ Walk through
ƒ Test plan reviews
ƒ Test case reviews
 8

Test metrics
ƒ Incoming fault rates (faults per sw/hw release)

ƒ Categories of faults found

ƒ Specification inconsistencies
ƒ Quality perception
ƒ Severe impact

ƒ Time to fault closure (time to correct faults)

ƒ Reliability
 9

Test metrics (continued)

ƒ Faults found not during testing

ƒ Testability
ƒ Specifications written able to generate test criteria
ƒ System or component designed to be able to test

ƒ Performance to invalid inputs

 10

Test statistics
ƒ Model defect arrival
rates against
Rayleigh distribution
ƒ Establish a defect
containment matrix
to see if defects
trapped between
releases
 11

Test Artifacts
Lab Testing
Sample parts Scrap

Test plan Test report

 12

Test Artifacts
Production Line Testing
Production parts Scrap

Parts passed

Test plan Test results stored

 13

Product Requirements
ƒ Compliance testing (against specs)
ƒ Extreme testing (challenge the product)
ƒ Combinatorial testing (cover the product)
ƒ Stochastic testing (find new ways to
torture the product)
 14

Span of Tests
ƒ Software
ƒ module test (unexpected inputs)
ƒ build tests
ƒ regression testing
ƒ Component test

ƒ System
ƒ Sub-system test
ƒ System Integration test
ƒ System “live test”
ƒ Production test
 15

Span of Tests (continued)

Quality Council of Indiana

ƒ Static analysis
ƒ Static tests
ƒ Structural Analysis
ƒ Dynamic tests
ƒ Mathematical evaluation
ƒ Simulation
ƒ Prototyping and simulation
 16

White box testing

ƒIntimate knowledge of outputs, inputs,
and the “box”
ƒ In short, we should know how the
transformation of inputsÆoutputs occurs
ƒ May be modeled as sets of inputs and
outputs inside the “box”
 17

Black box testing

ƒ We know the inputs
ƒ We know the outputs
ƒ The “box” is a mystery
ƒ The goal is to characterize the relationship of
the outputs to the inputs
ƒ Often a candidate for a combinatorial
approach
 18

Test Deliverables and

Metrics
ƒ Integration into development process
ƒ Design for testability
ƒ Clear pass fail criteria (function or physical)
 19

Production test equipment

ƒPurpose is to remove failing material,
NOT to pass good material
ƒHigh degree of automation
ƒMay use specialized equipment
ƒ In-circuit testers
ƒ Customized to a specific product
 20

Outsourcing
ƒ Objectives with outsourcing test
ƒ (what to outsource)

ƒ Risks & Risk Mitigation

 21

Test reporting

ƒSystem qualities
ƒ Accessibility by team
ƒ Traceability
ƒ Categorization
ƒ Assigned responsibilities
ƒ Statistical analysis
ƒ Speed! !
 22

Review test results

ƒ Hardware issues
ƒ Software issues
ƒ Corrective actions
 23

Supplier Integration into

development
ƒSupplier (contractor) requirements
ƒSupplier (contractor) participation
ƒSupplier (contractor) audit
 24

Planning in- or out-sourcing

ƒCurrently using a scrum approach to
testing
ƒ By product backlog
ƒ Daily scrum meeting
ƒ Burn down charts
 25

Why does this work?

ƒ Microtasking breaks larger items down to ‘atomic’
level
ƒUnwise to decompose further
ƒCompletion is Boolean
ƒ Yes or no
ƒ Rollup can provide a % for upstream
managers
 26

Microtasking
ƒ Like eating an elephant—one bite at a time!

ƒ Immediate reinforcement to testers

ƒFeeling of accomplishment
ƒMeasurable completion
ƒ Employees not overwhelmed at any given task
 27

Reviews
ƒ Sprint review equivalent to a monthly or fortnightly in-
process review (IPR)

ƒ Basically, good systems engineering

ƒ Out-source can use technology

ƒ Example: Microsoft Live Meeting
ƒ Example: eroom
 28

Month versus fortnight

ƒFortnightly sprints increase review
burden but also improve feedback and
control
ƒA calendar month may be too long a
period, especially during periods of
great change
ƒJudgment call
 29

Objectives: with outsourcing

test (what to outsource)
ƒObjectives with outsourcing test
ƒ what to outsource
ƒ Outsource
ƒ What we cannot test with our own
resources
ƒ What an outside lab can do better
ƒ Unusual tests

ƒRisks & Risk Mitigation

 30

Outsourcing: Risks & Risk

Mitigation
ƒRisk
ƒ Don’t know product
ƒ Don’t know how to use fixtures and support
tools
ƒ May not understand special instructions
 31

Outsourcing: Risks & Risk

Mitigation
ƒMitigation
ƒ Up front review of outsource facility
ƒ Review of outsourced:
ƒ test plans
ƒ Test fixtures
ƒ Test cases (test execution specifics)
ƒ Test report demands identified

ƒAudit – conduct the a few of the tests in house

the answers should be the same as the
outsourced test activities
 32

Outsourcing: Risks & Risk

Mitigation
ƒ Send an engineer to test site (extra cost)
ƒ Provide fixture (or test cases)
ƒ Frequent test status review
ƒ Clear test metric identification and tracking
ƒ High Reliability and Confidence interval
ƒ Large sample size
 33

Outsourcing: Risk Mitigation

ƒ Clear and detailed Responsibility and
Authority Matrix
ƒ Communications plan
ƒ Use of test standards
ƒ Clear pass fail criterion
ƒ Physical
ƒ Functional
 34

Outsourcing: Pros / Cons

ƒ Pros
ƒIncrease capacity instantly
ƒOften, special expertise
ƒ Cons
ƒCost
ƒProduct-related issues
ƒCommunications and Logistics
ƒExpertise retention
 35

Outsourcing: Reliability
ƒUse accredited labs
ƒSolicit samples of reports
ƒVisit site
 36

Test configuration
management
ƒProduct baseline
ƒProduct change management
ƒProduct configuration (CMP)
ƒTest configurations
ƒConfiguration audit
 37

Test Planning
ƒ Project integrated Test plan
ƒ Project management responsibility
ƒ Test Evaluation Master Plan
ƒ Product release schedule (revision and
functional content)
 38

Human resources

ƒ Manager

ƒ % Engineers
ƒMore complex problems
ƒTest cases & procedures
ƒCan write code for test machines
ƒ % Technicians
ƒBasic test execution
ƒSimpler problems
 39

Additional information
ƒ Kim Pries: khpries@gmail.com

ƒ Jon Quigley: jonmquigley@aol.com