Skill Category 2

Building the Test

Environment

1
Skill Category 2
• Building the Test Environment .............................................................. 133
• Management Support ............................................................................. 133
• Management Tone ..................................................................... 134
• Integrity and Ethical Values ..................................................................... 135
• Commitment to Competence ..................................................................... 137
• Management’s Philosophy and Operating Style .............................................................
137
• Organizational Structure ...................................................................... 138
• Test Work Processes .............................................................................. 140
• The Importance of Work Processes ...................................................................... 140
• Responsibility for Building Work Processes ....................................................................
142
• Developing Work Processes ...................................................................... 149
• Tester’s Workbench ...................................................................... 150
• Analysis and Improvement of the Test Process ..............................................................
153
• Test Tools .............................................................................. 167
• Tool Development and Acquisition ...................................................................... 167
• Tool Usage ...................................................................... 179
• Testers Competency .............................................................................. 180

2
 The test Environment is comprised of the
following conditions within the organization:
• policies
• procedures
• culture
• attitudes
• rewards
• test processes
• test tools
• methods for developing and improving test processes
• management’s support of software testing
• test labs developed for the purpose of testing software
• multiple operating environments
• assuring the test environment fairly represents the
production environment to enable realistic testing to occur 3
 Management Support = Good for testers.

Following will be discussed:

• Management “tone”: Management sets the tone by

providing testers the resources and management time
needed to do their job effectively.

• Test process alignment: Management creates a test

function that is aligned with the business needs of the
organization.

4
 Testers need to know 3 things
about the control environment:
1. The control environment is established by the
highest levels of management and works
downward through the organization.
2. The test function cannot create the
organization’s control environment, but can
influence how that environment is implemented
within the test function.
3. The control environment will influence the way
in which testers perform the work which may
be ethical or unethical.
5
 Integrity and Ethical Values

Excerpt:
“Integrity is a prerequisite for ethical
behavior in all aspects of an enterprise’s
activities. A strong corporate ethical
climate at all levels is vital to the well-
being of the corporation, all of its
constituencies, and the public at large.”

6
Incentives and Temptations

Individuals may engage in dishonest,

illegal or unethical acts simply because
their organizations give them strong
incentives or temptations to do so.

7
 Incentives cited for engaging in fraudulent unethical
behavior:
• Pressure to meet unrealistic performance targets, particularly for
short-term results
• High performance-dependent rewards.

Temptations for employees to engage in improper

acts:
• Nonexistent or ineffective controls, such as poor segregation of
duties in sensitive areas that offer temptations to steal or to conceal
poor performance.
• High decentralization that leaves top management unaware of
actions taken at lower organizational levels and thereby reduces the
chances of getting caught.
• A weak internal audit function that does not have the ability to detect
and report improper behavior.
• Penalties for improper behavior that are insignificant or unpublicized
and thus lose their value as deterrents.

8
Providing and Communicating
Moral Guidance

The most effective way of transmitting a

message of ethical behavior throughout
the organization is by example.

9
 Commitment to Competence

• Competence should reflect the knowledge

and skills needed to accomplish tasks that
define the individual’s job.

• Management needs to specify the

competence levels for particular jobs and
to translate those levels into requisite
knowledge and skills.
10
 Management’s Philosophy and
Operating Style
• Management’s philosophy and operating style affect the way
testing is managed, including the kinds of business risks
accepted.
– Informal: An informally managed company may control
operations largely by face-to-face contact with key managers.
– Formal: A more formally managed exception reports.
• Other elements of management’s philosophy and operating
style include:
– attitudes toward financial reporting
– conservative or aggressive selection from available alternative
accounting principles
– conscientiousness and conservatism with which accounting
estimates are developed
– attitudes toward data processing and accounting functions and
personnel.

11
 Organizational Structure
• An entity’s organizational structure provides the framework within
which its activities for achieving entity-wide objectives are planned,
executed, controlled and monitored.

• Activities that relate to the value chain:

– inbound (requirements) activities
– operations or production
– outbound (software)
– deployment and maintenance

• Support Functions:
– administration,
– human resources
– Technology development

12
 Assignment of Authority and
Responsibility

• Management has an important function with the

assignment of authority and responsibility for operating
activities, and establishment of reporting relationships
and authorization protocols.

• Management function deals with:

– policies describing appropriate business practices
– knowledge and experience of key personnel
– resources provided for carrying out duties

13
 Human Resource Policies and
Practices
Messages to employees:
• Standards for hiring the most qualified individuals, with emphasis on educational
background, prior work experience, past accomplishments and evidence of
integrity and ethical behavior, demonstrate an organization’s commitment to
competent and trustworthy people.
• Recruiting practices that include formal, in-depth employment interviews and
informative and insightful presentations on the organization’s history, culture and
operating style send a message that the organization is committed to its people.
• Training policies that communicate prospective roles and responsibilities and
include practices such as training schools and seminars, simulated case studies
and role-play exercises, illustrate expected levels of performance and behavior.
• Rotation of personnel and promotions driven by periodic performance appraisals
demonstrate the entity’s commitment to the advancement of qualified personnel
to higher levels of responsibility.
• Competitive compensation programs that include bonus incentives serve to
motivate and reinforce outstanding performance. Disciplinary actions send a
message that violations of expected behavior will not be tolerated.

14
 Test Work Processes
• Work processes are the policies, standards and
procedures in a quality IT environment. Test work
processes are those work processes specific to the
testing activity.

– Policy: Managerial desires and intents concerning either

process (intended objectives) or products (desired attributes).

– Standards: The measure used to evaluate products and identify

nonconformance. The basis upon which adherence to policies is
measured.

– Procedure: The step-by-step method that ensures that

standards are met.

15
 The Importance of Work
Processes
• It is important to a quality IT environment
to establish, adhere to, and maintain work
processes in the testing activity. It is also
critical that the work processes represent
sound policies, standards and procedures.

16
 Major purposes and advantages
to having work processes

• Improves communication
• Enables knowledge transfer
• Improves productivity
• Assists with mastering new technology
• Reduces defects and cost

17
 Responsibility for Building Work
Processes
• IT management is responsible for issuing IT policy. Policies
define direction and by definition are general rules or
principles. It is the standards, which will add the specificity
needed for implementation of policies.
• Policies are needed in:
– Building Systems
– Testing Systems
– Maintaining Systems
– Operating Systems
– Quality of Systems
– Security of Systems
– Allocation of Resources
– Planning for Systems
– Training Personnel

18
Responsibility for Standards and
Procedures
The workers who use the procedures and
are required to comply with the standards
should be responsible for the development
of those standards and procedures.
Management sets the direction and the
workers define that direction. This division
permits each to do what they are
best qualified to do.

19
 The key concepts of a process
engineering program are:
• Management provides an organizational
structure for the workers to develop their own
standards and procedures.
• The program is driven by management policies.
• Absolute compliance to standards and
procedures is required.
• A mechanism is provided for the continual
maintenance of standards and procedures to
make them more effective.

20
 Test Process Selection
Selecting, developing, and acquiring work
processes is an overall IT organization
responsibility. Software testers need to both
understand how the activity operates AND
participate when test processes, and
related processes, are selected and put
into practice.

21
Test Process Selection (cont’d)
IT groups should develop a plan for implementing and
operating a process engineering
program. This would require a policy for standards, and
a charter or job description for the
function. These need to be customized for each
organization. The specific components that
need to be addressed include:
– Building a Process Engineering Organization
– Developing a Standard and Procedure for Standards
– Planning for Standards
– Writing, Storing, and Retrieving Standards and Procedures
– Enforcing Standards

22
Building a Process Engineering
Organization

The structure that is put in place to

develop and update policies, standards,
and procedures must involve both staff
and management. The process
engineering program should be directed
by management, but implemented by
staff.

23
Building a Process Engineering
Organization (cont’d)
Some guidelines on establishing an organizational
structure for process engineering are:

– Establish a Process Engineering Committee compromised of the

most senior IT managers possible.
– Represent all IT organizational areas on the Process
Engineering Committee.
– Appoint an individual as the Process Engineering Manager.
(Note that this can be a part-time assignment.)
– Appoint Ad Hoc Committees (i.e., special task forces) to develop
individual standards and procedures.
– Let the Standards Ad Hoc Committees develop the technical
standard.
24
 Recommended Standards
Organizational Structure
The recommended organizational structure is
comprised of the following three
components:

– Process Engineering Manager

• A full-time or part-time individual responsible for managing
the standards program.
– Process Engineering Committee
• A policy-setting board, which runs the standards program.
– Ad Hoc Committee
• Small groups which develop single standards at a time.

25
Organizational chart for process engineering

26
 Role of Process Engineering
Manager
• Promote the concept of process
engineering.
• Be the driving force behind processes.
• Administer the standards program defined
by the Process Engineering Committee.
• Be a resource to the Process Engineering
Committee and Ad Hoc Committees.
• Ensure involved parties are adequately
trained.
27
Role of the Process Engineering
Committee

• Accept topics for processes.

• Set priority for implementation of
processes.
• Obtain the resources necessary to
develop the process.
• Approve or reject developed processes.

28
Role of the Ad Hoc Committee
• Gain representatives from all involved areas.
• Ensure that the committee has between three
and eight members in size.
• Create the standard and procedure.
• Coordinate reviews of the standard with involved
parties.
• Periodically review and update the standards
and procedures previously developed by
• the Ad Hoc Committee.
29
 Selecting Process Engineering
Committee Members
The makeup of the Process Engineering Committee
is important. Some guidelines in selecting
members are:
• Select the highest-level manager who will accept the
position.
• Assign individuals who are supportive and enthusiastic
over standards.
• Make long-term assignments to the Process Engineering
Committee.
• Select individuals who are respected by their peers and
subordinates.
• Ensure the appropriate areas of interest are involved.
30
 Selecting Process Engineering
Committee Members (cont’d)
Some guidelines to ensure the appropriate areas of
interest are included in developing
standards are:

• Is every organizational function within IT involved?

• Are the activities that interface to the IT function
involved; for example, key users and customers?
• Are activities having a vested interest involved, such as
auditors?
• Are all the IT “businesses” represented such as the
project leaders, systems programmers, data library,
security, help desk, and so forth?
31
Developing Work Processes

Prior to creating any processes, the Process

Engineering Committee must develop a
standard and procedure for developing
standards and procedures. The standards and
procedures developed by the standards program
are products. Those products must be
standardized just as
the other products produced in an IT function.

32
 Defining the Attributes of a
Standard for a Standard
The standard for a standard and a standard for a
procedure, define the format, style, and
attributes of a document called a standard and a
document called a procedure. This standard
for a standard and standard for a procedure become the
prototype that will be used by the Ad
Hoc Committees in developing standards and
accompanying procedure(s).
This standard should define the following:
– A testing policy
– A test workbench to support the policy (i.e., standards and
processes)

33
 Establishing a Testing Policy
• A testing policy is management’s objective for testing. It
is the objective to be accomplished.
• Good testing does not just happen, it must be planned,
and a testing policy should be the cornerstone of that
plan.

Sample Testing Policy

34
 Tester’s Workbench
• The tester’s workbench shows that input products drive the
workbench, which uses procedures and standards to produce
output products.

35
 Professional Test Standards
Test professionals should be familiar with the
standards published by organizations
such as the International Standards Organization
(ISO), U.S. Department of Commerce
(National Institute of Standards and Technology),
and IEEE. In addition, testers should know
when these standards apply, and how to obtain the
latest versions.

36
 Professional Test Standards
(cont’d)
• IEEE publishes a body of standards that address software engineering and
maintenance. A portion of these standards addresses software testing and
test reporting. Listed below are the current standards that apply to the test
process.

– 829-1998 IEEE Standard for Software Test Documentation

– 830-1998 IEEE Recommended Practice for Software Requirements
– Specifications
– 1008-1987 (R1993) IEEE Standard for Software Unit Testing (ANSI)
– 1012-1998 IEEE Standard for Software Verification and Validation
– 1012a-1998 IEEE Standard for Software Verification and Validation –
Supplement to 1012-1998 – Content Map to IEEE 12207.1
– 1028-1997 IEEE Standard for Software Reviews

• Other professional organizations have similar standards. The CSTE

candidate should have an overall familiarity with the purpose and contents
of these standards.

37
Analysis and Improvement of the
Test Process
• Test Process Analysis: Test quality control is performed during the
execution of the process. Analysis is performed after the test
process is completed.

– Effectiveness and efficiency of test processes

– The test objectives are applicable, reasonable, adequate, feasible, and
affordable
– Testers do not have the needed competencies to meet the test
objectives
– The test program meets the test objectives
– The correct test program is being applied to the project
– The test methodology is used correctly
– The task work products are adequate to meet the test objectives
– Analysis of the results of testing to determine the adequacy of testing
– Adequate, not excessive, testing is performed

38
 Continuous Improvement
• A testing process assessment is one excellent way to determine the
status of your current test process.
• Process assessment is applicable in the following circumstances:
– Understanding the state of processes for improvement.
– Determining the suitability of processes for a particular requirement or class of
requirements.
– Determining the suitability of another organization’s processes for a particular
contract or class of contracts.
• The framework for process assessment:
– Encourages self-assessment.
– Takes into account the context in which the assessed process operates.
– Produces a set of process ratings (a process profile) rather than a pass or fail
result.
– Addresses the adequacy of the management of the assessed processes through
generic practices.
– Is appropriate across all application categories and sizes of organization.

39
 Test Process Improvement
Model
• A model for test process improvement has
these eight steps:
1. Examine the Organization’s Needs and Business
Goals
2. Conduct Assessment
3. Initiate Process Improvement
4. Analyze Assessment Output and Derive Action Plan
5. Implement Improvements
6. Confirm Improvements
7. Sustain Improvement Gains
8. Monitor Performance
40
 Test Process Alignment
• In establishing the test environment management must assure that
the mission/goals of the test function are aligned to the
mission/goals of the organization.

41
 Adapting the Test Process to Different
Software Development Methodologies
• Understanding the software development
process used in his/her organization is
important in conducting software testing
for the following three reasons:
– Understanding the developmental timetable
and deliverables.
– Integrating software
– Understanding how software is developed

42
 Six major categories of
developmental methodologies

• Waterfall
• D-shaped
• Prototype
• Spiral
• RAD
• Incremental

43
Testers testing software developed by a specific
software development methodology need to:

1. Understand the methodology.

2. Understand the deliverables produced when
using that methodology which is needed for
test purposes.
3. Identify compatible and incompatible test
activities associated with the developmental
methodology.
4. Customize the software test methodology to
effectively test the software based on the
specific developmental methodology used to
build the software.

44
 Test Tools
• It is difficult to perform testing
economically without the aid of automated
tools.
• The most important aspect of software
testing tools is the process used to acquire
those tools. Most of this component of the
environment will focus on acquiring or
developing testing tools.

45
 Tool Development and
Acquisition
• Difficulties surrounding the introduction of tools can arise in three
areas:
– Organizational obstacles
– Problems arising from the tools
– Obstacles in the computer environment

• The activities associated with the introduction of tools should include

these activities:
– Identifying the goals to be met by the tool (or by the technique
supported by the tool), and assigning responsibility for the activities
required to meet these goals.
– Approving a detailed tool acquisition plan that defines the resource
requirements for procurement and in-house activities.
– Approving the procurement of tools and training, if this is not explicit in
the approval of the acquisition plan.
– Determining, after some period of tool use, whether the goals have been
met.

46
 Tool Development and
Acquisition (cont’d)
• Test management is assigned responsibility for:
– Identifying tool objectives.
– Approving the acquisition plan (it may also require approval by funding
management).
– Defining selection criteria.
– Making the final selection of the tool or the source.

• The Test Manager is responsible for the following activities. Note

that a selection group for the more important test tools may assist
the Tool Manager.
– Identifying candidate tools.
– Applying the selection criteria (in informal procurement) or preparing the
RFP (in formal procurement).
– Preparing a ranked list of tools or sources.
– Conducting any detailed evaluations or conference room pilots.

47
 Tool Development and
Acquisition (cont’d)
• This distribution of responsibilities reduces
the chances of selecting a tool that:
– Does not meet the recognized needs of the
organization
– Is difficult to use
– Requires excessive computer resources
– Lacks adequate documentation

48
 Event Sequencing

• The following event sequence addresses

only the introduction of existing tools.
• The recommended event sequence allows
for two procurement methods for bids:
– 1. Informal procurement (by purchase order)
– 2. Formal procurement (by a request)

49
 Event Sequence Steps
• Goals
• Tool Objectives
• Acquisition Activities for Informal Procurement
• Acquisition Plan
• Selection Criteria
• Identify Candidate Tools
• User Review of Candidates
• Score Candidates
• Select Tool
B. Acquisition Activities for Formal Procurement
• Acquisition Plan
• Technical Requirements Document
• User Review of Requirements
• RFP Generation
• Solicitation of Proposals
• B6 – Technical Evaluation should be Consistent
• Source Selection
• Procure Tool
• Evaluation Plan
• Implementation Plan
• Training Plan
• Tool Received
• Acceptance Test
• Orientation
• Modifications
• Training
• Use in the Operating Environment
• Evaluation Report 50
• Determine if Goals Are Met
 Tool Usage
The most commonly used tools can be
grouped into these eight areas:

1. Automated Regression Testing Tools

2. Defect Management Tools
3. Performance/Load Testing Tools
4. Manual Tools
5. Traceability Tools
6. Code Coverage
7. Test Case Management Tools
8. Common tools that are applicable to testing

51
 Tool Usage (cont’d)
• Most testing organizations agree that if the
following three guidelines are adhered to tool
usage will be more effective and efficient.
– Guideline 1 – Testers should not be permitted to use
tools for which they have not received formal training.
– Guideline 2 – The use of test tools should be
incorporated into test processes so that the use of
tools is mandatory, not optional.
– Guideline 3 – Testers should have access to an
individual in their organization, or the organization that
developed the tool, to answer questions or provide
guidance on using the tool.
52
 Testers Competency
• Test competency is a direct responsibility
of the individual and the organization that
employs that individual.
• Software testing organizations should
develop a roadmap for testers to pursue to
advance their competencies. The
roadmap will have these two paths:
– Skill Sets
– Performance Skills

53
 This figure is typical of how a software testing
organization may measure an individual tester’s
competency.

54