Training Effectiveness,

design and delivery-

an ISO-9000 perspective.
Agenda
• Overview:
• ISO Model
• Requirements (9K, 14K, 18K).
• Kirpatrick Model for Training
Evaluation.
• ISO-10015: Guidelines for
Training.
Background
Quality Management System Model
CONTINUAL IMPROVEMENT OF THE
UNIFIED MANAGEMENT SYSTEM

C
S
a C
t
uR Management i u
responsibility s

s qe f
a
s
t i
u
Resource
Measurement, c
t t
r analysis, i
o
o me management
improvement
o
n
m ne m
e ts Product Product e
Input
realization Output

r r
6. Resource Management

6.2 Human Resources

6.2.1 General

Personnel whose work affect product

quality to be COMPETENT on the
basis of appropriate education,
training, skills and experience

5
6. Resource Management
(Contd.)
6.2 Human Resources
6.2.2 Competence, Awareness and Training
The organization to
• Determine necessary competence for personnel performing
work affecting product quality,
• Provide training or take any other actions to satisfy these
needs***,
• Evaluate effectiveness of the actions taken.***

Ensure that its personnel are aware of

• relevance and importance of their activities
• how they contribute to the achievement of the Quality
Objectives
Maintain records of education, experience, training and skills
*** Changed in 2008 version

6
6. Resource Management
(Contd.)
6.2 Human Resources (Contd)
6.2.2 Competence, Awareness and Training
(Guidelines)
Make employees at each relevant function level aware of
• importance of conformance with the Quality Policy & QMS
requirements
• significant impacts of their work activities on quality, actual or
potential
• benefits of improved personnel performance
• roles and responsibilities in achieving conformance to quality
policy
• the potential consequences of departure from specified
procedures

7
 Changes in ISO-9001-2008
Paragraph 6.2
Words, "affecting product quality" changed to "affecting
conformity to product requirements”

Paragraph 6.2.2
Sub-clause (b) -"provide training or take other actions
to satisfy these needs" changed to "where applicable
training needs to be provided to achieve the
necessary competence".

Sub-clause (c) – obligation to ensure that the staff

attained the expected competence.
ISO-14000 & OHSAS-18001
4.4.2 Training Awareness &
competence

• Identify training needs

• Establish training procedures
• Importance of conformance
• Consequences of departure from
procedures
• Personnel competent by education,
training, and/or experience

9
4.4.2 Training Awareness &
competence

• Training department completes

Training Needs Analysis (TNA) and
training schedule, reviews at least
annually and revises as necessary
• Permanent on-site contractors
included in training program
• Training records maintained

10
 Effectiveness
Extent to which planned activities are realized and planned
results achieved.
ISO 9000:2000 3.2.14

ISO 9001 specifies requirements for a quality management

system that can be used for internal application by
organizations, or for certification, or for contractual purposes. It
focuses on the effectiveness of the quality management system
in meeting customer requirements.
ISO 9001: 2000 0.3
Training Design: ISO-9001

Part of Product Realization Process,

Clause 7.3
 Training Design Planning
 Design Inputs
 Design Output
 Design Review
 Design Verification
 Design Validation
 Control of changes

12
7. Product Realization (Contd.)
7.3 Design & Development (D/D)
7.3.1 Design and Development Planning
• Plan and control Design and Development of the product
• During planning, determine
• Stages of design and development
• Review, verification and validation appropriate to each
Design and Development stage
• Responsibilities and authorities for Design and Development
• Manage interface between different involved groups to ensure
effective communication and clear responsibilities
• Update planning output, as appropriate, as Design and
Development progresses

13
7. Product Realization (Contd.)
7.3 Design & Development (D/D) (Contd)
7.3.2 Design & Development Inputs
❚ Determine the inputs related to product
requirements and maintain records
❚ Design & Development inputs to include
❙ Functional and performance requirements
❙ Applicable statutory and regulatory requirements
❙ Where applicable, Information derived from previous
similar designs
❙ Other requirements essential for Design and Development
❚ Review inputs for adequacy. Requirements to be
complete, unambiguous and non-conflicting

14
7. Product Realization (Contd.)
7.3 Design & Development (D/D) (Contd)
7.3.3 Design & Development Outputs
❚ Outputs of D/D provided in a form that enables
verification against D/D inputs. This shall be
approved prior to release
❚ D/D Outputs shall :
❙ meet D/D input requirements
❙ provide appropriate information for purchasing,
production and for service provisions
❙ contain or reference product acceptance criteria
❙ specify product characteristics essential for safe and
proper use
15
7. Product Realization (Contd.)
7.3 Design & Development (D/D) (Contd)
7.3.4 Design & Development Review
❚ At suitable stages, conduct systematic D/D reviews
❙ to evaluate the ability of results of D/D to fulfill requirements
❙ to identify any problems and proposed necessary actions
❚ Involve and include representatives of functions concerned with
design stage(s) being reviewed
❚ Records of results of reviews and necessary actions to be
maintained
7.3.5 Design & Development Verification
❚ Perform verification to ensure that D/D outputs have satisfied D/D
input requirements
❚ Records of results of verification & necessary actions to be
maintained

16
7. Product Realization (Contd.)
7.3 Design & Development (D/D) (Contd)
7.3.6 Design & Development Validation
❚ Perform D/D validation in accordance with planned
arrangements (see 7.3.1) to ensure
❙ Resulting product is capable of fulfilling the requirements
for the specified or known intended use or application
❚ Wherever practicable, validation to be completed prior
to delivery or implementation of the product
❚ Records of results of validation and necessary actions to
be maintained

17
7. Product Realization (Contd.)
7.3 Design & Development (D/D) (Contd)
7.3.7 Control of Changes
❚ Changes in D/D to be identified and records
maintained
❚ The changes to
❙ be reviewed, verified and validated, as appropriate
❙ be approved before implementation
❙ include evaluation of the effect of changes on
constituent parts and delivered products
❚ Records of the results of validation and
necessary actions to be maintained

18
Training Evaluation

6 - 19
How the training is evaluated?

Factors affecting evaluation?

 Evaluation
- Myths

I Can’t Measure the Result of My Training Efforts.

I Don’t Know What Information to Collect.
If I Cannot Calculate the ROI, the Evaluation Is Useless.
My CEO Does not Require Evaluation, So, Why Should I Do It ?
There Are Too Many Variables Affecting the Behavior Change for
Me to Evaluate the Impact of Training.
Evaluation Will Lead to Criticism.
I Don’t Need to Justify My Existence; I Have a Proven Track Record.
 Factor How Factor Influences Type of
Evaluation Design
Change Can program be modified?
potential
Importance Does ineffective training
affect customer service,
product development, or
relationships between
employees?
Scale How many trainees are
involved?
Purpose of Is training conducted for
training learning, results, or both?
Questions we must
answer …
How do participants feel about our
training program?
Are participants learning?
Is their learning transferring to the
job?
Does the organization benefit from
our training efforts?
Donald Kirpatrick’s
Model
for
Training Evaluation
Finding answers …
Kirkpatrick’s 4 Levels of Evaluation
Level 1: Reaction
How do participants feel about our training
program?
Level 2: Learning
Are participants learning?
Level 3: Behavior
Is their learning transferring to the job?
Level 4: Results
Does the organization benefit?
Source: Kirkpatrick,1998
  DONALD KIRKPATRICK'S 4 LEVELS OF EVALUATING TRAINING
Levels Description Comments
Level 1 Reaction Trainee reaction to the Most primitive and widely-
course. Does the trainee used method of evaluation.
like the course? Usually It is easy, quick, and
in the form of evaluation inexpensive to administer.
forms, sometimes called Negative indicators could
"smile sheets". mean difficultly learning in
the course.
Level 2 Learning Did trainees learn what Learning can be measured
was based on the course by pre- and post tests,
objectives? either through written test
or through performance
tests.
Level 3 Behavior Trainee behavior Difficult to do. Follow-up
changes on the job - are questionnaire or
the learners applying observations after training
what they learned? class has occurred.
Telephone interviews can
also be conducted.
Level 4 Results Ties training to the Generally applies to training
company's bottom line. that seeks to overcome a
business problem caused by
lack of knowledge or skill.
Examples include reductions
in costs, turnover,
absenteeism and
grievances. May be difficult
Appeal of Kirkpatrick’s
Model:
Assesses important areas

Widely known

Simple framework

Easy to explain and understand

However . . .
Widely Know ≠ Widely Used
Level 1: Often (over 90%)
Level 2: Sometimes (less than 35%)
Level 3 & 4: Rarely (less than 15%)
Why is this a problem?
Level 3 and 4 often perceived as:
Difficult to measure
Time consuming
Beyond the realm of most trainers
Level 1 result does not always mean similar
Learning / Transfer / ROI results
Source: Pershing & Gilmore, 2004
Other problems …
Undermines Management Partnership
Training ≠ “Silver Bullet”
Training is only one strategy within entire
Performance System
Level 3 & 4 should include evaluations of entire
Performance System - not just training
Lacks Performance System Focus
What about rest of Performance Environment?
What factors impede / enable usage of training?
Feedback Goes to Wrong People
Feedback to training function only is incomplete
Must include Performance Environment owners

Source: Brinkerhoff & Dressler, 2002

In Summary:
An evaluation tool must be integrated
into the training programs.
Kirkpatrick’s Four Levels of Evaluation is
well known, but has limitations:
Unlikely completion of all 4 Levels
Lacks performance system focus
 ISO-10015
Guidelines for Training
Background of ISO-10015
22 country representatives developed the draft
text over several years, culminating in the
publication of ISO 10015 issued December 1999.

The advantages:

• based on the process oriented concepts of the

new 9000:2000 ISO family of standards.
• Offers specific guidance for training design,
review and implementation.
• Provides context to organization's needs.

32
 Key features of ISO 10015
1. Linking training investment with company
performance  
• Key client is the organization, not only the persons being trained.
(ROI Approach)
• A company has to recognize first what is the performance challenge
it faces and the causes for it.
• ISO 10015 offers a clear road map to connect training to
performance goals and use it for individual and group performance
improvement.
2. Organizing training on the basis of pedagogical
principles and processes
• Training as a tool to cover an identified performance gap.
• Establishing an appropriate training design and effective learning
processes.
• Ensures that training uses resources (finances, time and energy)
optimally.
Training: for organizational needs
4.1Training is 4-step
process
For selecting and implementing training to
close the gaps between required and
existing competence, management should
monitor the following stages:
a) defining training needs;
b) designing and planning training;
c) providing for the training;
d) evaluating the outcome of training.
The Training Process
4.2 Defining Training
Needs
4.2.1 General
4.2.2 Defining the needs of the
organisation
4.2.3 Defining the competence needs
4.2.4 Reviewing Competence
4.2.5 Defining Competence gaps
4.2.6 Identifying solutions close to
competence gaps
4.2.7 Defining the specifications for
training needs
Training Processes-1
Training Processes-2
Training Processes-3
Training Processes-4
4.3 Designing and planning training

4.3.1 General
4.3.2 Defining the constraints
4.3.3 Training methods and
criteria for their selection
4.3.4 Training Plan specifications
4.3.5 Selecting a training provider
4.4 Providing the training

4.4.1 General
4.4.2 Providing Support
1. pre-training support
2. training support
3. end-of-training support
4.5Evaluating training
outcomes
4.5.1 General
4.5.2 Preparing evaluation report
Contents of evaluation report:
– specification of training needs
– Evaluation criteria and description of
sources, methods and schedule for
evaluation.
– Analysis of data and interpretation
– Review of training costs
– Conclusions and recommendations for
improvement.
5 Monitoring and improving the training
process

5.1 General
5.2 validation of training process
Monitoring is the key tool:
– Monitoring is to be done by competent personnel
only.
– Monitor records at all stages. Detect nonconformity
at all stages. Take preventive and corrective
actions.
– if procedures followed and requirements met, then
update the personal competence records.
– if procedures not followed yet requirements are
met, revise procedures and update training records.
– If procedures followed and requirement not met,
either correct the training process or seek a non-
training solution.
• Identify further opportunities for improvement.
Thank
You