Evaluating HRD Programs

Show me the value!

These are challenging

economic times.

HRD professionals struggle to

find a seat at the table.

Organizational leaders do not

understand the value of HRD.

You can show how HRD programs

support the bottom line.

Use the four-level evaluation

model to demonstrate HRD impact.

Evaluation
Front-end Process
Concurrent with Design or Purchase
Identify Target/Goals
Collect Pre-training or Baseline Data

Four Levels of Evaluation

Reaction
Learning
Application (Behavior)
Impact (Results)

Reaction
Customer Service Index
Usually Questionnaire
Feedback on Program
Content
Activities
Instructor
Did they like it?

Learning

Quality Assurance Index

Paper & Pencil Tests
Skill Demonstrations
Simulations
Projects
Did they achieve the objectives?

Behavior/Skill Application
Effectiveness Index
Transfer of Learning to the Workplace

Are they applying their new skills on their jobs?

Non-Observable Results
Effectiveness Index
Occurring in the Mind
Cognitive (problem-solving skills)
Affective (attitudes, commitment)

Are participants applying non-observable

outcomes on the job?

Impact
Results Index
Bottom-Line Impact
Return on Investment
In the application of skills, has there been
any impact on the business?

Formative & Summative

Formative
During program development
How effective are the strategies?
To improve

Summative
After program development
Is the program effective?
To determine merit, worth, value

Characteristics of4 Levels

Criteria
Reaction

Value of
Frequency of Difficulty of
Information
Use
Assessment
Lowest

Frequently

Easy

Highest

Infrequently

Difficult

Learning
Application
Impact

Needs Assessment & Evaluation

Organization

Organizational Results

Work/Task/Job

Job Behavior

Individual/Person
Preferences

Learning
Reaction

Learning & Workplace Interventions

Adapted from: Phillips, J., & Holton, E. (Eds.). (1995). In action: Conducting needs assessment.
Alexandria, VA: American Society for Training & Development.

Training for Impact

Needs Assessment & Evaluation
Learning
Activity

On-the-Job
Performance

On-the-Job
Performance

Operational
Results

Level I Evaluation
Level II Evaluation (Post)
Level II Evaluation
(Pre)

Needs Assessment
(Should & Is)
Level III Evaluation
(Pre)

Time

Needs Assessment
(Should & Is)
Level IV Evaluation
(Pre)

Operational
Results

Level III Evaluation

(Post)
Level IV Evaluation
(Post)

Level I - Reaction
Level II - Learning
Level III - Behavior
Level IV - Results

copyright Sharon Korth, 1994

Reaction Evaluation
General Principles
Determine PurposesDecisions to be Made
Determine How Data Will be Tabulated
Elicit a Personal Response
Make Decisions Based on Item Responses

Design of
Reaction Evaluations

Introductory Statement
Closing Statement
Directions for Filling Out Form
Easy-to-Follow Layout
Professional Appearance
Statement about Confidentiality or
Anonymity

Dimensions of Reaction
Evaluations
Program
Objectives/Content
Program Materials
Delivery
Methods/Technologies
Instructor/Facilitator
Instructional Activities
Program Length/Time

Training Environment
Planned Action/Transfer
Expectation
Logistics/Administration
Overall Evaluation
Recommendations for
Program Improvement

Lee, S. H., & Pershing, J. S. (1999). Effective reaction evaluation in evaluating

training programs. Performance Improvement, 38(8), 32-39.

Question Construction
Common Errors
Double-Barreled Questions
The instructor was knowledgeable and

enthusiastic.

Leading or Subjectively-Worded
Questions
Why was this the best training you ever

attended?

Question Response Formats

Closed-Ended Questions
Multiple choice
Yes-No or True-False
Rating Scales (Likert)

Use a Balanced Scale

Use a Conventional Scale
Label Each Choice

Question Response Formats

Have Responses Match the Questions
Presentation of information

High -------------------------------Low

Quality of Course Materials

Important---Neutral---Unimportant

Effectiveness in responding to questions

Very Poor -- Good Very Good

Reaction Evaluations
What about.

Open-ended questions / comments?

When to have them completed?
Asking people for their names?
Efficiency - One-Size Fits All?
Asking about confidence & values?
Asking about expected ROI?

Problems?
Dont Smile About Smile Sheets
Jones, J. J. (1990). Dont smile about
smile sheets. Training & Development,
44(12), 19-21.

Humor
Ten Rules for Perfect Evaluations
http://www.karinrex.com/tc_evals.html

Learning Evaluation
Purpose:
Did they learn what was intended?

Knowledge Acquisition
Skill Development
Micro - Macro
Drivers Ed Written & Driving Test

Why Measure Learning?

Learning Experience
X Work Environment

Business Results

Designing a Learning
Evaluation
9 Determine who/what decisions will be made
9 Link learning objectives to learning activities
9 Pilot learning tests
9 Revise tests as necessary
9 Validate tests and establish norms/standards

Validity

Does it measure what it was designed to measure?

It is linked to the learning objectives?

Content Validity

Represents the content of the program

Construct Validity

Represents the construct it purports to measure

Concurrent Validity

Agrees with the results of other similar instruments

Predictive Validity

Ability to predict future behaviors

Reliability
Does it consistently measure the ability to perform?
Does it consistently identify those who can and cannot?

Test-Retest
Parallel Form
Inter-Item
Inter-rater

(for subjective measures)

Item Analysis
Item Difficulty Index

# people answering correctly

Item Distractor Analysis

Wrong answers in multiple choice

Item Discrimination Index

Items missed by high scorers vs low scorers

 Norm

Norm vs CriterionReferenced

Compare to others or norm group

Identify 5 best applicants for training
Remedial math offered to those who need it
most

Criterion

Qualified to perform a certain function

Compete against their own performance
Pass/fail grade most appropriate

Pre- & Post-Testing

Raw Gain Score Increase
Pre Avg.

Post Avg.

60

80

20 point average increase

Percent Increase
Post Pre
Pre

x
100

80-60
60

x
100 = 33% increase

1.
2.
3.
4.
5.
6.

Knowledge Tests
Write-in or Short Answer
Binary (including True-False)
Multiple Choice
Matching Column
Essay/Open Answer
Practical Application Item

The challenge is to test for higher-level thinking skills, such as

critical thinking (evaluating, predicting), problem solving,
synthesis, and creativity/innovation.

Write-In or Short-Answer
"The influencing approach that involves painting an
image of the future and the benefits or achieving the
goal are _______ _______."
Limited # of Correct Answers - Easy to Score
Does Not Adapt to "How" or "Why" Questions
Test Items Can Cue Correct Answer
Cannot be Machine-Scored

Binary
"A manager who believes that employees cannot be
trusted and will work only when being closely
controlled is called (a) a Theory X manager; (b) a
Theory Y manager."
"When reviewing a person's performance, you should
compare the person's actual results to the
performance standards agreed upon for that
employee. True or False
Easy to Score - Machine Scoreable
Questions Limited in Scope
Ambiguity - "Trick Questions"

Multiple-Choice
"Greater pressure aids carbonation because
a. it makes water boil
b. CO2 breaks down chemically
c. it forces CO2 molecules in between water
molecules
d. CO2 clings to the inside of the glass
Easy to Score - Machine-Scoreable
Can be More Complex than Binary Questions
Less Likely to Guess
Take Time to Develop - Logical Wrong Answers

Matching Column

Match the part of the gas combustion engine to

the function it performs.

Part
___1. Alternator
___2. Carburetor
___3. Spark Plug
___4. Radiator

Function
a. Cools the engine
b. Ignites fuel
c. Keeps track of mileage
d. Mixes oil and gas
e. Mixes gas and air
f. Recharges the battery

Essay/Open Ended Answer

"Describe the four major types of influencing
strategies, including when to use each, the factors
that would make each effective, and the limitations of
each."
Easy to Construct
Allows for Freedom in Answering
Adapts Well for "How" and "Why" Questions
Difficult to Score
Writing Ability May Affect Score

Practical Application Item

Using the customer profile provided, compose

three probing questions to determine the risk of
flight to another company.

Provide a scenario or realistic situation

Have learner compose, prepare, design, construct,
plan something (synthesis)
More than one right answer or approach
Use criteria in a rating scale or scoring rubric to
evaluate

Competency Demonstrations
Behavioral Checklist
Behaviorally Anchored Rating Scale (BARS)
Behavioral Frequency Worksheet
Behavioral Observation Scale
Effectiveness Rating Worksheet
Best-Solution Approach

Behavioral Results
Are people using the skills & behaviors they were
taught?
Is the work environment supporting the use of these
skills and behaviors?
Focus on whether the partnership of training and
management has worked... not in identifying which
partner had the most impact.

Micro & Macro Evaluation

Microevaluation
Individuals
Assists in individual skill development

Macro-evaluation
Groups
Identify benefits returned to organization

Behavioral Results
5 Critical Questions
Who is the client?
What is the business need?
What outcomes, in behavioral terms, are
expected?
How will you know these have occurred?
How long should you wait before checking?

Data Collection Methods/

Sources
Behavioral Observation
Interviews/
Focus Groups
Questionnaires

Questions about Work

Environment
Learner

Organization

Or the 7 buckets

Manager

Tracking Designs
Option 1: Pre-and Post-Training Design
Trainees

Pre-Training
X

Training
yes

Post-Training
X

Option 2: Control Group Design

Trainees
X
Non-Trainees X

yes
no

Option 3: Pre, Ending (Then), Post

Trainees

X
X
(self-report only)

yes
X (then)

Option 4: Post Only

Trainees

--

yes

Pre-Then-Post Level 3
Evaluation
Pre - Before I take singing
lessons, I rate myself as a
pretty good singer
Then - After I take singing
lessons, I realize that I was a
terrible singer back then
Post - Now that I have taken
lessons, I rate myself as a
GREAT singer

Non-Observable Results

Changes in cognitive skills, beliefs & values

Who is the client?
What is the business need?
What are the specific outcomes, in terms of
mental skills, values & beliefs?
What method will you use to measure the
outcomes?
How long should you wait before measuring?

Mental Skills
Ability to analyze situations,
make decisions, & solve
problems.

Operators determining why the quality of water is not

within the purity standards.
Sales personnel selecting product benefits that are
most relevant to the needs of particular customers.
Supervisors creating legal and appropriate questions
to ask job candidates.

Mental Skills
One-to-One Interviews with Participants

Have you had an opportunity to solve this type of

problem?
What thought process did you use?
How did using these steps affect the outcome?

Compare thought processes (steps) with what was taught.

Values & Beliefs

Growth in the value of working collaboratively rather
than competitively.
A sense of ownership in plant operation and the belief
that it is important to get involved in solving problems.
Bank tellers' belief that it is important to sell banking
services to customers.

Values and Beliefs

Questionnaires with Closed Questions
The most effective way for people to work in
our organization is to be cooperative, not competitive.
Strongly
Disagree
1

Disagree
2

Somewhat
Disagree
3

Somewhat
Agree
4

Agree

Strongly
Agree

1 = most important to you to achieve a good quality of life

2 = next most important
3 = third most important
_____ A job which pays me well.
_____ A loving family and home life.
_____ A feeling of achievement in what I do.

Operational Results

Measuring Impact on the Business

Impact of Training on a Specific Business Factor

Compare Results Across the Organization
Identify Job Behaviors with the Greatest Impact
Cost-Benefit Analysis/Return on Investment
(Phillips Level 5)

Why is there increased

pressure to show the
bottom line impact of
training?

Operational Results
7 Questions
Client and business need?
Cause of problem? How to maximize opportunity?
What operational results to track?
What knowledge & skills are causally linked to
operational results?
What is the cost of the program?
What info. will determine if results are occurring?
How long to wait?

When to Do ROI?
Factors to Consider

clear business need

scope, $$ of intervention
visibility of project
organizational support
complexity-other factors
ability to convert to $$
decision to be made
pre-post comparison
availability

needs assessment done

evaluation design
cost of evaluation
time to see results
importance to business

How do you do it?

Plan evaluation during needs assessment

Collect pre training data-Level 3 & 4
Collect post training data-Level 3 & 4
Isolate the effects of training
Convert data to monetary $$$ value
Tabulate program costs
Calculate ROI - costs vs $$ benefits
Identify intangible benefits
Adapted from Jack Phillips, Performance Resources Organization

ROI Model
Evaluation
Instruments

Evaluation
Purposes

Collect
Data
Evaluation
Timing

Tabulate
Program
Costs

Isolate the
Effects of
Training

Evaluation
Levels

Convert Data
to Monetary
Value

Calculate
Return on
Investment

Identify
Intangible
Benefits

Phillips, J. J. (1997). Return on investment in training and performance

improvement programs. Houston, TX: Gulf.

Collect Post Training Data

Compare with pre training data

Level 3 and Level 4
Sampling, Timing
Methods

organizational records
questionnaires
follow-up sessions
others

Isolate Effects of Training

Control Groups
Trend Line Analysis
Estimates of Trainings Effects (%)
participants, supervisors, management
% confidence

Convert Data to $$$

Hard Data
Soft Data
Direct Conversion
Expert Opinion
Participant, Supervisor,
Management Estimation

Tabulate Program Costs

Participant Costs
Instructor Costs
Instructional Development Costs
Facilities / Equipment Costs
Program Maintenance Costs

Training Impact

Return
or
Benefit

Investment
or
Cost

Training Impact Measures

Total Return on
Investment

Benefits
- Costs
Return

$60,000
-$20,000
$40,000

Benefit /
Cost Ratio

Benefits
Costs

$60,000
$20,000

Total ROI=
$40,000
BCR=
3 or 3:1

Net Benefits (Benefits - Costs)

X 100
Costs
ROI Percentage
(Net)
Net ROI %=
60,000 - 20,000
X 100
200%
20,000

Non-Monetary Benefits
How much is it worth?
saving lives by decreasing
emergency response time
increasing customer satisfaction
by fewer delays in airline travel
others

Evaluation Results, Conclusions,

Recommendations
Results
the outcomes of the analysis

Conclusions/Implications
judgments made from the results

Recommendations
made after careful considerations of
conclusions

Presenting Evaluation
Results
Who will receive report
What will be reported
Where will the report be presented
How will the report be presented

Now YOU can use evaluation

techniques to show HRDs value.

If we dont show our value, HRD

will continue to be discounted.

Evaluation techniques link

HRD to organizational goals.

Create an evaluation strategy in

partnership with your client.

Show how HRD enhances

organizational performance.

And everything is wonderful right?

Hold on a
minute!

Back to ROI
Are you being asked to prove
the worth of HRD?

Is ROI a savior or a nemesis?

Good News They Ask!

Bad News They Have

Doubts!

If youre asked or before youre

asked

Wake up call = Opportunity

Alignment & Collaboration
Performance Consulting
Find out what they really want to know!
ROE

Tobins Law
If you start and end all of your learning
efforts by focusing on the organizations goals,
you will never be asked to do an ROI analysis
to justify your budget.

http://tobincls.com/fallacy.htm

Heres a thought!
The process is more important
than the numbers.
What do YOU think?