CORPORATE LEADERSHIP COUNCIL

FEBRUARY 2004
www.corporateleadershipcouncil.com

LITERATURE KEY FINDINGS

Assessing Training Effectiveness

METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS

The Modified KirkPatrick Model

Key metrics for evaluating

training programs under the
Modified Kirkpatrick Models:
!
!
!
!

Amount of knowledge acquired

Change in employee on-the- job
behaviors
Change in profitability as
expressed through productivity,
quality of work, and sales
Trainee satisfaction

The Modified KirkPatrick Model assesses various metrics connected to training effectiveness
with each level progressively building on the previous. This section provides an overview of
1,2,3
the Modified Kirkpatrick Model.
!

Level one measures the following metricsTrainee satisfaction with aspects of the
program including topic, speaker etc., and quality of program, and applicability of
materials taught.

ToolsAttendees completed surveys and evaluation forms assessing satisfaction with training,
baseline comparison for training program attendance, and hours of training
Between 85 to 89 percent of organizations assess training programs at the first level

Level two measures the following metricThe amount of knowledge acquired during
training including improved skills and professionalism.
ToolsExams, self-assessment, facilitator assessment, active testing (simulations, case
studies, skill practices etc.), and team assessments
Between 37 to 41 percent of organizations assess training programs at the second level

Level three measures the following metricThe extent to which participants change
their on-the-job behavior.
Tools360 Degree evaluations, formal surveying of skills prior to training and following training,
on-the-job observation, follow-up focus groups, action planning, and program assignments
Between 12 to 17 percent of organizations evaluate training programs at the third level

Case in Point: Level Three Evaluations

4
Conducted by Hutchinson Technology
Hutchinson Technology conducted level three Kirkpatrick evaluations by
assessing employees skill levels prior to training by surveying customers,
supervisors, and peers on how frequently employees displayed particular
behaviors. After the training, the company surveyed the same individuals
using a similar survey. The survey revealed that employees used the
selected skills 75 percent of the time, as opposed to 50 percent of the time
prior to undergoing training.

Level four measures the following metricsThe impact training has on profitability,
productivity, quality of work, sales, turnover, and expenses.
ToolsBaseline and trend-line comparisons of hours in training, training session attendance,
employee turnover, sales rates, profits, expenses, customer satisfaction, and monitoring of
overall business performance
Approximately seven percent of organizations utilize four of the five levels of the Modified
Kirkpatrick model4

2004 Corporate Executive Board

CATALOG NUMBER: CLC11TQ4SZ

CORPORATE LEADERSHIP COUNCIL

ASSESSING TRAINING EFFECTIVENESS

PAGE 2
KEY FINDINGS

METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS (CONTINUED)

Alternate metrics for evaluating
training programs are the
following:
!

Alternate Methods and Metrics to Evaluate Training Effectiveness

!

Customer service gains and

operational efficiency
Gain formula
Internal promotions
Return on expectations

!
!
!

Compaqs Success Case ModelCompaq evaluates its training programs after their
5
conclusion based on content learned as well as job-application as highlighted below:
Post-training surveyEmployees fill out a survey testing them on the fundamentals of the
content presented, and polling on their opinions regarding the applicability of the training to their
job requirements
Post-training interviewsCompaq interviews the employees who provided the 15 best
responses and the 15 worst responses to obtain information about the programs strengths and
weaknesses.

Verizons Return on Expectations ModelVerizon evaluates training programs prior

6,7
to the programs start based upon control groups and productivity outcomes:
Prior to trainingVerizon divides employees into two groups: a control group that will not
undergo training and a group that is trained. The controlling executive documents his/her
expectations for the trainings goals
Post-trainingThe controlling executive indicates whether results met expectations.
Analysts then compare the answer to the difference in productivity between the control group
and the group that underwent training

As organizations plan their own

training metrics, they should
keep the following CEO-type
questions in mind:
!
!
!

Does training work fast enough

to make a difference?
Does having a well-trained
employee really make a
difference in our industry?
Does having a well-funded
training function help the
organization attract and retain
the best people?
If the training budget was
doubled, would productivity
double?

-Managing Training and

Development,
Nine Managers Reveal the Business
Impact of their Training Efforts
March 2003

Southwest Airlines Balanced Scorecard ModelSouthwest evaluates training

programs prior to the programs start, based on customer service gains and operational
8
efficiency.
Prior to trainingTranslate value drivers, such as customer service and operational efficiency
into metrics. In this manner, Southwest implements training evaluation at the top of the
organization and works down rather than the inverse method followed by most models
Post-trainingAnalyze metrics using tools such as baseline comparisons and surveys to
determine if targeted goals are met

IBMs Web-based Assessment ToolIBM uses a Web-based assessment tool to

9
evaluate its leadership development training:
1.

2.

Participants receive a questionnaire, at least six months after completing the development
training, that asks for feedback on how the training impacted participants behaviors.
The questionnaire asks the participants to assign a monetary value to the training they
received
The responses are correlated to the departments and functions of participants, which helps
IBM understand which training components are most valuable to specific employee groups

Merck & Companys Training Effectiveness MetricMerck & Company uses a

unique formula, detailed below, to measure the impact of specific training programs.
With this model, Merck determined that its average ROI for training programs is
10
84 percent and terminated 53 programs that were not producing high enough returns:
CalculationGAIN = d x SD$ x JSI x N

dShift in performance by average individual undergoing training expressed in standard

deviations from pre-training average
SD$The value in dollars of one standard deviation of performance shift
JSIPercentage of job skills impacted by training
NNumber of participants who underwent training

Fords Career Path Tracking System measures the effectiveness of its Leadership
Development Center by monitoring the career paths of alumni of the center. This helps
the company assess whether programs achieve the goal of producing successful leaders
11
that can be promoted.

2004 Corporate Executive Board

CORPORATE LEADERSHIP COUNCIL

ASSESSING TRAINING EFFECTIVENESS

PAGE 3
KEY FINDINGS

METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS (CONTINUED)

Alternate Methods and Metrics to Evaluate Training Effectiveness (Continued)
!

The tactics to determine

training success are the
following:
!
!
!

Dells Web-based Global Training Measurement SystemIn 1998, Dell Learning

launched a Web-based global measurement system that provides online access to
training statistics at global, business, segment and department levels. For example, the
Training Snapshot Report documents all training activity including classes taken, total
tuition and enrollment and total hours by region, business, and segment. The Training
by Type Report documents training activity by category including management,
executive, customer service, sales, new product, technical, new hire, business initiatives,
12
professional development and compliance.

TACTIS TO DETERMINETRAINING SUCCESS

Tactic # 1: Collect Data to Demonstrate Change in Behavior
!

Collect date to
demonstrate change in
behavior
Isolate the effect of
training
Lower turnover by
spending on training

To determine if training has been successful, organizations can collect data that
demonstrate changes in behavior through evidence provided via employee observation
in specific situations both before and after training. The data can be collected in the
13
following ways:

Surveys
Questionnaires
On-the-job observation
Post-program interviews
Focus groups
Performance monitoring
Performance contracts in which a participant, the instructor, and the participants supervisor
agree on specific outcomes from training.

Tactic # 2: Isolate the Effect of Training

!

Another way to determine if training has been successful is by isolating its effect.
14,15
Literature reveals the following strategies to isolate the effect of training:

360-degree feedbackFirms can measure the change in behavior from different

perspectives.
Control groupsExperimental group receives training while a control group does not.
Post-training measurements demonstrate performance differences between the two groups.
Customer inputCompanies solicit input on service quality directly from customers.
Follow-up assignmentsThese can demonstrate to companies the application of training
content.
One-on-one interviewsCompanies can glean specific details on the results of training i.e.,
by asking the participants how much of the change in behavior is attributable to training.
Surveys and questionnairesThese can capture participants accomplishments after
training.
Trend-Line analysisCompanies draw a line from current performance to future
performance, assuming the trend will continue without training. Then performance after
training is compared to predicted performance without training on a trend-line.

Tactic # 3: Lower Turnover Rates by Spending on Training

!

Research reveals that companies that spend more than average amount on training
have a higher placement of internal hires and lower annual turnover rates. Companies
that spend $273 dollars per employee have less than seven percent annual voluntary
turnover. Companies that spend $218 dollars per employee in training have more than
16
16 percent annual voluntary turnover.

2004 Corporate Executive Board

CORPORATE LEADERSHIP COUNCIL

ASSESSING TRAINING EFFECTIVENESS

PAGE 4
KEY FINDINGS

Ajay Pangarkar and Teresa Kirkwood, "Systematic Strategies," CMA Management (1 December 2002).
(Obtained from Factiva).
Mark Van Buren and William Erskine, "American Society of Training and Development State of the Industry Report
2002," www.astd.org/ (February 2002). [Accessed Feb 23 2004]
(Due to copyright restrictions, a copy of this report cannot be provided).
3
Jack J. Phillips, A Rational Approach to Evaluating Training Programs Including Calculating ROI.
The Journal of Lending & Credit Risk Management (1 July 1997) (Obtained from Factiva).
4
Beverly Geber, "Does Your Training Make a Difference? Prove It!" Training (March 1995).
(Obtained from Lexis-Nexis).
5
Author Unknown, "Training ROI How Compaq Uses the Success Case Method to Prove ROI of Training,"
Managing Training & Development (July 1, 2002). (Obtained from Factiva).
6
John Berry, "Corporate Training -- The E-Learning Center," InternetWeek (6 November 2000).
(Obtained from Factiva).
7
Pam Leigh, "Training's New Guard 2001 (Profilles of New Employee Trainers)," Training and Development
(1 May 2001). (Obtained from Factiva).
8
Author Unknown, "How Southwest Airlines Developed its Balanced Scorecard Analysis,"
Financial Analysis, Planning & Reporting (1 July 2002). (Obtained from Factiva).
9
Corporate Leadership Council, Voice of the Leader, Washington: Corporate Executive Board (September 2001).
10
M. Quintin Jarrett, Measuring Return on Training at Merck & Company, Ernst & Young Center for Business Innovation
(Date Unknown). (Obtained through www.businessinnovation.ey.com). [Accessed 30 January 2003].
(This source is no longer available online).
11
Stewart D. Friedman, Leadership DNA: The Ford Motor Story, Training & Development (March 2001).
(Obtained through Lexis-Nexis).
12
John Cone, How Dell Does It, Training & Development (1 June 2000). (Obtained through Factiva).
13
Carroll Lachnit, "Training Proves Its Worth," Workforce (September 2001). (Obtained through ProQuest).
14
Carroll Lachnit, "Training Proves Its Worth."
15
Jack J. Phillips, "Was it the Training?," Training and Development (March 1996). (Obtained from Lexis-Nexis).
16
Caroll Lachint, "Training Proves Its Worth."
2

2004 Corporate Executive Board