Evaluating

Learning and Development Activities

Formal Report
(approx. word count 3.000 words)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 1

Contents
Table of figures ....................................................................................................................................... 3
Introduction ............................................................................................................................................ 4
Methodology........................................................................................................................................... 5
Nature of Evaluation ............................................................................................................................... 6
Purposes of Evaluation ........................................................................................................................... 7
Related Terms and Processes ................................................................................................................. 8
Purposes of evaluation for 3 different stakeholders ............................................................................ 10
Return on investment and return on expectation and its measurement ........................................ 12
Evaluation Methods .............................................................................................................................. 14
Evaluation Tools .................................................................................................................................... 17
Conclusions ........................................................................................................................................... 21
References ............................................................................................................................................ 22

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 2

Table of figures
Figure 1 Ripple Model of Evaluation
Figure 2 Phillips ROI model
Figure 3: The Logic of Training (Robert O. Brinkerhoff)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 3

Introduction
Firstly, I will discuss nature and some of the purposes of conducting an evaluation and how
the term evaluation differs from terms such as validation, assessment, testing, monitoring and
review. I will then further elaborate on some of the purposes and/or benefits of evaluation for
different stakeholders. Subsequently, I will try to explain the concepts of return on investment and
return on expectation and possible ways of measuring them. Afterwards, I will present 3 different
evaluation methods and I will wrap up this report by providing various evaluation tools and their
advantages and drawbacks.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 4

Methodology
I drew on the following primary research:

Learning and Development Practice 2nd edition

ASTD Handbook
Telling Aint Training
CIPD material
How to Measure Training Results
Designing and Delivering Training
Current evaluation practice in our organisation
Discussion with colleagues from L&D Graeme Turnbull and Chris Evans
Discussion with Operations Trainer Ian Sommerville

I also drew on the following secondary research:

Online material on CIPD website

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 5

Nature of Evaluation
There are many definitions of the term evaluation. Lets look at some of the definitions found
in dictionaries of English language.

A judgment about how good, useful, or successful something is

Longman Dictionary of Contemporary English (5th Edition)

The process of studying all the available information about sb/sth and forming an opinion about
them/it
Oxford Business English

Beevers and Rea (2010) define the term evaluation as:

Measuring and analysing various aspects of L&D provision, with a view to determining the
effectiveness and value of the provision and informing decisions about how it can be improved

Based on a factsheet Evaluating Learning and Development from CIPD website the term evaluation
is defined in detail as follows:

The evaluation of learning and development (L&D) is the formal or informal assessment of the quality
and effectiveness of an employers learning and development provision, usually by some measure of
the merit of the provision itself (the input, for example the quality of course content and
presentation) and/or by monitoring its impact (the outcomes, for example improved
skills/qualifications or enhanced productivity/profitability).

In general, learning is the desired goal of any training programme. However, if learning does not create
the desired change in behaviour in the workplace, we have not successfully accomplished the overall
goal of applying learning to the workplace. This is sometimes quite difficult to measure in the first
place. Lets have a look at purposes of evaluation.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 6

Purposes of Evaluation
There are many reasons for evaluating learning programmes. In Chapter IV Measuring and
Evaluating Impact of ASTD Handbook by Biech (2014), Donald L. Kirkpatrick summarised some of them
in the below list:

to determine whether the particular programme should continue or be dropped

to learn how a learning programme can be improved
to justify the training budget
to ensure learning compliance
to maximise the value of training
to align training with strategy

Beevers and Rea (2010) look at evaluation in terms of different stakeholders. In any evaluation, there
will be learner, trainer and organisational interests considered. They also list other internal and
external partners, some of them are:
L&D and HR Team
Finance function
Management
Awarding, accrediting and regulatory bodies

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 7

Related Terms and Processes

I have tried to draw on some of the definitions of the term evaluation. Lets discuss some
other terms which are associated with evaluation but which have slightly different meanings. Those
terms are:

Validation

Assessment
Testing

Monitoring and review

Return on investment (ROI)

Beevers and Rea (2010) define the above terms in more detailed way.

Validation

Validation is usually included within evaluation process. Revisiting learning outcomes at the end of a
learning session or programme and making sure they are closely tied in with organisational goals and
especially achieved is just one example of validation. The evaluation of a learning intervention must
prove its relevance, alignment and must be measurable (RAM approach). In order to achieve that, all
four of the Kirkpatricks evaluation levels should be established as each bit of evidence can help to
build the overall case of the programmes effectiveness.

Assessment

Assessment tracks individual progress and can be measured by a range of methods. It can be
undertaken prior to, during or after a learning activity. Assessment can be formal or informal
depending on the learning outcomes.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 8

Testing
Testing is similar to assessment. It can be described as an assessment activity and its format (oral or
written) will vary depending on what is tested. Stolovitch and Keeps (2002) state the following key
points about testing:
Testing is a natural part of learning. It helps both learners and trainers confirm performance
objective attainment or identify where something is missing and requires corrective feedback.
Testing doesnt need to be threatening but sometimes it is.
Testing is an excellent way to teach. It lets the learner try out her or his learning with a bit of a
challenge.
Testing requires that feedback be given following the test. It either confirms objective attainment
or offers corrective information specific to how the learner performed on the test.
Because testing requires active learner engagement, it should be used frequently. Meaningful
engagement enhances comprehension and retention.
However, we need to be careful as testing frightens adult learners, particularly those who may not
have had great success with it at school.

Monitoring and review

Monitoring and review is a collection of activities to gauge learner progress towards goals. Information
gained from monitoring and review of learning activities is often useful to evaluation process. It
identifies development areas and whether any further learning intervention is needed.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 9

Purposes of evaluation for 3 different stakeholders

The main stakeholders in the evaluation process are:

1. Learner
2. Trainer
3. Organisation

Lets have a look at some of the purposes and benefits of evaluation for the above stakeholders.

Potential purposes (and benefits) of evaluation:

Learner

Enable learner to express their opinion and influence future development of learning programme

Demonstrate a recognition of learners inputs

Helps to identify any further training needs and/or identify future talents

Enable learner to identify his/her preferred learning styles

Trainer

Feedback on trainers performance

Identification of areas of development

Challenges the trainer to think out of the box and maybe makes us rethink our ingrained
approach

Achievement of trainers own objectives

Establish whether learner enjoyed the learning and whether selected learning methods were
effective

Helps trainer to test out new techniques and methods

Helps to improve the overall learning experience

Trainer can justify the validity of the learning to management

Gives evidence towards trainers credibility

Organisation

Align with organisational goals

Financial expectations - ROI

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 10

Audit reasons

Increased productivity, decreased attrition, KPIs

Increases competitiveness

Enables the organisation to look for exceptional employees

Enables to maintain and gain professional qualification and accreditation

Establish costs of L&D

Supports the role of L&D function within an organisation

Those were just some of the purposes of evaluation. I will now discuss the terms ROI and ROE.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 11

Return on investment and return on expectation and its

measurement
Beevers and Rea (2010) define the term return on investment (ROI) as a measure of the
financial impact of training/learning activities on an organisation. ROI is usually considered to be the
highest level of evaluation. In order to calculate the ROI, I need to be able to assess costs of training
provision and its financial benefits.
Measuring ROI should be a requirement in most organisations. ROI was popularised by Jack Phillips.
There are different formulas which can be used to measure ROI. In simple terms, the ROI formula is:
(Return Investment)
Investment

Its typically expressed as a percentage or as a cost-benefit ratio. ROI can be seen as an extension of
Kirkpatricks evaluation model. Thus ROI can be seen as level 5 of Kirkpatricks model. Collecting level
4 data helps to convert them to monetary values.

Return on Expectation
Beevers and Rea (2010) state that return on expectation (ROE) is more concerned with
stakeholder satisfaction and whether learning expectations of key stakeholders have been met rather
than monetary values.
According to a CIPD factsheet The value of learning: A new model of value and evaluation, we should
be asking the following key questions which underpin a return on expectation approach:

What were the original expectations of organisation stakeholders for the learning or training?
Have those expectations since changed?

What changes have occurred as a result of the learning processes?

To what extent have stakeholder expectations been met?

Measures of ROE can be achieved through focused dialogue, real understanding and mutual trust
between key stakeholders senior management and L&D function. Measuring ROE requires softer
and qualitative data compared to hard numerical data. CIPD has been championing this approach for

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 12

some time now. ROE is described in detail in one of CIPD publications The Value of Learning: From
return on investment to return on expectation by Valerie Anderso.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 13

Evaluation Methods
There are various approaches (frameworks) to evaluation. I have listed some of them below:

Ciro Model
ROI
The Ripple Model of Evaluation
Triangulation in evaluation
Kirkpatricks Model

Let us examine some of them in more detail. Simmonds (2002) explains in his book Designing and
Delivering Training so called Ripple model of evaluation. Figure 1 depicts visually the Ripple Model
of evaluation.

Figure 1 Ripple Model of Evaluation

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 14

Simmonds (2012) illustrates learning evaluation as constantly moving, dynamic and organic and not
as something static or linear. The satisfaction of learners needs as a requirement is in the centre. This
is translated into learning which is transferred to the workplace and changed job performance. Finally,
its value is determined to the wider organisation. So called ripples also flow back towards the centre.
The learning will be constrained by organisational culture, financial constraints and stakeholder
expectations. The organisation will affect scope of change and learning in the department as well as
on individual level.

I have briefly explained the concept of ROI by Phillips in the previous section. Phillips explained his ROI
model in his book How to Measure Training Results (2002). Collecting level-five data (hard and soft)
and converting them to monetary value is quite demanding and complex. Phillips model is illustrated
in Figure 2 below.

Figure 2 Phillips ROI model

The most widely used evaluation model is Kirkpatricks four level of evaluation. It consists of:

Level 1 Evaluation: Reaction

Level 2 Evaluation: Learning

Level 3 Evaluation: Behaviour

Level 4 Evaluation: Results

All four levels are described in Evaluating Training Programs: The four levels (Kirkpatrick and
Kirkpatrick, 2006). Most companies conduct the evaluation at least on level 1 and level 2. They are
generally less costly and easier to measure (happy sheets, formal and/or informal assessment tests,
quizzes, etc.) whereas evaluation on level 3 and 4 requires more planning and is more difficult to
Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 15

measure. It also requires some time in order to be measureable. Nonetheless, its vital to have a
system of evaluation in place which aims for all four levels.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 16

Evaluation Tools
We have different ways to evaluate the learning session and determine if the transfer of
knowledge, skills and change of behaviour in the workplace has been successful. Even though the
change of behaviour as a result of learning intervention is much more difficult to measure, we can
assess the learners progress (knowledge and skills) throughout the training as well as at the very end.
Assessment can be classified as part of evaluation (Kirkpatricks level 2). We generally distinguish
between formative and summative assessment. Under formative assessment, we understand
continuous assessment throughout the learning session or programme. It is used to map the learners
progress. The goal of summative assessment is to evaluate student learning at the end of the learning
session against some standard or benchmark. In more lecturer-based environment (such as school/uni
environment), we can come up with the following overview of different assessment tools.

Formative and Summative Assessment tools (source: Cambridge Community)

Lets again further examine testing. Stolovitch and Keeps (2002) state that tests must be linked to
overall objectives of the learning session and must be created with caution as failing the exam may
be associated by the learner with I am a failure. However, tests are generally a good
assessment/evaluation tool. Tests throughout the training are used in my workplace and both
declarative and procedural knowledge is tested. In order to put the learners at ease, I do a funny
learning activity such bubble busters, who wants to be a millionaire, etc. before every test.
Stolovitch and Keeps (2002) present a variety of ways how a test can look like. Here are just some of
them:

Binary test

Matching test

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 17

Multiple-choice test

Completion test

Short-answer closed question test

Open-ended essay test

Each of them has its advantages and drawbacks. It is good practice to mix them up depending whether
we want to test procedural or declarative knowledge.
After each learning programme, which takes 4 weeks, my learners fill in immediate evaluation forms
(Kirkpatricks level 1), so called happy sheets. This questionnaire is designed to collect both
quantitative and qualitative data. It contains open, closed questions and different scales of 1 to 10.
Nancy Kristiansen (2014) in Chapter 28 of ASTD Handbook lists some of the benefits and advantages
of using a well-designed level 1 evaluation strategy:

Improve decision making

Prepare for levels 2, 3, and 4 cross-functional strategies

Meet regulatory compliance requirements

Reinforce a culture of learning

Involve participants as contributors to the betterment of the training process

Inquire directly whether or not what was learned met participants expectations
and objectives

Lets have a look at Kirkpatricks level 3 evaluation in greater detail. Level 3 gathers the information
on how (and how well) the skills, knowledge and attitudes, that were acquired in a learning
programme, are being used in workplace performance (Brinkerhoff and Mooney, 2014, p. 524). In one
word, level 3 is about application. The below figure depicts the value to the organisation through
improved job performance as a result of a learning intervention.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 18

Figure 3: The Logic of Training (Robert O. Brinkerhoff)

Evaluation tools on level 3 can range from interviews, direct observation, indirect observation (e.g.
call monitoring), analysis of KPIs (analyses effects of behaviour rather than behaviour itself), surveys,
the success case evaluation method (Brinkerhoff, 2003, 2006). Currently I utilise direct and indirect
observation and surveys (3 months evaluation form) as available evaluation tools on level 3. Using
surveys leave a lot to be desired though. People may feel surveyed out and resistant to taking the
time to complete them. They are also subject to bias of people telling you what they think you want
to hear. Typical answers are: Of course, Im using my training, Im supposed to after all. Sitting with
the employee and watching his job performance directly provides me or team manager with much
more accurate data and is less susceptible to misinterpretation bias. However, its impractical and
time consuming and the person being observed might feel he or she is being watched. This way, its
closer to level 2 evaluation rather than a level 3. Choosing and scoring random calls and giving
immediate feedback is more appropriate for level 3. This is also part of weekly PEP sessions conducted
in my workplace. Lets look at the most common question regarding the timing of level 3 evaluation.

How soon after training should we follow up with trainees?

Brinkerhoff and Mooney (2014, p.536) answer the question as follows:

It is a mistake to use a standard timeframe, such as after three months, six months, nine months, and
so forth. The right way to determine the correct lag time prior to level 3 inquiry is to ask: How soon
after the training is it reasonable to expect that people should have been able to apply their learning?
In some cases, this may be a matter of days, such as with call-centre employees learning a technique
for routing calls more effectively. In other cases, such as with purchasing executives , it may be

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 19

necessary to wait a full quarter. But always it is wise to remember this guideline: most training, if its
not applied in the first two to three weeks after training, will probably never be used.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 20

Conclusions
I tried to illustrate the importance and the benefits of having some kind of framework of
evaluation in place. Failing to so will have drastic consequences on

the

overall

business

unit/organisations performance in todays highly competitive environment. I hope this report will
provide some valuable insight for other L&D practitioners in my organisation as well and further
improve and extend the evaluation system we have in place.

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 21

References
Books
Biech, E., ed. (2008). ASTD Handbook for Workplace Learning Professionals. Alexandria, VA: ASTD
Press.

Beevers, K. and Rea, A. (2010). Learning and development practice. London: Chartered Institute of
Personnel and Development.

Kirkpatrick, Donald L., and James D. Kirkpatrick. 2006. Evaluating training programs: The four levels.
3d ed. San Francisco: Berrett-Koehler

Stolovitch, H. and Keeps, E. (2002). Telling ain't training. Alexandria, VA: ASTD.

Simmonds, D. (2003). Designing and delivering training. London: Chartered Institute of Personnel and
Development.

Phillips, J. and Stone, R. (2002). How to measure training results. New York: McGraw-Hill.
Online

CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2015) Evaluating learning and

development [online]. Factsheet. London: CIPD.
Available at http://www.cipd.co.uk/hr-resources/factsheets/evaluating-learning-talentdevelopment.aspx (Accessed 20 May 2016)
http://www.cipd.co.uk/toolclicks/learning/training-tools/evaluating-training/default.aspx

CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2015) Evaluating learning and

development [online]. Factsheet. London: CIPD.
Available

at

http://www.cipd.co.uk/hr-resources/factsheets/costing-benchmarking-learning-

development.aspx (Accessed 20 May 2016)

http://www.trainingzone.co.uk/deliver/coaching/cipd-tells-trainers-to-move-away-from-traditionalevaluation (Accessed 20 May 2016)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 22

CHARTERED INSTITUTE OF PERSONNEL AND DEVELOPMENT. (2007) The Value of Learning: From
return on investment to return on expectation. Factsheet. London: CIPD
Available at Available at http://www.cipd.co.uk/hr-resources/factsheets/evaluating-learning-talentdevelopment.aspx (Accessed 20 May 2016)
http://cambridge-community.org.uk/professional development/gswafl/index.html (Accessed 20 May
2016)

Martin Frano | CIPD Foundation Certificate in Learning and Development Practice | May 21, 2016 23