Fall

08
October 9
16

Project 3: Levels 1-4 Evaluation Report

Melissa Fruechte

EDCI 577 Strategic Assessment and Evaluation

Executive Summary
Evaluation Purpose
The purpose of this evaluation was to evaluate the effectiveness of the [Product] Basics
course to determine if the course was effective in increasing top line growth and sales revenue of
commercial and residential industry solutions. The [Product] Basics program was offered as a 40-
60 minute eLearning module offered to sales personnel whose primary focus is the [Product]
product line.

The evaluation questions for each level were:

Level 1: How did learners react to the relevance, format, functionality and ease of use of
the online course format?
Level 2: Have learners gained knowledge and met learning objectives by completing this
module?
Level 3: Have sales personnel successfully sold [Product] Basics solutions to commercial
and residential stakeholders?
Level 4: How much did total sales for the [Product] product line increase because of the
training curriculum?
Evaluation Design
N learners participated in this training and data was received from them at levels one
through four. N supervisors participated at level 3 to gather observational data. One sales manager
assisted in helping gather sales numbers for level 4 evaluation.
Level 1 data was collected from learners using the Learner Satisfaction Survey upon
completion of the eLearning module. Level 2 data was collected from learners using the
[Product] Basics Final Assessment, also upon the learners completion of the eLearning
module. Level 3 data was collected using the Sales Checklist: Self-Assessment & Observation
at intervals 2-3 months post-training as well as 6 months post-training completion. Level 4 data
was collected using the Sales Data Collection data sheet two quarters prior to learners
completing the eLearning module as well as four quarters post-completion of the module.
Quantitative data was collected and analyzed using data from each of the four levels,
including Likert-scale responses, open-ended questions, multiple choice and Yes/No responses.

Findings
Learner Reaction
Learners reacted favorably to the relevance of the training and were extremely satisfied
with the functionality and navigation of the course. The one area in which learners did not react
overwhelmingly favorably to the course was the confidence in their ability to apply the content in
their daily work.

Achieved Learning
Learners achieved an increase in mastery of content as identified by their pre- and post-
test scores. Learners increased their ability to explain the high-level benefits of the [Product] by
N%, their ability to demonstrate correct [Product] connections by N%, their ability to repair a
bent or kinked [Product] tubing by N% and increased their ability to identify the different support
systems for [Product]-a tubing by N%.

Performance Observation
 Data gathered from learners and supervisors indicated that learners were consistently able
to perform each of the six behaviors at 2-3 months and six months post training both when self-
assessed and when observed by a supervisor. The only behavior that showed a slight decline was
the ability to sell both commercial and residential solutions. The solution for this decline is
addressed in the recommendations section.

Increase in Top Line Revenue

A comparison between sales data preceding the completion of the [Product] Basics
module and post-completion showed that sales increased by N% in the first quarter post-training
completion, N% in the second quarter post-training completion, N% in the third quarter post-
training completion and N% in the fourth quarter post-training completion. These findings
provide proof that the [Product] Basics has contributed to the original purpose of its initiation: to
increase top line growth and revenue of the product line.

Recommendations and Conclusions

As a result of this evaluation, it is recommended that the [Product] Basics course
continue to be offered for all sales personnel responsible for selling this product line, with the
opportunity to partake in refresher training bi-annually to ensure sales continue to increase.

Introduction
Purpose
The purpose of this evaluation was to assess the value and worth of the [Product] Basics
program, housed within the overarching Sales curriculum. The Sales curriculum consisted of four
separate topics with the [Product] Basics module being one of those four topics. Additionally, this
evaluation will serve as a resource and asset in future decision making about improvements to the
course (Fitzpatrick, Sanders & Worthen, 2010). The company will use this evaluation as a
resource to identify the overall effectiveness of the objective to drive top line growth and revenue
of commercial and residential industry solutions by its sales personnel.

Instructional Program Overview

The title of the program evaluated was [Product] Basics. The subject of the program was
foundational skill and knowledge of the product in order to develop the skills needed to drive top
line growth of commercial and residential industry solutions. The learning objectives of the
course were that upon completion of the overarching curriculum, the learner would be able to
successfully sell the [Product] solutions to commercial and residential stakeholders. In order to
achieve this objective, the [Product] Basics course will help learners explain the high-level
benefits of [Product] products, identify correct handling and storing techniques for [Product]
tubing, demonstrate correct [Product] connections, repair bend or kinked [Product] tubing and
identify the different support systems for [Product]-a tubing.
The format of the program is a self-paced eLearning module that is one of multiple
modules within a comprehensive online Sales curriculum. This program is a 40-60 minute
module. This program was created to assist sales personnel in developing stronger product
knowledge in order to more effectively sell solutions within the [Product] line. The entire
curriculum was designed to focus on developing the skills needed to drive top line revenue
growth for the company. The company was finding that sales personnel did not have the
knowledge or skills needed to perform effectively and thus, this program was created.

Context
The company is a leading international provider of systems and solutions for heating,
cooling and safe water delivery systems. The company is for-profit that has operations in thirty
countries and sells its products to commercial and residential buyers in almost 100 countries. The
companys main headquarters are located in Helsinki; however, this evaluation was
commissioned by the company location in Minnesota, USA as that is where the [Product] is
manufactured as well as where the training for the [Product] is designed.

Target Audience
The learners who have completed this training are sales personnel whose territory and
focus is the [Product] market. The learners range from those who are new to sales and extend to
experience levels of 20+ years in sales. Learners academic backgrounds range from a high
school degree to a bachelors degree in manufacturing or business.

Evaluation Questions
Level 1 Evaluation Question: How did learners react to the relevance, format,
functionality and ease of use of the online course format?
Level 2 Evaluation Question: Have learners gained knowledge and met learning
objectives by completing this module?
Level 3 Evaluation Question: Have sales personnel successfully sold [Product] Basics
solutions to commercial and residential stakeholders?
Level 4 Evaluation Question: How much did total sales for the [Product] product line
increase because of the training curriculum?

Evaluation Plan and Procedures

Evaluation Design
This evaluation implemented a one-group descriptive study in the form of a reaction
sheet at the end of the course module for level one data collection. Dow (1992) indicates that this
method of evaluation can be utilized when descriptive information about the course is desired.
For the collection of level 2 data, the study utilized a one-group pre-assessment/post-assessment
evaluation design. Dow (1992) states that a one-group pre-assessment/post-assessment evaluation
design allows for the determination of change on outcome variables. Learners were assessed prior
to the training with a pre-assessment, during the module, learners had formative assessment
questions incorporated into the module and learners were also give a post-assessment. Level 3
evaluation was designed as a checklist, as Shrock and Coscarelli (2008) identify as a tool to
reduce subjectivity of the individual who is doing the rating. The checklist was developed to
easily identify the desired learner behaviors while they are on sales calls or in the field. The
evaluation design for level 4 was created to compare pre- and post- completion sales data to
identify if the training has met its objectives. Kirkpatrick and Kirkpatrick (2006) suggest this
before and after approach, which is especially applicable to a sales training scenario.

Sample
The sample consisted of all 57 members of the target audience. All Sales personnel
assigned to this product line were required to take this course and their performance was used in
this evaluation study. The sample was the same across both level 1 and level 2 evaluations.

Instruments
Level 1 data was collected using the Learner Satisfaction Survey (See Appendix A).
The survey was delivered to learners electronically automatically upon completion of the
[Product] Basics eLearning module. The Learner Satisfaction Survey consists of five Likert-
scale questions, three open-ended questions and one Yes/No question. Reaction sheets are
recommended by Kirkpatrick and Kirkpatrick (2006) to collection level 1 reaction data.
 Level 2 data was collected using an objective test titled [Product] Basics Final
Assessment (See Appendix B). The assessment was delivered electronically upon the learners
completion of the [Product] Basics eLearning module. The [Product] Basics Final Assessment
is an objective test and contains ten questions. There is one True/False question, five multiple-
choice questions and four interaction/simulation questions. As Kirkpatrick and Kirkpatrick (2006)
state, a pencil-and-paper, or objective test can be used to identify if an increase in knowledge has
occurred.
Level 3 data was collected using a checklist titled Sales Checklist: Self-Assessment &
Observation (See Appendix C). The checklist consisted of two parts: self-assessment format and
observation by an individual in a supervisory role. Each behavior in the checklist that the learner
and supervisor were asked to witness correlated to one of the objectives of the eLearning module.
The checklist was administered two ways: for the learners self-assessment, a pencil-and-paper
format was used, for the supervisory observation an in-person observation of the learners
performance was utilized to complete the checklist. The self-assessment and observation took
place at two intervals: 2-3 months post-training and 6 months post-training. The checklist consists
of six observable behaviors that can be validated with a Yes or No indicator of behavior. As
Shrock and Coscarelli (2008) identify, a simple checklist radically reduces the degree of
subjective judgment required of the rater (p. 190).
Level 4 data was collected using a data sheet titled Sales Data Collection (See
Appendix D). The data sheet collects quarterly sales information for each product that was
addressed in the eLearning module. The data was collected from the Sales Manager. The layout of
the data sheet includes each learners sales information per product identifying his/her sales in the
two quarters preceding his/her completion of the training and then each subsequent quarter (for a
year) after completion of the training with a summary of percent increase or decrease in sales per
product line. The data will be collected in an online spreadsheet. Kirkpatrick and Kirkpatrick
(2006) encourage the practice of measuring results both before and after the program (p. 66) and
mention that this before and after comparison approach is especially applicable in a sales training
scenario; which is the goal of this training.

Data Collection Procedures

Level/Evaluation Instrument Subjects/Participants Person Timeframe

Question Responsible
Level 1: How did Learner Learners LMS Upon
learners react to the Satisfaction administrator completion
relevance, format, Survey of the
functionality and eLearning
ease of use of the module
online course
format?
Level 2: Have [Product] Learners LMS Upon
learners gained Basics Final administrator completion
knowledge and met Assessment of the
learning objectives eLearning
by completing this module
module?
Level 3: Have sales Sales Learners Learner & 2-3 months;
personnel Checklist: Supervisor and
successfully sold Self- 6 months
[Product] Basics Assessment & after
solutions to Observation completion
commercial and of the
residential eLearning
stakeholders? module
Level 4: How Sales Data Learners Sales Manager Two
much did total Collection preceding
sales for the quarters pre-
[Product] product training; and
line increase quarterly for
because of the one year
training post-training
curriculum? completion

Data Analysis
As Dow (1992) suggests, the data collected by each evaluation instrument will need to be
interpreted in a way that allows for compilation, summarization and analysis of learner responses.
The open-ended questions as well as the one Yes/No question included in the Level 1 reaction
survey were summarized using a categorical frequency analysis. The main idea of each response
was identified and recorded. A continuum of 1 for strongly disagree (or the lowest identifier) to 5
for strongly agree (or the highest identifier) were coded for the responses to the Likert-scale
items. Frequency distributions were utilized to analyze the responses of all three types of
questions asked in the level 1 reaction survey.
In order to complete a data analysis for level 2 data was collected from the level 2
instrument [Product] Basics Final Assessment. The multiple choice, True/False and
simulation/interaction questions were be coded to identify the correct and incorrect responses
and/or steps for each assessment question. Each variables (assessment question) correct response
was coded with a value and the total assessment score was recorded in the companys LMS. This
data was retrieved via the learning and development department at the company and frequency
distributions were utilized to analyze the responses of each level 2 final assessment instrument.
Level 3 data was collected using the Sales Checklist: Self-Assessment & Observation
documentation. Each learners self-assessment observation response was coded with Yes = 1 and
No = 0 for each checklist criteria. Data was summarized to identify how many learners exhibited
the appropriate behaviors (Yes) versus those that did not (No). Analysis also consisted of a
frequency analysis to identify any potential areas of the course that could benefit from revisions
(Dow 1992).
Data at level 4 was summarized using the Sales Data Collection data sheet as seen in
Appendix D. Sales data for each product addressed in the [Product] Basics training module has
been be gathered for each learner two quarters preceding the completion of the module as well as
for four quarters post-completion of the training. A summary of percent change for each product
has been summarized for stakeholders in an executive summary as Dow (1992) suggests. While
there may be other factors that impact the changes in revenue, as Kirkpatrick and Kirkpatrick
(2006) emphasize, the preponderance of evidence (p. 68) is the ultimate goal to prove that
training was effective.

Findings
Level 1
Through the reaction survey administered to receive level 1 data, it was determined that
overall, the learners reacted favorably to the learning module with a few elements (highlighted
below) being less favorably received than others.
 Learners reacted favorably to the relevance of the training content, with N% of learners
rating the relevance as relevant or very relevant. Only N% of learners rated the relevance as
somewhat irrelevant or very irrelevant.
Learners responses to their confidence in being able to apply the content of the course
was distributed almost evenly among the five ratings scores, with N% feeling very unconfident,
N% feeling neutral and N% feeling very confident in being able to apply the content of the course
to their daily work.
Additionally, learners had a very high level of satisfaction with the course, with N%
rating their satisfaction as very satisfied. None of learners rated their satisfaction as very
dissatisfied.
Learners found the functionality of the course to be very intuitive, with N% rating the
intuitiveness as very intuitive. Only N% of learners found the course to be not intuitive at all or
somewhat intuitive.
Finally, N% of learners rated the course as very easy to navigate. N% of learners rated the
course as very difficult to navigate or somewhat difficult to navigate.
A majority of learners identified two elements of the course as most useful. The primary
aspect of the course, with N% of respondents including it in their responses, was that the
simulations were extremely beneficial in gaining quasi-hands on experience performing certain
actions. Additionally, N% of learners identified that the use of scenarios within the assessment
questions required them to think more thoroughly about the practical application of the objective
being assessed.

Level 2
Table 1 illustrates the pre- and post-assessment scores for learners related specifically to
each objective. Despite the pre-assessment being a pre-requisite for learners to take the [Product]
Basics course, the incompleteness of the pre-assessment created a gap in which we were not able
to fully measure the increase in knowledge across all learners.
For the learners that did complete both the pre-assessment and the most-assessment, there
was an increase in mastery of content. Learners increased their ability to explain the high-level
benefits of the [Product] by N%. They increased their ability to demonstrate correct [Product]
connections by N%. Learners increased their ability to repair a bent or kinked [Product] tubing by
N% and increased their ability to identify the different support systems for [Product]-a tubing by
N%. For the learners that only completed the post-assessment, an increase in knowledge could
not be identified, as there was no pre-assessment data with which to compare results.

Table 1
Pre- and Post-Assessment Scores of [Product] Basics eLearning Module

Objective Pre-Assessment Post-Assessment

Explain the high-level benefits of [Product] N% N%
Demonstrate correct [Product] connections N% N%
Repair bend or kinked [Product] tubing N% N%
Identify the different support systems for N% N%
[Product]-a tubing

Level 3
Table 2 illustrates the self-assessment and supervisor ratings for learners related
specifically to each task in the checklist. Learners consistently rated themselves as being able to
successfully perform (a Yes response) each behavior at 2-3 months post training; however, at
the 6 month checkpoint, the results became less consistent, specifically regarding the first
behavior I have successfully sold the [Product] solutions to commercial and residential
stakeholders. The remainder of the self-assessed behaviors remained consistently positive with
a slight decline at six months post-training.
Aligned with the learners self-assessment, supervisory observations at 2-3 months post
training as well as six months post training showed similar themes across each behavior.
Supervisors noted that most learners were able to continually demonstrate each of the latter five
behaviors; however, there was a slight decrease across the board at the six-month observation.
Supervisory observation of the first behavior was consistent with the learners self-assessment in
that the ability to successfully sell [Product] solutions to both commercial and residential
stakeholders slightly fell.

Table 2
Sales Checklist: Self-Assessment & Observation Summary
Self-Assessment
Behavior # of Learners # of Learners # of Learners # of Learners
to Identify to Identify to Identify to Identify
Yes No Yes No
(2-3 months) (2-3 months) (6 months) (6 months)
I have successfully sold N N N N
the [Product] solutions to
commercial and residential
stakeholders.
I have explained the high- N N N N
level benefits of [Product]
products.
I have identified correct N N N N
handling and storing
techniques for [Product]
tubing.
I have demonstrated N N N N
correct [Product]
connections.
I have repaired bent or N N N N
kinked [Product] tubing.
I have identified the N N N N
different support systems
for [Product]-a tubing.
Supervisory Observation
Behavior # of Learners # of Learners # of Learners # of Learners
Exhibiting Not Exhibiting Exhibiting Not Exhibiting
Behavior Behavior Behavior Behavior
(2-3 months) (2-3 months) (6 months) (6 months)
The learner successfully N N N N
sold the [Product]
solutions to commercial
and residential
stakeholders.
The learner explained the N N N N
high-level benefits of
[Product] products.
The learner identified N N N N
correct handling and
storing techniques for
[Product] tubing.
The learner demonstrated N N N N
correct [Product]
connections.
The learner repaired bent N N N N
or kinked [Product] tubing.
The learner identified the N N N N
different support systems
for [Product]-a tubing.

Level 4
Through the sales data collected for two quarters prior to learners taking the [Product]
Basics course as well as the four subsequent quarters post-training, it was determined with
substantial proof, the completion of the course significantly improved sales data across all
[Product] product lines. A slight dip in sales two quarters after the training aligned with level 3
findings that learners ability to make sales six months after training showed a slight decline. The
most noticeable change was the quarter immediately after the learners completed the training.

Recommendations and Conclusions

Overall, learners reacted very favorably towards the relevance of the content, functionality and
ease of use of the course. With only one area receiving neutral, evenly distributed results (i.e. the
learners confidence in applying the learned content). Learners were able to achieve the learning
objectives. Additionally, learners utilized their learning with ease when back on the job. In
conclusion, the [Product] Basics eLearning module was found to be effective in achieving its
goals. The findings of this evaluation strongly support the continuation of this program with the
proposal to include opportunities for short, refreshers periodically for learners to reinforce the
original learning.
 References

Dow (1992). Systematic Evaluation.

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: alternative
approaches and practical guidelines (4th ed.). Upper Saddle River, N.J.: Pearson Education.

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs. San Francisco:
Berrett-Koehler Publishers, Inc.

Shrock, S.A., & Coscarelli, W.C.C. (2008). Criterion-Referenced Test Development: Technical
and Legal Guidelines for Corporate Training. (3rd Ed.). San Francisco: Pfeiffer.
 Appendix A
Learner Satisfaction Survey
 Appendix B
[Product] Basics Final Assessment

Note: All company, product and image information is proprietary; therefore, all specifics
have been replaced with placeholders. Additionally, all interactions include images and are
not able to be included, they are; however, referenced in each assessment question.

[Product] Basics Final Assessment

To complete this module, you must obtain 100% accuracy. You need to answer each question
correctly before moving on to the next.

If you get an answer wrong, you will be prompted to return to the topic to research the correct
answer. When you are ready, you can return to the quiz questions to try again.

1. I do not have a specific location for all of the [Product] materials so they need to sit in the lot
beside the jobsite for a few months. The parking lot is an acceptable storage location for
[Product] materials.
a) True
b) False

2. Joe, the contractor on a hotel project shares that they are going to have a little greenhouse off
the hotel restaurant. The greenhouse irrigation is designed at 73.4F at 45 PSI. How do you
respond to Joe?
[Multiple choice]
a) There wont be any problems with these specs. Feel free to move ahead with the work.
b) There may be problems with these specs. Lets discuss your options further.

3. One of my interior decorators thought it would be fun to have some of the [Product] tubing
running inside some of the common areas. The decorator wants to know which kind of paint to
use. Pick the correct paint for the decorator.
[Multiple choice]
a) Acrylic paint
b) Rustoleum paint
c) Oil-based paint
d) Latex paint

4. What benefits are there with cold expansion fittings? Drag the appropriate items under
Benefits.
[Drag-and-drop]
Learner will drag and drop the images of the correct benefits to one side of the screen.

5. Joe, the contractor on the hotel project shares with you that the blueprints have some of the
tubing going underground to the hotel pool. He asks if he should worry about lead in the
[Product] products? How do you respond to Joe?
[Multiple choice]
a) All of [Companys] fittings are lead-free so there wont be a problem with direct burial.
b) There is no problem with the [Product] tubing. We will just need to make a modification
on the fittings.
c) Not a problem. There is a special lead-free jacket that we can install around the
[Product].
 d) Im sorry, Joe. [Product] products cannot be buried directly underground.

6. Confidence comes with practice. Show Joe that you can make a [Product] fitting connection in
less than 10 seconds. Drag the items to the correct place at the correct time. Click the expansion
tools trigger to turn it on.
[Interaction]
Learner drags the items to the correct place to design a fitting.

7. How far can you bend [Product] tubing before it kinks? Show Joe how far you can bend
[Product] without kinking the tube and then click Submit.
[Interaction]
Learner uses an interactive screen to bend the tubing to the correct degree.

8. Joe has a few apprentices that are still learning how to install tubing. He wants to know what
will happen if they kink the [Product] tubing at any point. Show him how easy it is to repair a
kinked [Product] tube.
[Interaction]
Learner utilizes on screen resources to repair a kink in the tubing.

9. Joe is hopeful that his contractors will insulate [Product] properly going to the greenhouse and
pool. But if they dont, what would they need to be aware of when fixing a frozen [Product] tube?
Select all that apply.
[Multiple choice]
a) A heat source being too hot
b) Not leaving the heat source on the affected area for enough time
c) Thawing too quickly
d) None of the above, you cannot thaw a frozen [Product] tube

10. Joe asks you if [Company] passed the [Industry] standard. What is he referring to?
[Multiple choice]
a) A fitting can be fit into a tube without heat.
b) The tubing does not allow potentially harmful substances to leach from the soil into
potable water.
c) All system components do not allow potentially harmful substances to leach from the soil
into the potable water.
d) The tubing can maintain its integrity at excessive water temperatures and pressures.
e) A plumbing system can maintain its integrity at excessive water temperatures and
pressures.

Answer Key
1. B 6. Proprietary product information
2. A 7. Proprietary product information
3. D 8. Proprietary product information
4. Proprietary product information 9. A & B
5. A 10. A
 Appendix C
Sales Checklist: Self-Assessment & Observation

Step 1: Self-Assessment
Performanc
e
Behavior
Yes No

I have successfully sold the [Product] solutions to commercial and residential

stakeholders.

I have explained the high-level benefits of [Product] products.

I have identified correct handling and storing techniques for [Product] tubing.

I have demonstrated correct [Product] connections.

I have repaired bent or kinked [Product] tubing.

I have identified the different support systems for [Product]-a tubing.

Step 2: Supervisory Observation

Performanc
e
Behavior
Yes No

The learner successfully sold the [Product] solutions to commercial and

residential stakeholders.

The learner explained the high-level benefits of [Product] products.

The learner identified correct handling and storing techniques for [Product]
tubing.

The learner demonstrated correct [Product] connections.

The learner repaired bent or kinked [Product] tubing.

The learner identified the different support systems for [Product]-a tubing.
 Appendix D
Sales Data Collection
Product Sales Sales Sales Sales Sales Sales Percent
Per Q1 Per Q2 Per Q3 Per Q4 Per Q1 Per Q2 Change
FY: FY: FY: FY: FY: FY: (Increase
YYYY YYYY YYYY YYYY YYYY YYYY or
Decrease)
[Product] $N $N $N $N $N $N +/- N%
X
[Product] $N $N $N $N $N $N +/- N%
X
[Product] $N $N $N $N $N $N +/- N%
X
[Product] $N $N $N $N $N $N +/- N%
X