Running head: EVALUATION PLAN

EDCI 577

Instructional Product Evaluation Plan

Level 1 Introduction to Food Safety and Hygiene Online Course

Danielle L. Moffat
Purdue University

EVALUATION PLAN

EDCI 577

Executive Summary
Purpose of the Evaluation
This evaluation plan has been devised for the purpose of gauging the effectiveness of the
recently developed Level 1 Introduction to Food Safety and Hygiene online course that is intended
to be fully implemented in due course as part of a new staff online induction program hosted in
BBs LMS in mid-2016. Given that it is a mandatory legal requirement for all staff to have
successfully passed the training course it is important that the course design and content achieve
the desired learning outcomes. Similarly, significant failures by staff to display the target food
safety and hygiene behaviors have the potential to result in serious legal and financial penalties
being incurred by BB. A trial and evaluation of the training course will provide for a detailed
assessment of the above factors in order that recommendations to be made for course
improvements can be made and actioned prior to its full implementation.
Evaluation Objectives
The primary objective of this evaluation is to measure the effectiveness of the training
course in terms of achieving the desired levels of participant learning and target food safety and
hygiene related behaviors in the workplace. The evaluation also aims to quantify the positive
business impacts relating to implementation of the training course across BBs 4 venues.
Evaluation Plan Summary
The evaluation plan set out in this document utilizes Kirkpatricks (2006) four-levels of
evaluation methodology. Participant reactions, learning gains and changes in behavior towards the
desired states shall be measured with reference to a group of approximately 30 participants
comprised of anticipated new hires over the next 3 months. Business results will be assessed
following full implementation of in mid-2016. Data capture, analysis and reporting procedures in
respect of each of Kirkpatricks (2006) four levels of evaluation have been set out in detail within
the main body of the accompanying Evaluation the online induction program which shall be
administered to all staff across BBs organization Plan Report.
Evaluation Results and Key Recommendations
Due to the fact that the course trial has not yet taken place, evaluation data is not yet
available. However, the board of directors will be provided with a Level 1 & 2 Evaluation Report
following the course trial which will provide a detailed analysis of course participant reactions and
learning gains as well as set out full recommendations for course improvements, where necessary,
to be actioned prior to full implementation of the online staff induction program. Approximately
6 months thereafter a further Level 3 Evaluation Report will be provided which will detail changes
in behavior achieved within the participant group as a result of attendance on the course.
Recommendations have been made within this Evaluation Plan Report in relation to steps to be
taken in advanced of the course trial order to create a workplace environment that promotes display
of the target behaviors. A final Level 4 Evaluation Report will be provided approximately 9 months
following full implementation of the online induction program due to the fact that the business
2

EVALUATION PLAN

EDCI 577

impacts of the course are not capable of being assessed on the basis of the small group of course
trial participants. The final Level 4 Evaluation Report will analyse existing documentation relating
to food safety and hygiene, including accident report forms, from across BBs 4 venues in order
to gather evidence of positive business impacts related to full implementation of the course.
The board of directors can expect to review various types of evaluation data within the
various Reports referred to above including: learner reaction survey results, data relating to the
achievement of learning outcomes including knowledge, skills and hazard perception test scores,
course participant behavioral interview transcripts and pre post course implementation safety
record comparisons.

EVALUATION PLAN

EDCI 577

Evaluation Plan Report:

Level 1 Introduction to Food Safety and Hygiene Online Course
Instructional Product Overview
Brightwing Breweries (BB) is a small but rapidly expanding group of gastro-inns, at this
time comprising 4 venues located across the Southwest of England. In light of the groups plans
to double its number of venues within the next 18 months, a board-level decision was made early
this year to implement a formal, group-wide staff professional development program in order to
promote uniformity of training and standards across the organization as well as to ensure full
compliance with relevant regulations and legislation. The new professional development program
forms one of a number of digitization initiatives currently being rolled out across the group
involving utilization of cloud-based technologies with the aim of improving and streamlining BBs
business operations.
BB have engaged the services of DM Consult Ltd (DMC), a specialist e-learning
solutions company, to design and implement a bespoke LMS to serve as the hosting platform for
the online aspects of the professional development program including e-learning courses,
personalized professional development plans and progress tracking, resources and knowledge
sharing functions for supervisors and management including online discussion forums.
In initial consultations, BB identified staff induction training which is presently provided
on an informal, face-to-face basis at each venue as part of a one day induction session as a critical
concern. A recent in-house review of the current induction training has revealed that it fails to fully
comply with current European and UK Health & Safety regulations and legislation, a situation
which has the potential to result in serious legal penalties being incurred by BB. Accordingly, it
was agreed that DMC would prioritize the development of a new staff induction training program,
comprising a number of standalone e-learning courses covering a variety health and safety
induction topics, to form part of the LMS implementation pilot.
This report sets out the proposed evaluation plan in relation to one of the health and safety
e-learning induction courses, Level 1 Introduction to Food Safety and Hygiene, which is intended
to be trialed during the upcoming 3-month LMS implementation pilot. It is intended that, in due
course, all new staff who have not undergone equivalent training within the last 5 years will be
compelled to complete the Level 1 Introduction to Food Safety and Hygiene regardless of
experience level, in accordance with current legislative requirements. A summary of the courses
structure and instructional content is outlined in Table A1 at Appendix A.
Purpose of the Evaluation
Given that all new relevant staff will be required to undertake the Level 1 Introduction to
Food Safety and Hygiene course, the purpose of this evaluation is to determine the overall
effectiveness of the course and to identify the ways in which it might be improved before full
implementation of the online staff induction training program via the LMS takes place in mid2016.

EVALUATION PLAN

EDCI 577

Evaluation Objectives
The following evaluation objectives form the basis for the evaluation data capture
procedure by setting out the specific factors that must be addressed during the course of the
evaluation in order to ascertain the effectiveness of the course and make recommendations for any
improvements that might be made thereto, if appropriate.
DMC recommends undertaking the course evaluation by means of Kirkpatricks (2006) 4
Levels of Evaluation model, which stresses evaluation of the chronically staged levels of (1)
reaction, (2) learning, (3) behavior, and (4) results. Accordingly, the following table sets out the
evaluation objectives in respect of each level together with an outline of the evaluation instruments
and resources that shall be utilized to measure for each objective.
Table 1
Evaluation Objectives, Questions and Criteria/Measures/Resources
Level Objectives
Criteria/Measures/Resources
1
To assess participant attitudes towards the Course reaction survey
course.
2
To quantify learning gains experienced by Pre-post
course
comparison
of
participants as a result of undertaking the knowledge and skills test results.
course.
Assessment of achievement of required
pass mark in posttests.
3

To assess the level to which the course

promotes the display of desired food safety
and hygiene-related behaviors in program
participants.
To calculate and/or describe the monetary
and/or intangible business impacts of the
course.

Participant interviews at 3 and 6 months

post-course.

Accident report forms/HSE reports.

Complaints records.
HSE food safety & hygiene inspection
records

Audience and Context

Key stakeholders. Fitzpatrick, Sanders & Worthen define stakeholders as, the individuals
and groups who have a direct interest in and may be affected by the program being evaluated or
the evaluations results (2004, p. 9). Accordingly, the following groups comprise the key
stakeholders in respect of the Level 1 Introduction to Food Safety and Hygiene course:
Board of Directors: the board of directors is comprised of 5 members who are joint equity
partners in the business. They are the sponsors of the professional development program. As such,

EVALUATION PLAN

EDCI 577

they will review the results of the evaluation and have authority for making key decisions regarding
the courses future.
HR Departments, Managers and Supervisors: this group comprises those persons within
the organization who have direct responsibility for the course (Fitzpatrick et al., 2004). The HR
Departments will be involved in the administration and co-ordination of the course as well as
assisting the evaluators with some elements of the data capture and collation process. Supervisors
will be involved in administering the pre- and post-course skills tests, and both Managers and
Supervisors will be instrumental in providing feedback regarding the effectiveness of the course,
in terms of transfer of the target knowledge and skills in to workplace practice.
Participants: this group comprises the intended beneficiaries of the course (Fitzpatrick et
al., 2004). The participants, being the target learners, comprise all new staff joining BB including
entry-level to experienced floor, bar and kitchen staff, supervisors and managers who are selected
for participation on the course for the purposes of this evaluation.
The target learner group demographics are presumed to fall in line with the most recently
published UK hospitality and tourism labor market statistics: 57% of the workforce is female; 44%
are under the age of 30 (with 31% of the same being aged between 16-24) with the average age of
restaurant managers and bar managers being 39 and 44 years respectively; 85% of the workforce
describes their ethnicity as white and 15% as being of black and minority ethnic background
(People 1st, 2013). The majority of the target learner group are likely to be relatively young.
Participants below supervisory level will tend to be under 30 years of age, with the majority of that
group falling within the age range of 16 26. Of that majority group, those between the ages of
16 19 will not be advanced learners, having been educated to high school level only.
Context. Due to the fact that the e-learning course is hosted in BBs cloud-based LMS it
is accessible from any computer via each learners individualized professional development
profile. Learners will undertake the course on dedicated training computers provided in the
administrative offices of each of BBs venues and they will be provided with a pencil and paper
before taking the course for note taking. New staff will spend their first two days at work in
induction, with the first of those days being devoted to undertaking the online induction courses.
Prior to taking the online courses, the learners will receive informal face-to-face training
with their induction training supervisor on how to log in to and use the LMS as well as undertaking
a short training module on how to navigate and use the e-learning courses and interactive activity
types. Therefore prior knowledge in that regard is presumed.
The Evaluation Process
The Level 1 Introduction to Food Safety and Hygiene online course will be trialed during
the upcoming 3-month LMS implementation pilot. BB have advised that, due to the nature of the
hospitality industry, there is a relatively high level of staff turnover across the group. It is projected
that somewhere in the region of 30 new hires will be made during the pilot period, which will
make up the target learner group for the purposes of the evaluation of Levels 1 - 3.

EVALUATION PLAN

EDCI 577

It has been agreed that a Level 1 and 2 Evaluation Report will be produced following the
initial trial of the online course, which will provide an early assessment of the effectiveness of the
course in attaining the required level of learning and allow for recommendations to be made and
actioned in respect of course improvements and refinement prior to full implementation of the
online induction program in mid-2016. A longer-term view will be taken in respect of evaluating
for Levels 3 and 4 as sufficient time must be allowed for changes in behavior to take place as a
result of acquiring the target knowledge and, subsequently, for the overall organizational impact
of such changes to become apparent (Kirkpatrick, 2006). The Evaluation Sampling Plan at
Appendix B and Gantt chart at Appendix C together set out the recommended project plan and
timelines. Evaluation instruments, data capture and reporting procedures in respect of each level
are explained in detail in the following sections of this Report.
Level 1: Reaction
Evaluating participant reactions will provide information that will enable an assessment of
the level of the course participants motivation to learn. It is recommended that participant
reactions are obtained by means of an electronic post-course reaction survey (Appendix D)
forming part of the online course, to be administered at the culmination of the learning and
electronic test elements of the same in accordance with the data capture procedure for Levels 1
and 2 set out at Table 2 below.
Motivation has a direct and significant impact on learning low levels of motivation will
inhibit learning while highly motivated learners will be involved in the initiation of, and
persistence in, learning activities. Accordingly, Level 1 evaluation results provide insight in to and
clarification of Level 2 results (Kirkpatrick, 2006). Where Level 1 data reveals a significant
negative participant reaction to the course then it can be inferred that any corresponding low levels
of learning highlighted in the Level 2 data are contributed to by factors affecting the participants
motivation to learn. In such event, participants written comments and suggestions gathered from
the open comment sections of the reaction survey will serve to pinpoint specific issues affecting
the participants learning motivation levels, such as for example not being appropriately
challenged by the course content, which can be used to make improvements and refinements to
the course before full implementation. Recommendations for capture of further clarification data
may be recommended within the Level 1 and 2 Evaluation Report in the event that a significant
negative participant reaction is found.
The raw data in relation to the reaction survey will be stored in the LMS which has been
designed to provide reporting functions enabling specific data to be extracted and presented in
graphical or numerical table format. Given that the course trial has not yet been undertaken, actual
Level 1 results are not presently available. In accordance with Kirkpatricks (2006)
recommendations, acceptable standards of success have been pre-agreed. Given that passing the
Level 1 Introduction to Food Safety and Hygiene is a mandatory legal requirement, and the pass
grade has been set at 80%, it has been agreed that a similarly high standard will be applied to the
acceptable standards in relation to the Level 1 results. A minimum mean response per survey
question of 4/5 will be considered the acceptable standard. Appendix E sets out the form in which
the results shall be reported to the board of directors within the Final Evaluation Report, which

EVALUATION PLAN

EDCI 577

shall be accompanied by a written Level 1 analysis summary and DMCs recommendations for
further action, if appropriate.
Level 2: Learning
BB have requested that all new hires undertake the trial course during the pilot period due
to the fact that (1) the food safety and hygiene training is a mandatory legal requirement, (2) the
current training regime in that regard is deficient, and (3) possessing the target knowledge and
skills is a pre-requisite to successful carrying out of workplace duties. It is therefore recommended,
for practical purposes, that the evaluation employs a one group pre-assessment post assessment
design, whereby the entire target learner group is assessed prior to and following the instruction to
determine the learning gains achieved as a direct result of undertaking the course (Dow Chemical
Company, 1992).
Table 2, below, sets out the data capture procedure in relation to the evaluation of Level 1
and 2. The Level 2 learning instruments referred to within Table 2, namely the pre- and post-skills,
transfer and knowledge tests, are set out in detail at Appendix F, G and H respectively.
It should be noted that the pre- and post-test multiple choice test comprises the same 20
questions in order that the result can be readily compared. No feedback will be provided to the
target learners responses in the pre-test and questions will be randomized in both tests so as to
avoid skewed results due to participants simply learning the correct responses to the test questions
rather than acquiring the target knowledge.
Table 2
Data Capture Procedure for Levels 1 and 2
Evaluation Instruments
Timeline
Skills pre-test (handwashing Administered immediately prior
demonstration) Level 2.
to undertaking online course to
Measuring prior skill level. target learners.

Knowledge pre-test
(electronic 10 question
multiple choice test) Level
2. Measuring prior
knowledge. Appendix H,
Table H1.
Knowledge pre-test
(identification of hazards, 8
possible hazards) Level 2.
Measuring prior knowledge.
Appendix G.
Hazard perception post-test
(identification of hazards, 8

Online test administered

immediately prior to commencing
the course.

Administration
Administered by induction
training supervisor. Grade
assigned using handwashing
rubric (Appendix F). Raw
data entered in skills pre-test
spread sheet.
Raw quiz scores stored in
LMS.

Module 1 hazard awareness

activity involving clicking on
potential food safety hazards
within an image.

Raw scores stored in LMS.

Hazard awareness in Module 1

administered in final Module 8.

Raw scores stored in LMS.

EVALUATION PLAN
possible hazards) Level 2.
Measuring transfer.
Appendix G.
Knowledge post-test
(electronic 10 question
multiple choice test) Level
2. Measuring post-course
knowledge. Appendix H,
Table H1.
Post-course reaction survey
Level 1. Measuring
attitudes to course and
overall course satisfaction.
Appendix D.

Skills post-test
(demonstration
handwashing) Level 2.
Measuring post-course skill
levels.

EDCI 577

Administered immediately at the

end of the course.

Raw scores stored in LMS.

Administered immediately
following the knowledge post-test
as a mandatory requirement for
course completion to ensure
100% response. Participants
provided the opportunity to
provide written feedback and
comments in text boxes.
Administered on the second
induction day after participant has
had the opportunity to practice
handwashing techniques at home
using printable step-by-step guide
provided in Module 3.

Raw data stored in LMS.

Administered by training
supervisor. Grade assigned
using handwashing rubric
(Appendix F). Raw data
entered in skills post-test
spread sheet.

In accordance with Kirkpatricks (2006) recommendations, the level of learning that has taken
place on the course will be analyzed as follows:

A comparison of mean pre post course test scores in the multiple choice and food hazard
perception (knowledge) and handwashing (skills) tests. No or low gains between the scores
achieved in the pre- and post-tests will indicate that the training has been ineffective while
substantial gains between pre- and post-tests will signify course efficiency.

A comparison of mean pre post test scores for individual questions in the multiple choice
knowledge test will provide specific information in relation to learning in the different
modules that can be used highlight areas where learning has been less effective, signaling
a need for improvement.

As with Level 1 results, the raw data in relation to the electronic pre- and post-course
knowledge and hazard perception (transfer) tests will be stored in the LMS and extracted for
analysis at the end of the trial period. The raw data in relation to the pre- and post-course skills test
will be manually input in to a designated spreadsheet at each venue by the training coordinators,
which will then be forwarded to DMC by the HR Departments for consolidation and analysis at
the end of the trial period. Acceptable standards in respect of Level 2 results have been pre-agreed

EVALUATION PLAN

EDCI 577

at a minimum result of 80% for all participants in all tests, in line with the pass mark they are
required to achieve in order to successfully complete the mandatory course.
The Level 2 results will be reported to the board of directors in the Level 1 and 2 Evaluation
Report in the form of pre- and post-course test comparison tables and graph, as set out at Appendix
I, together with an accompanying analysis summary detailing the evaluation findings (namely, the
extent of the learning gains in each area and a conclusion as to course efficacy in light of the same).
In relation to the comparison of average scores for individual questions in the pre- and post-course
multiple choice knowledge tests, the analysis summary will, if appropriate, highlight areas for
improvement and/or refinement, making specific recommendations in that regard to be actioned
prior to full implementation of the online induction course if appropriate.
Level 3: Behavior
Evaluating changes in participant food safety and hygiene behavior as a result of attending
the course will provide information that will enable an assessment of the level of transfer of the
target skills and knowledge in to the workplace, namely the level to which the participants go on
to display the desired food safety and hygiene behaviors they learned about on the course during
the course of their every day work (Kirkpatrick, 2006). Given the fact that BB stand to incur
potentially serious legal or financial penalties in the event that its members of staff fail to
adequately display the desired food safety and hygiene behaviors, DMC recommends taking the
following steps prior to the trial course in order to create a work environment that promotes transfer
of the target skills and knowledge:

DMC shall meet with supervisors and managers shortly before the commencement of the
course trial to explain its purpose and importance, present an abridged version of the online
courses content and set clear expectations in terms of their role in supporting and encouraging
display of the target behaviors.

DMC shall meet with the Head of HR in order to devise and appropriate reward system in
respect of the display of desired health and safety behaviors generally. This forms part of the
wider professional development program development remit and, as yet, remains to be fully
laid out. Therefore, this point is not dealt with in detail here except to draw attention to the
principle that incentivizing staff through a rewards system will encourage display of the desired
behaviors within the workplace (Kirkpatrick, 2006).

BB advises that the majority of the participants will be hired on the full-time basis in which
they will be expected work at least the statutory minimum of 35 hours per week. Accordingly, it
is assumed that the participants will have the opportunity to display the majority of the desired
behaviors within a relatively short period of time. Given that the majority of the new hires will
comprise entry-level candidates it is considered inappropriate in these circumstances, from a time
and costs point of view, to evaluate for pre-existing target behaviors prior to administration of the
trial course. It is recommended that participants are evaluated for changes in behavior at 3 and 6
months following attendance on the course by means of a patterned interview, the form of which
is set out at Appendix J. It is felt that there will be sufficient time for the training supervisors to
10

EVALUATION PLAN

EDCI 577

conduct the interviews with all of the course participants across BBs 4 venues, which will provide
a full view of the changes in behavior. The rationale for evaluating for Level 3 by means of the
more time-consuming method of patterned interviews, as opposed to using surveys, is that the
results obtained will provide greater detail regarding the behavioral changes as well as provide the
interviewing training supervisors with the opportunity to discuss relevant issues arising as well as
to identify areas in which additional ongoing support in relation to display of the target behaviors
may be provided, either on an individual or organizational basis.
The completed interview forms will be submitted via email to DMC by the respective
training supervisors in order that the Level 3 results can be tabulated. Quantitative data in relation
to the degree to which behavioral changes have occurred as a result of course attendance will be
set out for review by the board of directors within the Level 1 to 3 Evaluation Report in the tabular
and graphical forms set out at Appendix K. Specifically, the degree of behavior change will be
analyzed as follows:

An assessment of increases or decreases in the participants ability to identify the desired

food safety and hygiene behaviors encouraged by the course at 3 and 6 months following
the course. Any increase in that regard between the first and the second interview will be
assumed to be due an increased level of engagement in display of those target behaviors
leading to an enhanced awareness of the same.

A comparison of participants responses in relation to questions aimed at gathering data

relating to their eagerness to display the target behaviors at 3 and 6 months. Participants
will be provided with the opportunity to provide open comment on the reasoning behind
their choice of the response options. Such comments will be useful in assessing the
underlying reasons for any negative trends found in the results. Relevant participant
comments in that regard will be set out within the Level 1 to 3 Evaluation Report together
with DMCs recommendations for further action, if necessary.

An analysis of the participants perceptions relating to their ability to display the desired
behaviors, and a comparison of such responses at 3 months and 6 months. Where
participants have been unable to engage in the desired behavior they will be given the
opportunity to identify, out of a list of response options, the reason for this inability. Such
information shall be compared between 3 months and 6 months in visual graphical format,
and shall provide important information that can be used to identify and, where appropriate,
eliminate environmental and other factors that may be inhibit display of the desired
behaviors. Participants will be provided with the opportunity to provide further comments
where they have been unable to display the target behaviors for reasons other than the
available response options. Such comments will be reviewed by DMC in detail and may
result in recommendations in the Level 1 to 3 Evaluation Report for further investigations
or capture of additional clarifying data where one or more significant factors affecting
performance not previously considered are identified within the Level 3 results.

11

EVALUATION PLAN

EDCI 577

Level 4: Overall Results/Business Impact

Given the relatively small size of the target learner group in contrast to BBs employee
base as a whole, it has been agreed that it will be inappropriate to measure for Level 4 results in
relation to the course trial. Full implementation of the online induction program is due to take place
in mid-2016, shortly following which the majority of the existing staff are intended to undertake
the new online induction program. Accordingly, it has been agreed that Level 4 results will be
evaluated 1 year following full implementation of the program, essentially allowing a period of 9
months for changes in behavior to take place and any subsequent Level 4 results to become selfevident.
Level 4 results are notoriously difficult to measure for and it should be noted that findings
in this regard will generally only provide evidence that implementation of the training course has
resulted in the desired business results being achieved rather than representing proof beyond
reasonable doubt of the same (Kirkpatrick, 2006). Therefore any intervening factors that may arise
in the interim, such as substantive changes to workplace practices, policies or procedures, will
have to be identified and considered in the analysis of the Level 4 results.
It is recommended that Level 4 results are measured as follows:

Comparison a pre post course implementation accident report forms. A significant

reduction of accident reports relating to food safety and hygiene practices following
implementation of the training course will provide evidence that food safety and hygiene
practices have improved.

Comparison of pre post course implementation food safety and hygiene related customer
complaint records. Similarly, a significant reduction in relevant customer complaint
records, including those relating to objects found in food, spoiled food and poor employee
hygiene, following implementation of the training course will provide evidence that food
safety and hygiene practices have improved as a result of the same.

Comparison of pre post course implementation Health & Safety Executive food safety &
hygiene inspection ratings. Enhanced food safety and hygiene inspections ratings
following full implementation of the course will provide evidence of improved food safety
and hygiene practices.

Comparison of pre post course implementation food spoilage and wastage financial
records. Significant reductions in costs related to food spoilage and wastage following full
implementation of the course may be attributable to increases in food safety and hygiene
practices if other intervening factors are not identified such as physical improvements to
kitchen storage facilities.

BB has advised that the abovementioned documentation required for identification and
analysis of the Level 4 of business impacts resulting from implementation of the Level 1
Introduction to Food Safety and Hygiene online course are created and retained at each venue as
part of the ordinary course of business and therefore will be readily available for review. HR
12

EVALUATION PLAN

EDCI 577

Departments at each venue will be responsible for gathering together the relevant documentation
and forwarding the same by email to DMC for compiling, coding, analysis and interpretation.
DMC will conduct telephone interviews with each venue manager in order to ascertain any relevant
intervening factors before providing a final Level 4 Evaluation Report to the board of directors
summarizing the business impact results in relation to each venue as well as the organization as a
whole.

13

EVALUATION PLAN

EDCI 577
References

Dow's Systematic Evaluation Model - (Dow Chemical Company. (1992). Evaluation of training:
The Dow process. [Manual]. Midland, Michigan: Author.)
Fitzpatrick, J. L., Sanders, J. R. & Worthen, B. R. (2004). Program evaluation: alternative
approaches and practical guidelines (3rd ed.). Boston: Pearson/Allyn and Bacon.
Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels (3rd ed.).
San Francisco, CA: Berret-Koekler.
People 1st (2013). State of the Nation Report 2013: An analysis of labour market trends, skills,
education and training within the UK hospitality and tourism industries. Retrieved from
http://www.people1st.co.uk/getattachment/Research-policy/Research-reports/State-ofthe-Nation-Hospitality-Tourism/SOTN_2013_final.pdf.aspx

14

EVALUATION PLAN

EDCI 577
Appendix A

Level 1 Introduction of Food Safety and Hygiene Course Outline

Table A1
Summary of Course Structure and Instructional Content
Module
Instructional Content
1 Importance of Food Safety
Introduction narration learning objectives
Food safety info drop-down list: preventing food
poisoning, quality control, legal responsibilities
Activity: click areas in image to identify potential hazards
(no feedback)
2 Contamination
Contamination narration
Food contamination sources info drop-down list: microorganisms, chemical, physical
Activity: drag and drop examples of chemical, physical and
viral contaminants in to correct category (immediate
feedback)
Contamination prevention scroll-over list: physical, viral,
chemical
Cross-contamination narration
Cross-contamination info drop-down list: food handlers,
utensils, equipment and surfaces, raw food, rubbish, insects
and rodents
3 Handwashing
Handwashing narration
Handwashing demonstration instructional video
Printable step-by-step handwashing guide image,
instructions to practice handwashing technique
Follow-up activity: demonstrate proper handwashing
technique (administered by supervisor on-the-job)
4 Pests
Pest narration
Commercial kitchen pests info drop-down list: rats and
mice, flies, cockroaches
Activity: drag and drop evidence of pest infestation in to
correct pest category (immediate feedback)
Evidence of pest infestation scroll-over: additional images
5 Cleaning
Cleaning narration
Effective surfaces and equipment cleaning techniques
demonstration instructional video
Activity: effective cleaning techniques quizlet (immediate
feedback)
6 Personal Hygiene
Personal hygiene narration
Personal hygiene best practices info drop-down list: keep
yourself clean, prepare for work, unsanitary habits

15

EVALUATION PLAN

7 Cooking and Chilling

8 Conclusion

EDCI 577
Activity: drag and drop personal hygiene poster completion
(immediate feedback)
Cooking and chilling narration
Critical temperatures info timeline list: cooking, hot
holding, reheating, cooling
Activity: critical control points matching exercise in form
of timed memory game (immediate feedback)
Recap narration learning objectives
Activity: click areas in image to identify potential hazards
(same activity as in module 1, no feedback)

16

EVALUATION PLAN

EDCI 577

Appendix B
Evaluation Sampling Plan
Table B1
Evaluation Sampling Plan for Levels 1 4
Task

Start
Date

# Days Required

Percent
Complete

Location

Assignment

HQ.
HQ.
HQ.

DMC
DMC
DMC

HQ.

DMC

BB
Head
Office

DMC

Venues

DMC/HR/
Supervisors

HQ./Venues

DMC/
Supervisors

HQ.

DMC/BoD

HQ.

DMC/BoD

Venues

Supervisors

Venues

Supervisors

HQ.

DMC

HQ/
BB
Head Office

DMC/BoD

Venues

HR/
Supervisors

Venues

HR/
Managers

Evaluation Planning
Draft evaluation plan
Review/edit evaluation plan
Develop instruments
Finalize/submit evaluation
plan
Preliminary Matters
Pre-trial training session
with
managers
and
supervisors
Course Trial

9/5/15
9/12/15
9/26/15

7
14
15

100
100
100

9/26/15

15

100

10/16/15

90
Collect Level 1 & 2 data
10/19/15
Preparing and Reporting Level 1 & 2 Data

Collate Level 1 & 2 data

1/19/16
Analyze
data,
prepare/submit Level 1 & 2
Report
1/26/16
14
Actioning Level 1 & 2 Recommendations
Make course amendments
as
per
agreed
recommendations
2/9/16
30
Level 3 Evaluation
3-month
participant
interviews
1/19/16
90
6-month
participant
interviews
4/19/16
90
7
Collate Level 3 data
7/19/16
Analyze
data,
prepare/submit Level 1 - 3
Report
8/2/16
14
Full Course Implementation

0
0
0

Existing staff attend online

induction program
Level 4 Evaluation

9/3/16

90

Collect pre-course L4 data

9/3/16

30

17

EVALUATION PLAN

EDCI 577

Venues

HR/
Managers

Venues

HR/DMC

Venues

HR/DMC

HQ/
BB
Head Office

DMC/BoD

Gather post-course L4 data

9/3/16

365

Collate and forward precourse L4 data

10/3/16

30

Collate and forward postcourse L4 data

9/3/17

14

Analyze
data,
prepare/submit L4 Report

9/18/17

14

18

Running head: EVALUATION PLAN

EDCI 577

Appendix C
Gantt Chart

Figure 1C. Gantt chart setting out the projected project timelines in respect of the evaluation plan.

19

Running head: EVALUATION PLAN

EDCI 577

Appendix D
Post-course Reaction Survey
Reaction response scale:
Strongly Disagree
1
2

Strongly Agree
3

Brightwing Breweries is committed to providing you with the very best training. Please take some
time to complete survey with your honest feedback about your experiences on this course which
will help us to think about how we might improve it for the future. Thank you!
Course Content
1. I feel that this course is relevant to my job position at Brightwing Breweries.
2. I believe that the topics covered will help me to perform my job well.
3. The course activities helped to reinforce my understanding of the topics covered.
4. I felt engaged throughout the training.
5. Please provide us with any comments or suggestions you may have regarding the
instructional content in the text box below.
Course Format and Materials
6. I was able to navigate through the course content without any problems.
7. The images and other visual elements helped to reinforce my understanding of the topics
covered.
8. I felt that the modules were presented in a logical, clearly organized format.
9. Please provide us with any comments or suggestions you may have regarding the course
format and/or materials in the text box below.
Overall satisfaction
10. Overall, I was satisfied with this course.
11. Please provide us with any other comments or suggestions you may have for improving
the course and/or your learning experience within the text box below.
20

EVALUATION PLAN

EDCI 577

Appendix E
Level 1: Results Report, Analysis and Recommendations Format
Table E1
Results Report, Analysis and Recommendations Format
Question
Mean Score / 5
Relevant Participant Comments and Suggestions
Course Content
1
2
3
4
Course Content Section Mean Score / 5:
Analysis and Recommendations:

Course Format and Materials

6
7
8
Course Format and Materials Section Mean Score / 5:
Analysis and Recommendations:

Overall Satisfaction
10
Overall Satisfaction Section Mean Score / 5:
Analysis and Recommendations:

21

EVALUATION PLAN

EDCI 577

Appendix F
Handwashing Rubric
See step-by-step Poor: 1 point
guide below

Fair: 2 points

Good: 3 points

Excellent:
points

Followed Steps

Failed
to
demonstrate any
of the proper
handwashing
steps

Missed three or
more of the
proper
handwashing
steps

Missed one or
two of the proper
handwashing
steps

Correctly
demonstrated all
of the proper
handwashing
steps

Order

Failed
to
demonstrate the
proper
handwashing
steps in order

Demonstrated up
to six of the
proper
handwashing
steps in order

Demonstrated up
to nine of the
proper
handwashing
steps in order

Correctly
demonstrated all
of the proper
handwashing
steps in order

Time

Washed hands Washed hands Washed hands Washed hands

for less than 10 for up to 20 for up to 30 for 40 seconds or
seconds
seconds
seconds
more

Total Score / 15

22

EVALUATION PLAN

EDCI 577

Appendix G
Hazard Perception (Knowledge Pre-test/Transfer Post-test) Screenshot

23

EVALUATION PLAN

EDCI 577

Appendix H
Knowledge Multiple Choice Pre-test and Post-test Questions
Table H1
Pre- and Posttest Multiple Choice Questions and Answers
Question
Answer options
1. Foods unlikely to support A: Raw meat and poultry
bacterial growth are:
B: Seafood and shellfish
C: Dried pasta and noodles
D: Milk and cream
2. Rinsing your hands under True/False
hot water will remove
bacteria.
3. What is the main reason for A: Pest-free kitchens are more
pest control in commercial efficient.
kitchens?
B: Customers will get upset if
they see a pest.
C: Staff might trip over a pest
and hurt themselves.
D: Pests carry bacteria that
can contaminate food
4. Poor food hygiene might True/False
cause a few upset stomachs
but it wont cause serious
illness.
5. How can you tell that a food A: You cant, it will look and
has been infected with smell normal.
harmful bacteria?
B: It will smell bad.
C: It will taste off.
D: It will change color.
6. Micro-organisms are?
A: Only found in the sea.
B: Bacteria, mold, viruses and
other organisms too small to
see with the naked eye.
C: Are really small but you
can see them if you squint.
D: Bacteria.
7. Practicing good personal True/False
hygiene is only important
when you work in areas where
customers can see you.
8. What jewelry is acceptable A: Nose ring
for a member of kitchen staff B: Bracelet
to wear during service?
C: None
D: Wedding band
24

Answer
C

False

False

False

EVALUATION PLAN

EDCI 577

9. Cooked food should be True/False

cooled overnight at room
temperature before being
refrigerated.
10. Which of the following is A: All bacteria gives people
true of bacteria?
food poisoning
B: Bacteria and viruses are the
same thing.
C: Bacteria multiplies and
grows faster in warm, moist
environments.
D: Bacteria thrives in
temperatures above 60C
11. Food is safe when it is free True/False
from visible contamination.
12. Why must food handlers A: So they can take a break.
wash their hands thoroughly B:
To
prevent
crossafter handling raw food?
contamination.
C: To stop them from getting
dirt on their clothes.
D: To prevent food wastage.
13. Which of the following is A: Mold growing on a piece of
an example of a physical bread.
contaminant?
B: Pesticide residue on a piece
of fruit.
C: A food handler sneezing
over raw chicken they are
preparing.
D: A piece of jewelry found in
some food.
14. Always use a disposable or True/False
clean cloth to wipe down
disinfected surfaces.
15. Cross contamination of A:Washing hands before
food will occur when:
preparing food
B: Cleaning down work
surfaces and equipment
C: Food is frozen
D: Cockroaches walk across
food
16. A clean, healthy person True/False
can carry food poisoning
causing bacteria without being
ill themselves.

25

False

False
B

True

True

EVALUATION PLAN

EDCI 577

17. The temperature danger True/False

zone is between 40C and
60C.
18. How often should kitchen A: Frequently and before the
rubbish bins be emptied?
rubbish bin is completely
filled.
B: At the end of each service.
C: Twice a day.
D: When the rubbish bin is
full.
19. What should you if you are A: Get a good nights sleep.
feeling sick?
B: Take some medicine go to
work as usual.
C: Go to your doctor for a
checkup immediately and
report to your supervisor.
D: Let nature take its course
youll feel better in a couple of
days.
20. Food poisoning bacteria True/False
found in raw foods can
survive cooking if the food is
stored in the temperature
danger zone.

26

False

True

EVALUATION PLAN

EDCI 577

Appendix I
Level 2 Results Reporting Format
Table I1
Comparison of Mean Pre Post Test Scores
Mean Pretest Score
Knowledge Multiple /10
Choice Test
Hazard
Perception /8
Test
Handwashing Skills /15
Test

Mean Posttest Score

Learning Gain
(%)

/10
/8
/15

Table I2
Comparison of Pre Post Knowledge Multi-choice Test Scores Per Question
Question Times Correctly Answered Times
Correctly Learning Gain (%)
in Pretest / 30 Participants Answered in Posttest / 30
Participants
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

27

EVALUATION PLAN

EDCI 577

Posttest Pass Rates

30
25
20
15
10
5
0
Knowledge Multi-choice

Handwashing Skills

Achieved 80% Pass Mark

Hazard Perception

Did Not Achieve 80% Pass Mark

Figure I1. Column graph that shall be utilized to report the number of participants successfully
attaining the required pass mark in respect of each posttest versus the number of participants failing
to achieve the required pass mark.

28

EVALUATION PLAN

EDCI 577

Appendix J
Patterned Interview Form
Instructions for Interviewer: Clarify the purpose of the interview, which is to evaluate the
effectiveness of the trial online course in terms of how far the participant has applied the
encouraged behaviors to their job, and explain to the interviewee that all information they give
during the interview will remain anonymous so they are free to give frank and honest answers to
all of the questions posed. Please ensure that all questions are answered.
..
1. Q1 Instructions for Interviewer: Without showing the interview form to the interviewee use
the table below to elicit and review the food safety and hygiene behaviors that the trial course
encouraged. Discuss each encouraged behavior, asking the interviewee to identify whether or
not they have had the opportunity to display the same in the course of their work to date which
should be confirmed by the interviewee providing examples in that regard. Complete the
relevant table elements during the discussion.
Desired Behavior

Identified by Interviewee Interviewee has had the

during discussion (Y/N)
opportunity to display the
behavior (Y/N)

1. Taking steps to avoid

contamination and crosscontamination bacterial,
physical, chemical, viral
2. Correct handwashing
technique
3. Washing hands at
appropriate times
4. Identifying and reporting
pests
5. Keeping work spaces clean
6. Using appropriate cleaning
techniques
7. Maintaining appropriate
level of personal hygiene
8. Wearing appropriate work
clothes
9. Cooking, hot holding,
reheating and cooling foods
to appropriate temperatures
2. Since you completed the course how eager have you been to display the encouraged food safety
and hygiene behaviors we have just discussed?
29

EVALUATION PLAN

____ Very eager

EDCI 577

____ Quite eager

____ Not eager

Comments:

3. How well equipped have you been to behave in accordance with good food safety and hygiene
practices as suggested on the course during your every day work?
____ Very

____ Quite

____ Little

____ None

4. If you are not doing some of the things you were encouraged and taught to do, why not?

Desired Behavior (#)

a. It wasnt practical for my
situation
b. My boss discourages me
from changing.
c. I havent had the chance to do
it yet.
d. I tried it but it didnt work.
e. Other reasons
Other reasons comments:

Appendix K

30

How Significant?
Very
Somewhat

Not

EVALUATION PLAN

EDCI 577

Level 3 Results Reporting Format

Table K1
Assessment of Retention Rates
Desired Behavior
Number
of
interviewees
identifying
desired
behavior at 3 months /
30
1. Taking steps to avoid
contamination
and
cross-contamination
bacterial,
physical,
chemical, viral
2. Correct handwashing
technique
3. Washing hands at
appropriate times
4.
Identifying
and
reporting pests
5. Keeping work spaces
clean
6. Using appropriate
cleaning techniques
7.
Maintaining
appropriate level of
personal hygiene
8. Wearing appropriate
work clothes
9. Cooking, hot holding,
reheating and cooling
foods to appropriate
temperatures

31

Number
of % increase
interviewees
decrease
identifying
desired behavior
at 3 months / 30

or

EVALUATION PLAN

EDCI 577

Q.2 Responses (3 months)

Very eager

Quite eager

Not eager

Q.2 Responses (6 months)

Very eager

Quite eager

Not eager

Figure K1. Sample comparison pie charts showing the division of total participant answers to
response options between 3 month and 6 month evaluations, which form is to be utilized to report
findings in relation to interview questions 2 and 3.

32

EVALUATION PLAN

EDCI 577

Number of Participants

Desired Behavior 1:
Taking steps to avoid contamination and crosscontamination bacterial, physical, chemical, viral
8
7
6
5
4
3
2
1
0
Not Practical

Boss Discourages

No Opportunity

Tried But Didn't

Work

Other Reasons

Very Significant
3 months

6 months

Figure K2. Sample comparison bar graph showing the division of total participant answers to
response options between 3 month and 6 month evaluations, which form is to be utilized to report
findings in relation to interview question 4. Separate graphs will be produced for each significance
category very significant, somewhat significant and not significant.

33