Effectiveness of Training Pilot Programs

Stephanie Winstead
Colorado State University
EDRM 600
April 29, 2012
Dr. Karen Kaminski
2


Introduction
The economy has struggled and will continue to slowly recover from the Great Recession. In
2008 and 2009, the U.S. economy shed more than 8 million jobs; since 2009, the economy has created
only about 2 million (Holzer, 2012, p. 031). Despite the condition of the economy, the American
Society for Training and Developments (ASTD) 2010 State of the Industry Report demonstrates a
continued dedication to workplace learning. ASTD estimates that U.S. organizations spent $125.88
billion on employee learning and development in 2009 (Patel, 2010). There are many costs associated
with training employees. The Society for Human Resource Management (2006) describes costs
associated with development of training programs, which include salaries and benefits of personnel, as
well as any equipment that is needed. Other costs include costs of training materials, technology costs,
facilities, travel, trainers salary and benefits. There may also be loss of productivity or costs of
backfilling positions during training.
There are concerns that employees will leave their place of employment after the employer
invests large quantities of money to train employees. However, cost is even greater when employers
choose not to train and employees stay. Employers must understand that investing more money in
training can save them money. The return of investment (ROI) from an effective training program can
be measured in terms of reduced employee turnover, fewer mistakes caused by human error, improved
safety and quality control, better products and service, and increased customer satisfaction (Keller,
2008, p. 040).
What are characteristics of an effective training program? One way to measure effectiveness is
by using Kirkpatricks evaluation model. The first level measures reaction, which is the learners
perception of the training; what they thought and felt about the training. The next level, learning, is the
resulting growth in knowledge, skills and attitudes based on their participation in the training. The third
3


level measures behavior, which involves the ability of the participants to apply what they have learned
in the training. The last level measures results, which demonstrates if the training met the targeted
outcomes, as a result of learning events and subsequent reinforcement (Kirkpatrick & Kirkpatrick, 2011).
The organization being studied is a generics pharmaceutical company located in Wilson, North
Carolina. The research study is being conducted in the quality control laboratory department. One
small laboratory error could cost the company billions of dollars, as well as put patients lives in danger.
In this research study, analysts with a higher error rate were chosen to undergo the training pilot
program, in hopes that error rates will decrease after completion of the training.
Problem
Despite the existence of current training programs in the quality control laboratory, some
analysts struggle to perform their jobs adequately. As a result, the analysts have experienced larger
than average number of errors in the laboratory. Rather than placing the analysts directly on a
performance improvement plan (PIP), management wanted the analysts to undergo remedial training
through a training pilot program. The program was created to improve the analysts performance by
giving them tools to improve and refine their analytical skills and techniques.
The training program had four distinct phases, which all involved different types of learning.
The first phase involved the trainee performing their daily functions with a mentor observing them. This
provided a baseline for the mentors so they could assess the trainees strengths and weaknesses. The
next phase was the remedial training. In this portion of the program, the analysts underwent training
with the laboratory trainers. At the request of management, training was conducted starting with basics
such as weighing and pipetting techniques and ended with more complex techniques. Training modules,
standard operating procedures, and hands on exercises were used for each different technique, and
analysts were not allowed to progress to a new technique until competency was demonstrated in the
4


technique at hand. After successful completion of the remedial training, trainees shadowed their
mentors. Analysts were permitted to help with small duties that had no product impact. The final
phase of the training pilot program involved a qualification exercise. The trainees were expected to
perform specific tests on a specified number of lots with minimal error. The analysts successfully
completed the training pilot program if the results of their testing were comparable to historical data.
Management would like to have more analysts undergo the training pilot program, but it is
necessary to determine if the program is effective as it stands. If the training pilot program is not as
effective as hoped, it may need to be changed to improve effectiveness before more analysts undergo
the training.
Purpose of the Study
I plan to conduct a research study using a mixed methods approach since I will be collecting
both quantitative and qualitative data. The purpose of my study is to determine whether or not the
training pilot program was effective.
Research Question
At the conclusion of this research study, I will hope to answer the following question: Is the
training pilot program an effective means to reduce analysts laboratory errors in the quality control
laboratory? Also, should we recommend making this training mandatory for all laboratory personnel?
Research Study Needed
In reviewing the literature, employers remain dedicated to workplace learning and development
despite the uncertainty of the economic climate (Patel, 2010). However, care must be taken when
developing a training program to ensure that it is effective. Illustrating measurable results is critical. If
a manager or an education and training department can demonstrate the value added by an effective
5


learning program, this business unit can be viewed as a profit center worthy of additional funding
(Keller, 2008, p. 043).
Although there is much data supporting the importance of workforce training, little study has
been conducted on whether the workforce training conducted is effective. This is an indicator that
more research needs to be conducted. Due to the changes in our economy and our changing workforce,
the types of jobs available have changed over the years. The drastic change in our economy is yet
another reason additional research is needed.
Study Design
This study attempts to determine the effectiveness of an analyst training pilot program
implemented within the quality control laboratory. The Kirkpatricks evaluation model is a good way to
measure program effectiveness, so this model will be used in the study. The four levels of Kirkpatricks
model will be used to evaluate the training pilot program: reaction, learning, behavior, and results
(Kirkpatrick & Kirkpatrick, 2011).
Method Permissions
Permission to conduct this study was obtained verbally in a meeting with the quality control
director. Approval of the institutional review board was not required for purposes of this study.
Consent of participants will be demonstrated by completion and return of the consent form at the
interview (Appendix A).
Participants
The four analysts selected to undergo the training pilot program were selected by quality
control management using TrackWise, a system designed to track errors that trigger laboratory
investigations, deviations, and events. Analysts with high error in the TrackWise program were chosen
to participate in the training. Only four analysts could undergo the training pilot program at this time
6


because this program removes both the analysts and the mentors from testing for six weeks. However, I
do not believe the small sample size will affect results since I will be collecting both quantitative and
qualitative data to conduct the study.
Instrument and Data Collection Procedures
Since both qualitative and quantitative data will be collected, a mixed methods approach was
determined to be the most appropriate method for this study (Creswell, 2012). I utilized the TrackWise
system to compare the number of recordable laboratory errors before and after completion of the
training pilot program for each analyst. I also conducted one-on-one, qualitative interviews to obtain
opinions and feedback regarding the training pilot program, which was feasible considering the small
sample size. The interview questions consisted of open-ended questions. The responses were
recorded, and trends in responses were noted. The interviews conducted were semi-structured,
covering key issues and questions (Appendix B). However, additional question were asked, especially
when unplanned responses are encountered (David & Sutton, 2004). The specific instrument used is an
adaptation of a semi-structured interview conducted by the Institut national de sant publique du
Qubec (INSPQ) (2009). Interviews conducted by the INSPQ discussed crime, disorderly conduct, and
incivility. Citizens perceptions of safety and assessment of public services in their area were also
assessed during the course of the interviews.
Data Analysis
Both quantitative and qualitative data were collected and organized. Data from TrackWise was
analyzed by comparing the number of errors that triggered a laboratory investigation, deviation, or
event before and after the training pilot program for each analyst. Responses from the one-on-one
interviews were studied for trends of analysts perceptions of and attitudes toward the training pilot
program.

7


Development of the Research Hypothesis
This research has two research hypotheses. The first hypothesis examines the learning outcome
of the training pilot program. If the training pilot program is ineffective, one would expect analysts not
to learn any new knowledge or analytical skills from the training pilot program. The null hypothesis
confirms that there is no significant difference between the number of laboratory errors triggering
laboratory events before and after the training pilot program. The alternative hypothesis validates that
there is a significant difference between the number of laboratory errors triggering laboratory events
before and after the training pilot program.
The second research hypothesis studies the analysts reactions to the training pilot program. If
the training pilot is considered ineffective, one would expect analysts to express unsatisfactory attitudes
regarding the training pilot program during the interviews. The null hypothesis proves that there is no
association between the analysts responses and the training pilot program. The alternative hypothesis
supports an association between analysts responses and the training pilot program.
Results
In this study, the Friedman Test was applied to the quantitative data retrieved from TrackWise
(Table 1). The Friedman Test appeared to be the most appropriate statistical study since this non-
parametric test is ideal for detecting variances in treatments across multiple test attempts. Results of
the test concluded that there was no significant difference between the number of laboratory errors
triggering laboratory events before and after the training pilot program (
2
F = 1.125, df = 2, N = 4) at the
5% critical level. Because of the small sample size, these findings can only be considered
approximations. Friedmans test is more beneficial for samples sizes of seven and above. However, the
amalgamation of qualitative data from interviews and the quantitative data from applying the Friedman
Test will aid in measuring overall effectiveness of the training program.

8


Table 1. Recordable Laboratory Errors from TrackWise
August 2011
(Before program)
December 2011
(After program)
February 2012
(After program)
Analyst 1 1 0 0
Analyst 2 2 0 0
Analyst 3 0 0 0
Analyst 4 0 0 1

At the conclusion of the interviews, all responses were analyzed closely for trends regarding
analysts reactions to the training pilot program. All trends discovered were recorded, separating the
positive and negative responses (Table 2). A thorough review of the interview responses indicates that
the analysts believed that they benefited from the program. However, analysts felt that incorporating
some changes in the future would improve the overall success of the training pilot program.
First, all analysts would have preferred the training to have been more customized for their
specific needs. Management had everyone undergo the same training to protect the privacy of the
training pilot program participants. After discussion with the analysts at the conclusion of the training
program, management has agreed to disclose more information to the trainers. As a result, all trainers
will be more aware of the errors routinely made by analysts undergoing future trainings so they can
concentrate on areas of weakness.
Next, analysts suggested that more consideration should be given regarding the selection of
trainers for the program. An effective trainer must possess qualities such as patience, respectfulness,
and optimism. While one of the trainers selected was very knowledgeable in the field, she lacked many
of these key attributes. The analyst assigned to work with this particular trainer may have benefited
more had he been assigned to work with a better trainer. In the future, training coordinators would like
to be more involved with management during the selection of trainers since the trainer can directly
affect the outcome of the program.
9


The training pilot program was designed to accommodate sufficient time for trainers to observe
analysts. However, both analysts and trainers believed that observing the analysts for two weeks was
very monotonous. Both parties agreed that working together would prevent boredom and further
promote healthy working relationships.
Table 2. Summary of Findings Interview Responses
Pros Cons
Analysts feel they learned from program. Training needs to be more customized for each
analyst based on their specific errors.
Program raised awareness of standard operating
procedures/training modules.
Better selection process between trainers/trainees
should be considered.
Program helped build healthy working
relationships.
Some phases of the program lasted too long.
Program contained a good mixture of educational
exercises.
More management involvement needed (group
meetings, one on ones, etc.).
Program provided communication of best
practices.
Program should allow trainees and trainers to help
each other to prevent boredom during observation
phase of program.

Discussion
This study examined the effectiveness of a training pilot program implemented within the
quality control laboratory in an effort to reduce analyst error. Effectiveness was assessed by applying
aspects of Kirkpatricks evaluation model, which measures reaction, learning, behavior, and results.
The results of the quantitative data indicated that there was no significant difference between the
number of laboratory errors triggering laboratory events before and after the training pilot program.
This result alone would render the training pilot program ineffective due to the impact of having a small
sample size. The analysts perceptions of the training usefulness and their efforts to gain knowledge and
skills are significant variables in determining training effectiveness. Feedback regarding the training
pilot program was analyzed by conducting semi-structured interviews. The results of the qualitative
analysis revealed that although analysts gained knowledge from the trainings, there are many changes
that should be implemented to increase the effectiveness of the program. The training should be
10


customized to suit the specific needs of the analyst. In an effort to respect privacy of the participants,
quality control management did not want to disclose the specific problems each analyst possessed.
However, the analysts would have preferred to center the training program on their specific sources of
error. One of the analysts had difficulty working with their assigned trainer. Because the trainer did not
possess qualities of an adult educator, the analyst was not receptive to the trainers demeanor. More
time and effort needs to be devoted to trainer/trainee selection in the future. In conclusion, the
researcher recommends that changes be made to the current training pilot program to improve training
effectiveness. Based on the feedback given from the analysts, this study reveals that the training pilot
program can be further enhanced to accommodate the differing weaknesses of all participants.
11


References
ASTD. (2010). Continued Dedication to Workplace Learning. Retrieved from
http://www.astd.org/TD/Archives/2010/Nov/Free/1110_2010+State+of+the +Industry.htm.
Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and
qualitative research (4
th
ed.). Boston, MA: Pearson.
David, M. & Sutton C.D. (2004). Social Research the Basics. London: SAGE Publications.
Holzer, H. (2012). Better Skills for Better Jobs. Issues in Science and Technology, Winter 2012, 031
040).
INSPQ. (2009) Safety Diagnosis Tool Kit for Local Communities. Retrieved from
http://www.crpspc.qc.ca/Guide_entretien_versionWEB_eng.pdf.
Keller, D. (2008). Tune-up for trainers: Help employees stay on top of their game. MLO: Medical
Laboratory Observer, 40(6), 40 43.
Kirkpatrick, J. D., & Kirkpatrick, W. (2011). Creating ROE: THE END IS THE BEGINNING. T+D, 65(11), 60-
64.
SHRM. (2006). Training Cost Per Employee. Retrieved from
http://www.shrm.org/Research/Articles/Articles/Pages/MetricofthemonthTrainingCostPerEmpl
oyee.aspx.



12


Appendix A
CONSENT FORM FOR TRAINING PILOT PROGRAM RESEARCH STUDY
I agree to participate in this research study, whose conditions are as follows:
The study is aimed at determining the effectiveness of the training pilot program. For this
purpose, semi-structured interviews will be conducted with participants of the training pilot
program.
Interviews will last for approximately one hour, and questions are based on participants
perceptions and attitudes toward the four specific phases of the program.
At any time, I can decline to answer certain questions, discuss specific topics, or even
discontinue the interview without bias.
All interview data, including notes taken during the interview, will be handled so as to protect
participants confidentiality. Therefore, no names will be mentioned and information will be
coded.
All data will be destroyed at the conclusion of the study.
For any information about the research study or training pilot program, I can contact Stephanie
Winstead at (252) 234-2405.
Participants signature: ______________________________________________________
Date: _______________________________
Interviewers signature: _____________________________________________________
Date: ________________________________

13


Appendix B
SEMI-STRUCTURED INTERVIEW PLAN FOR TRAINING PILOT PROGRAM
Introduction of interviewer:
Hello, my name is Stephanie Winstead, and I will be asking you some questions regarding the training
pilot program you recently participated in. During the interview, I would like to discuss your
perceptions, opinions, and feedback on the training pilot program.
Primary Questions Secondary Questions Additional Clarifying Questions
Phase 1 Questions:
Can you describe your
experience during the first
phase of the program, which
consisted of the mentor
observing you?
What problems, if any,
arose during the first phase
of the training pilot
program?
What are some ways that
you think this phase can be
improved for future
participants of the training
pilot program?












Can you expand a little on
this?
Can you tell me anything
else?
Can you give me some
examples?















Phase 2 Questions:
Can you describe your
experience during the
second phase of the
program, which was a review
of basic laboratory skills with
the QC training
coordinators?
What problems, if any,
arose during the second
phase of the training pilot
program?
What are some ways that
you think this phase can be
improved for future
participants of the training
pilot program?
Phase 3 Questions:
Can you describe your
experience during the third
phase of the program, which
consisted of you shadowing
your assigned mentor?
Was your assigned mentor
receptive, helpful, and
amicable? Did your mentor
offer suggestions to improve
your current skills and
techniques in the lab?
Would you recommend
your assigned mentor to
participate in future training
programs, pilot or
permanent?
What problems, if any,
arose during the third phase
of the training pilot
program?
14


What are some ways that
you think this phase can be
improved for future
participants of the training
pilot program?






Can you expand a little on
this?
Can you tell me anything
else?
Can you give me some
examples?












Phase 4 Questions:
Can you describe your
experience during the final
qualification phase of the
program?





What problems, if any,
arose during the final
qualification phase of the
training pilot program?
What are some ways that
you think this phase can be
improved for future
participants of the training
pilot program?
Conclusion of Interview:
Are there any other issues
with the training pilot
program that we have not
discussed?
Do you have any additional
feedback on the training
pilot program that would be
helpful for future
implementation of the
program?