Evaluation of Determining Instructional Purposes (DIP) A Proposal Submitted to Far West Laboratory for Educational and Research Development

By Zae Evaluation Services July 10, 2012

Introduction The Far West Laboratory for Educational and Research Development (FWL) has made a preliminary investment in the design and implementation of a training program in the form of a handbook entitled Determining Instructional Purposes (DIP). They now decided to do an evaluation before moving forward with continued investment and possible expansion of the project. FWL is accepting bids for Request For Proposals (RFPs) from suitable candidates that will focus the evaluation on providing information and recommendations for use in making decisions regarding marketing and sale of the DIP units. This document is being submitted by Zae Evaluation Services (ZES) in accordance with the Request For Proposal (RFP) put forward by FWL. Description of Program Being Evaluated The DIP training package was developed by FWL for the purpose of training school administrators and graduate students in educational administration in skills related to planning of effective school programs. The DIP package consists of a coordinators handbook with three units with each containing six modules and activities that actively engage learners in authentic administrative duties. Each unit focuses on a certain concentration: (i) setting goals, (ii) analyzing problems, and (iii) deriving objectives. FWL estimates of the training time required for the units are 10-15 hours each for Units 1 and 3 and 12-18 hours for Unit 2. The units can be run individually or concurrently depending on the needs of the school system at a given instance. The coordinators handbooks can be purchased individually or by the set. A coordinator is required to organize, guide, and monitor activities in which participants use the materials and procedures contained in the units. According to the program developers, the coordinator should have worked through the relevant unit(s) on his/her own before embarking on any training exercise although he/she does not require a special skill set or prior knowledge of the content area before he/she administers the training sessions. Evaluation Method The success level of any school system is largely dependent on prudent spending, accurate decision making by key stakeholders, and the willingness of stakeholders to participate in training exercises. However, before moving forward from this standpoint, a comprehensive set of evaluation questions needs to be addressed. Is the training material adequate and appropriate for the target audience? Are the three units designed and implemented properly to meet the intended learning outcomes?

What impact will the program have on participants when they return to their normal administrative duties? The goal of the assessment exercise is to determine if administrators will find participating in the training a worthwhile monetary investment that should be sustained. The DIP training program will be evaluated on several fronts. It is the intention of the evaluator to categorize the evaluations as either formative or summative in nature. Formative assessments will focus on two target populations the training participants and the training coordinators. The summative assessments will focus on program participants and school system administration. For this evaluation, it is suggested that the training units should be run three-days per week from Monday-Wednesday over the six weeks period allowing administering of assessments prior to the next training week. Formative Assessments Assessment 1 Training participants will be asked to take part in surveys and interviews throughout the training to assess the effectiveness of the instruction and facilitation. A Likert-type scale will be incorporated to assess the effectiveness of each training session. The surveys will also contain areas for participants to make anecdotal comments about the program, identifying areas of strength and recommend them for correction. Did the participants feel that the instruction was adequate in meeting learner need? Did the participants feel that the level of instruction was in line with expectations? All participants will be interviewed since the group is relatively small. Assessment 2 Interviews will be held with training coordinators twice per unit. The interview questions will focus on two areas: The effectiveness of the training preparation, and areas for revision based on implementation timeline. It is important to determine if the coordinator preparation is adequate for the training programs to run effectively. As such FWL will need to assess the validity of the method used to select coordinators. Summative Assessments Assessment 3 Both Pre and Post-testing will be administered to all program participants to assess learning outcomes for each of the three project units. The pre and post tests will consist of educational scenarios similar to the modules and group activities present in each instructional unit. Participants will be asked to make administrative decisions based upon the information presented. The post testing will be used to assess how decision making skills have changed and adapted based upon new understandings. Some scenarios may be repeated in both the pre and post testing to further measure growth. The purpose is to assess if participants meet the objectives as outlined in the program protocol. Answer to question such as: Were students able to make adequate progress in the training program? ...will be solicited. Assessment 4 Interviews with administrators to assess overall satisfaction with the program as it pertains to meeting desired outcomes. This will be conducted three months following the training. This later date will be done to determine the impact of the training in the actual school setting.

Administrators will be interviewed to assess the authentic impact of the training on the administrative staff and the overall impact on their routine tasks in the work place. Task Schedule Task 1. Preliminary 2-day meeting to be conducted at FWL. This will entail opening dialogues between FWL and ZES. All three representatives of ZES will be present. The goal is to determine evaluation strategies and the locus of focus. 2. Videoconference on http://www.gotomeeting.com/fec/ between Mr. Zae and FWL representatives. The purpose of this meeting is to review the questionnaires, surveys, and interviews designed by Mr. Zae for the evaluation process. 3. Final submission of evaluation design revisions to ZES by FWL for final mark-up. 4. Meeting with FWL to examine final drafts of questionnaires, surveys, and interviews. Meetings will also include planning sessions for evaluation strategizing. 5. Initial posting of instruments on collaborative wiki http://kirklandlawrence.wikispaces.com/ to facilitate editing before posting instruments to Google Docs Survey. 6. Full six weeks training session for all three units in the DIP training program during September and part of October. All scheduling of workshops will be coordinated through the wiki and by emails. 8. Meeting with FWL to forward ZES with all Pre and Post test data, all interviews and all surveys. This meeting will also be used to make determinations about which participant will be used for the long term impact assessment after the training session. Responsibl e FLW Deadline June 10, 2012

ZES

July 3, 2012

FWL ZES

July 24 August 15

ZES

August 29

FWL

September 3 - October 12, 2012.

ZES

December 18, 2012

9. Emails will be sent to participants detailing their role in the process. They will each be given the link to Google Docs Survey to participate in the DIP assessment. This site will be used to track and monitor participants as they complete the surveys online. 10. Meeting with FWL to summarize the pre and post test data, the participant interviews and surveys, and the coordinator interviews. 11. Close the DIP survey portal in Google Docs Survey. Begin review and analysis of data. 12. Submit final report to FWL .

FWL

January 20, 2013

FWL

February 30, 2013

FWL ZES

March 10, 2013 April 14, 2013

Project Personnel Rick Zae - President of ZES, he has a masters degree in sales and marketing and have done several evaluation of similar training programs. Kirkland Lawrence - CEO, possessed a masters degree in educational technology and have 5 years experience conducting evaluations which are technology related, and Clancy Jones - secretary/ liaison officer, he has a bachelors degree in business administration, he will coordinate meetings, making phone calls, sending emails etc. Budget or Fee Budget of Expenditure Personnel Subtotal "Mr. Zae - Project Director (40 days @ $450/day)" $18,000 "Mr. Lawrence - Evaluation Coordinator (40 days @ $375/day" $15,000 Mr. Jones -secretary/ liaison officer $6,500 TOTAL $40,500 Travel & Accommodation Flights from Miami > Boise (June 15, August 15, Feb. 30, 2013 & April 14). Subtotal $2,240.00

Hotel Accommodation $2,000.00 Estimated mileage between hotel & FWL 260 miles@ $0.555 $143.00 TOTAL $4,383.00 Supplies/Materials Paper, binding materials Printing services Stationery supplies TOTAL Miscellaneous Per diem meals TOTAL Program Supplies Coordinator hand books Unit handbooks Other TOTAL Sub Total $600.00 $375.00 $280.00 $1255 Sub Total $1, 350.00 $1,350.00 Sub Total $195.00 $1250.00 $300.00 $1745.00

Total Estimate of Expenditure = $49,233.00 Payment Schedule Payment Percentage Payment 1: 25% due after first meeting Payment 2: 25 % due at the commencement of training Payment 3: 25% Due at summary meeting Payment 4: 25% due at final meeting with FWL to present report. Due Date June 10, 2012 Dec. 18, 2012 February 30, 2012 April 4, 2013 Amount $12,308.2 5 $12,308.2 5 $12,308.2 5 $12,308.2 5