‘Running head: UNIV 101 UNIV 101: A Formative Evaluation Plan ‘Andrea De Leon and Jonathan Mersill Loyola University ChicagoUNIV 101 ‘Table of Contents Tntroduetion 3 ‘Theoretical Framework 3 UNIV 101 Objectives 5 ‘UNIV 101 Stakeholders 6 ‘UNIV 101 Course Content 7 ‘Evaluation Logie Model 8 "Assumptions and External Factors 10 Evaluation Purpose 1 Evaluation Design 3 Strengths and Limitations 14 ‘Quantitative Methodology 15 Population and Sampling Frame 15 ‘Research Desi 16 ‘Survey Instrumeni Deseription 16 Pilot Testing 17 Procodures and Implementation 18 ‘Analysis and Reporting. 9 ‘Qualitative Methodology 20 Participants 21 Instrument 2D Implementation 2 ‘Analytical Approach 24 Dissemination 26, Budget 26 Timeline 27 Next Steps 2 ‘References 29 3 ‘AppendicesUNIV 101 3 introduction institution, is home to more than Loyola University Chicago, a privat, Jesuit, Caol 15,000 graduate and undergraduate stents. With the undergraduate stulent population siting 1 around 10,000 stent, the institution focuses much ime and effort on developing and providing programs and opportunities that support undergraduate’ news, The Office of First and Socond Year Advising (FSYA) exis a8 «functional unit within Loyola's Academic Ais pmtey © sian kate © ota © ste ereon © sedate raw voz © Nott Toor ser sen © shontioa © Tay a cote © Veena ven 13, Does your UNIV 101 course section have a Constructor? * ‘Does your curse section have to instructors in ation o the Peer Adwa0r? Mark only one ova. Ors Ow 14, Please indicate your status” ‘Mork ony one ova. (D Resident Start (Wve n residence all) (O Commuter Student (do not ive ina reser hal) 415, Please indicate your racelthnicity * Mark only one ova D AticanaticanAmercanack fiisin erica O Henarictsino OD wutwacst Neve Amercaiaosk Nave Nave Hanalen/oter Pete slander D vetcaveasin © otter:16. Please indicate your gender * ‘Mark ony one ova © Female O mete O Tenegondor CO Oiter: BB coogi roomsAppendix ©UNIV 101 Assessment Post-Test Survey ‘The information you provide on this survey wil help us Improve UNIV 101 courss fr next year: t Wil algo help us understand UNIV 101s meeting is goals. Your responses wil remain confident |We are Interested in ganaal information an wll ol be using any names in out evaluation report. ‘We appreciate you Taking the tims fo complete the survey. Once you have completed both the UNIV 401 BartopantPre-Survey and the UNIV 101 Parcpant Post-Suvey ou ill be entered ino & rain for one of ten $25 Amazon git carés. Yu wil be contacted via ema no ler than December 31,2016 you area winner. Please contact Jonathan Menta jesil@luc ed or (779) 508-680 you have any questions * Required 1. Survey Da” Please ener the G-ig ID # sent fo you nthe post teat survey invaton ema (ex. mitt) Section | Please rate your lovel of canfdence with the flowing tems 2, My ability to consider perspectives that differ from my own. * ‘Mark ony one oval O Ver content © Consent © Somewhat Content (O Semennat Unconfident ( Not content At All 3, My ability to analyze arguments presented In class. * Mark only one ova. >) Very Confident 5) Confident ¢ c © Somewhat Contdent (© Somewhat Unconfident 5) Not Confident ALA4. My ity to prtpate in classroom activites. * ‘Such as: clean onine cuss, ru aks, ae otf classroom co-curcor Mako oa. © vov content OD content Somewhat Contdent (Somewhat Unconident Not Content ACA 5, My ably to seek oul eorcuricular activities on campus.” ‘Mark only one ova. O Ver content © Content Somennat Content () Somensat Unconfcont O Notconident att 6, My ability to select an academic malor that aligns with my career goals.~ Mark only one ova © ver Content © Contdent O Somewtat Confitent © Somewhat Unvontdent Nat Confident At Section Please rat your evel of understanding ofthe following isms 7 The characteristics ofa Jesuit Education.” Mork ony one ova Peay unaortns O voderstand Semen Understand (O Semen D0 Not Understand CO No voderstanaing At At8. The various support services on campus." ‘Such as: Financial Ai, Academse Advising, Weiness Centr, Campus Minis, Mark nly one ova OD Fay Understand © bnderstand (Some Understand © Somat Oo Not Understand O No Understanding AA ‘8. The strategies to navigate my transition into college lite." Such as: Personal transition, academic tension spinal anton, ‘Marken one ova © Fry Understand © Understand © Somewhat Understand (© Somewhat Do Not Understand (No Understanding LAL 10. The academic requirements that | must meetin order to graduate. * Ma ony ono ova, O Py understend © dooerstand (© Somowtat Understand ( Somenat Do Not Understand ( NoUnderstaning ALA 11, How the decisions make in college affect my career goals. ‘Such as: Academic decisions, personal decisions, spiual decisions, et. Mark only one ova © Fatty Understand CO dnderstana Somewhat Understand (Somewhat De Not Understand Noneerstanding At all Section Ill Preace rate yur level of slefacton with the folowing items12. The quality of teaching provided by the course instructor(s).* ‘Mark ony ane ova O ver sates © satstos & Somewhat Satsied O Someunat Dissatses ( Disatistes 19, The way In which course content was delivered." For example: Readings, lectures, online discussions, co-cuicular ates, et. ‘Mark only one ova. CO wor sates O sates CH Somewhat Satises G Somentat Dssoished diessistos Part IV Please rate the level o which you agree withthe following 14, My course instructr(s) ws prepared to teach the course Nore speccaly My couse ns) was organized, planed ahead, ntetional wih content dave. Markoni ore ova, © Swronaly Agree O Aare CD somennat Agee somentat Deagroe O Strongly sages 18. My course instructor(s) was knowledgeable of cour ak onyone ova! O Strona Aare O hace CD somennat gree O Somentat Osage O Sronaly Disagree46. My course instrucor(s) was confident In thelr dlivery of course contont. ‘Mark oly one ova O sronay Agree O favo D Somentat Agree D somennat Dsagree O srongy osagree Bi cooge oomsAppendix D(asansog) wesZoug UL aoualsadxg ||e42AO gto ‘sto ‘yt ‘ero'71D 7 auowasinbas ‘aamanns eee eC) ‘quapene seo ayeyotue pue puersiapun sanssy euonisuen 229481 6D << sountoo sauppe o sasayens aesisuoweg sazunosey 5,1] pue ejoAo- ucts yo soa wns ojo. | pue eOK07 sndu vo somosaipueoqioes ynuap, “/ \NOQE aBPa/Mouy oaeanpaansor 0 onaponesey ay aeponue puePuRUN Dp <——— on sofeu eyo uon2oies 90 << _anpue Bguuee soos san0id ewig ssaoong aIWapery 10} ue s2uepuane Jo aouevodtu ayy pueisiapun SIPS Ul e2UapyUoD so‘vo << suonsoésp // oui onsen po oLed uonsano sawior3ng BulWe3) seweyl dew yonsysuoD Aeains TOT AINA.Appendix BDear Student, 'As a current participant in Loyola University Chicago's UNIV 101 course, we are inviting you to participate in an assessment project that is taking place this semester. The main goal ofthis assessment isto better understand how UNIV 101 is meeting its goals, which will allow us to make improvements for future courses. \We are asking current students to submit survey responses at both the beginning and the end of their participation in UNIV 101. Your involvement in tis project would require that you submit the fist survey no later than September 11, 2015 and the second survey no later than December 20, 2015. Note that not al, students who express interest will be eligible to participate. Email invitations to complete the surveys will be sent to eligible participants. Students who elect to participate in this project, and who complete both surveys as requested will be entered into a drawing to win one of ten $25 Amazon gift cards. The drawing will take place in December and winners will be notified via email no later than December 31, 2015. “To express interes in taking part in this project, please submit the form via the link provided below no later than August 28, 2015. Use this link to express interest: http://tinyur.com/m7ev837 you have any questions or concerns, please contact Jonathan Merrill at jmerril@luc.edu, Thank you for your consideration in helping to make UNIV 101 a great experience forall students Sincerely, UNIV 101 Assessment Project TeamAppendix FUNIV 101 Assessment Participant Interest Form “Thank you for considering participation inthe UNIV 101 Assessment Project. Please submit this frm ro later than Frigay, August 28, 2015. Ifyou have any question or concerns, please contact ‘Jonathan Berri at mariah ou + Roguired 4. First Name" 2, Last Name” 3, LUG Email Address * twol@ucedu 4, Phone Number * yw 5. Lunderstand that by submitting thie form, | am commiting to comploting the protest ‘survey no later than September 11, 2015 and the posttest survey no later than December 20,2015." ‘Note: elibe to parcpate, you wil receive an ema inviting you to complete each survey ‘Mark ony one ova Ow On Bi coogi FameAppendix GDear Student, “Thank you for expressing interest in participating in the UNIV 101 Assessment Project. As you may recall, the ‘main goal of this assessment is to better understand how UNIV 101 is meeting its goals, which wil allow us to ‘make improvements for future courses. Your participation i incredibly valuable. [At this time, we are asking that you complete the UNIV 101 Pre-Survey by using the link provided below. Please submit this survey no later than September 11, 2015. Use this link to submit the UNIV 101 Pre-Survey:hito://tinyurl.com/nasendn Survey ID #: waste 'As a reminder, students who complete both surveys as requested will be entered into a drawing to win one of ‘ten $25 Amazon gift cards, The drawing will take place in December and winners will be notified via email no later than December 21, 2015, ‘You will be sent an email invitation to complete the UNIV 101 Post-Survey toward the end of November. I you have any questions or concerns, please contact Jonathan Merrill at jmersil@luc.edu. ‘Thank you for your consideration in helping to make UNIV 101 a great experience for all students Sincerely, Brian Keiller Assistant Dean & Director of AdvisingAppendix HDear Student, “Thank you for expressing interest in participating in the UNIV 101 Assessment Project. You received an email invitation to submit the UNIV 101 Pre-Survey on August 31, 2015. At ths time we are sending a reminder invitation, asking that you complete the UNIV 101 Pre-Survey by using the link provided below. Please submit this survey no later than September 11, 2015. Use this link to submit the UNIV 101 Pre-Survey: httn//tinyurl.com/oasepdn Survey 1D: ait [As you may recall, the main gnal of thic accaccment isto better understand how UNIV 101 is meeting its goals, \which will allow us to make improvements for future courses. Your participation is incredibly valuable. As a reminder, students who complete bath surveys as requested willbe entered into a drawing to win one of ten ‘$25 Amazon gift cards. The drawing wil take place in December and winners will be notified via email no later than December 34, 2015. You will be sent an email invitation o complete the UNIV 101 Post Survey toward the end of November, If you have any questions or concerns, please contact Jonathan Merrill atjmerril@luc.edu, ‘Thank you for your consideration in helping to make UNIV 101 a great experience forall students Sincerely, Brian Keiller ‘Assistant Dean & Director of AdvisingAppendix 1Dear Student, ‘Thank you for participating in the UNIV 101 Assessment Project. As you may recall, the main goal ofthis assessment isto better understand how UNIV 101 is meeting its goals, which wil allow us to make improvements for future courses. Your participation is incredibly valuable. [At ths time, we are asking that you complete the UNIV 101 Post Survey by using the link provided below. Please submit this survey no later than December 20, 2015 Use this link to submit the UNIV 101 Post Survey: http://sinyus.com/ovmaném ‘Survey ID #: wae ‘As a reminder, students who complete both surveys as requested will be entered into a drawing to win one of ‘ten $25 Amazon gift cards. The drawing will take place in December and winners willbe notified via email no later than December 31, 2015, Ifyou have any questions or concerns, please contact Jonathan Merril at jmerrill@luc.edu, ‘Thank you for your consideration in helping to make UNIV 101 a great experience forall students Sincerely, Brian Keiller Assistant Dean & Director of AdvisingAppendix 3Dear student, ‘Thank you for participating in the UNIV 101 Assessment Project You received an email invitation to submit the UNIV 101 Post Survey on November 30, 2015. At this time we are sending a reminder invitation, asking that you complete the UNIV 101 Post Survey by using the link ‘provided below, Please submit this survey no later than December 20, 2035. Use this link to submit the UNIV 101 Post-Survey:http://tinyurl.com/ovmaném Survey 1D #: Haat [As you may recall, the main goal of this assessment isto hetter understand how UNIV 101 is meeting its goals, ‘which will allow us to make improvements for future courses. Your participation Is incredibly valuable. As 3 ‘reminder, students who complete both surveys as requested will be entered into a drawing to win one of ten ‘$25 Amazon gift cards. The drawing will take place in December and winners will be notified via email no tater than December 31, 2015 Ifyou have any questions or concerns, please contact Jonathan Merril at jmerrill@luc.edu. ‘Thank you for your consideration in helping to make UNIV 101 a great experience for all students Sincerely, Brian Keller Assistant Dean & Director of AdvisingAppendix K‘To: (PARTICIPANT'S NAME) From: (INTERVIEWER'S NAME) ‘Subject: UNIV 101 Assessment Projet Dear (PARTICIPANTS NAME), ‘Lam writing today to invite you to paricpete in an evaluation project for the UNTV 101 course at Loyola University Chicago. Asa course instructor, you are Faia withthe design, format, and program outcomes. An evaluation tam has been working ona project to better understand how the training and preparation for course instructors impacts the quality of content delivery and, therefore, overall outcomes achievement. The quantitative part of our project consisted of survey distribution o UNIV 101 students in an effort to understand outcomes achievement. “The qualitative component of our project is focused on conducting one-on-one, semi-structured {interviews with UNIV 101 course instructors. We hope to engage in fruitful conversations about the experience of teaching UNIV 101, and how training and preparation impacts that experience. ‘We ate inviting you to participate in our qualitative asessment. This isa voluntary opportunity, ‘but highly encouraged, as your paticipation will help us to make improvements to future ‘raining and preperation programs. If you would like to commit to participate in this opportunity, please respond to this email no later than December 20,2015, with ttee (3) interview datestimes that you prefer, selected from the list below. Once our team has confirmed your interview slot, you will eceve further ‘communication providing more details and next steps fr this process, Interview Dates Times: February 2, 2015: 9:00am-9:4Sam; 10:00am-10:4San;12:00pm-12:45prn February 3, 2015: 9:00am-0-4Sam; 10:00am-10-4Sam; 12:00pm-12:45p February 4 2015: 9:003m-9-4Sam; 10:00am-10:4Sams; 11:00am 1:48am; 12-00pm-12:4Spm Februaty 9, 2015: 9:00am-94Sam; 10:00am-0:4Sam; 12:00pen-12:45pm February 10, 2015: 12:00pm-12:4Spas,1:00pmy-1:43pm; 200pm-2:45pm ebruaty 11,2015: 12:00pm-12:45pm:;1:00pm-I:4Spe;2-00pm-2-45pm ‘Thank you, so much, foe your consideration, Your participation is only going to help us make the UNIV 101 experience better for both teachers and students. Please do not hesitate to contact ‘me with further questions or concems. Best, (INTERVIEWER'S NAMB)Appendix LCONSENT TO PARTICIPATE IN INSTRUCTOR UNIV 101 INTERVIEW Project Tite: University 101 Assessment Evaluator: Jon Mesilla Andrea De Leon Introduction: ‘You are being est take par in an interview to help Loyole FYSA staTttr understand how the proces of course instrtor taining impacts the achievement of learning outcomes. ‘You erebsing asked to participate because you curently instructor eo-nstrut an UNIV {0 couse, Please rad this form creflly and ask any questions you may have before deciding ‘wether fo pticpate in tis interview. Purpose: “The purpose ofthe interview isto gather information about insteectors' experience during training and in delivering course content, This information willbe used oelp improve UNIV 101 courses for future students Procedures: If you agro to pantcipate inthis interview, you wll be asked to respond to a series of questions ‘hat wil lst foraproximstely 45 minutes. You have the option o choose to not answer ‘questions and/or withdraw from the iterview at anytime without penalty sks/Benefits: “There are no known risks involved in participating inthis study IPat anytime you feel uncomfortable answering & question, you may remove yourself and/or decide not to answer the ‘questionConfidentiality: ‘Your name and identifying information will not be associated with your responses inthe interview reports. We wll compile a report of basic themes and share it with you prior to finalizing “The information gathered in today’s interview will ony be shared with Bian Keller, Assistant Dean and Director of Advising and other FSYA staff members. Other Loyola stat include, but are not limited to, Deans and Assistant Deans, and Assistant Provost. ‘Voluntary Participation: Pantsipation inthis interview i voluntary, but highly encouraged. Even if you decide to paiticpate, you are eto witha fom participation a anytime without penalty Contacts and Questions: you have questions abou the interview, you can contac Jon Merl t mes. Statement of Consent: ‘Your signature below indicates that you have read and understood the information provided hove, have ha an opportunity to ask questions, and agree to participate in this oeus group. ‘You will be given a copy of tis form to keep for your recon, Participants Signature Date Tntorviewer's Signature DateAppendix M‘One-on-One Semi-Structured Interview Protocol, ‘Overarching Focus ofthe Interview: To examine in greater depth how the experience in delivering course content, and therefore the achievement of leaming outcomes, is impacted by ‘taining and preparation ‘Specific Foci ofthe Interview: 1. Expecations and Assumptions ofthe Training/Preparation Program 2. Experience withthe Truining’Preparaion Program Process 3. Experience of Delivering Course Content in the Classroom 44. Impact of Training on Performance and Overell Outcome Achievement Interview Seript Introduction ‘Thank you for joining us today to participate in this interview. My name is (INSERT NAME), 1nd my colleague (INSERT NAME}. Our goal fortis interview is o gain a beter understanding, ‘of how your taining impacted your experience as an instructor in UNIV 101. We will discuss ‘expectations of and experience with taining, experience teaching th course, and overall ideas about how training impacted your preparedness and effectiveness in teaching. By taking time to sit and chat about your experiences delivering course content, we hope to add depth tothe fecdback that was gathered during the surveys. We greatly appreciate your participation thus far! “The interview should take approximately 45 minutes Pease note that you are fre to end this interview or skip any questions that ee raised if at any point you fel uncomfortable, The information gathered today will be shared with UNIV 101 sta and be wsed to improve the ‘quality ofthe taining and preparation program for fatare UNIV 10] instructers. Because ofthis, ‘we encourage you to provide open and honest feedback, and do so withthe comfort of knowing tha the evalustion team will not disclose any of your identifying information in ts report, At this time [ask that you review our intervie” consent form to make sure that you understand the details of what will occur today. By signing te form you consent to participate in this interview. we of the Interview "Now that you ave completed the consent form, I would like to provide you with some ‘nditional guidelines for our conversation today. | havea series of questions [ willbe using to ‘Buide our conversation, The goa isto bette understand your experience so you might notice me ‘asking fr clarification on specific points you make or perhaps posing follow-up questions 80 ‘that ean probe deeper into something you have shared. Again, if at any point in our dialog we ‘each a point where you are no longer comfortable encourage you to make me aware so that We ‘ean move forward 1 the next question In order to get started, could you please begin by introducing yourself and talking about why you decided to get involved in the UNIV 101 course?Foc! 1: Bxpectations & Assumptions of the Tr ing/Preparation Program “To get ws started, the following questions wil focus on Your expectations and assumptions of the ining or preparation program. ‘© What wore your expectations for the taining program? ‘© Ideally, the interviewee will deseribe what they hoped to gain from the ‘raining and what they anticipated learning by participating in the raining program, Please use the following probes if necessary. ‘+ Probing, Questions ‘©. What id you expect to gain from the teaning/preparation program? ‘© Who did you expect to deliver the training/preparation program? Foci 2: Experience with the Training Preparation Program Process ‘Thank you! We would also like to learn more specifically about your experience withthe ining or preparation program. ‘© Please deseibe your experience in the trining/preparation program. Think about how it was structured and organized (© Encourage them fo share a particular story (© Ideally, the interviewee will describe what the trining/preparation program ‘was like, including ils structure and organization. Please use the following robes, iFnecessay, ‘+ Probing Questions ‘© What are your thoughts on the organization ofthe taining? '© How accessible was the information presented to you? (© How did you understand the program content? Foci 3: Experience of Delivering Course Content inthe Classroom, So far, weave focused on your expectations of and experience withthe training program, Now, lets dive ito your actual experience of teaching the UNIV 101 course ‘+ Please describe your experince delivering the various modules throughout the © Ideally, the interviewee will describe, in detail, what it was like to teach the ‘couse, including how thay felt prepared to teach, how they aetualy taught {stylefapproach), and how they felt supported. Please use the following probes, if necessary ‘+ Probing questions 6 What wa the role af the eo-insimctor a6 om tight the cere? {© What was easy for you in delivering course content? What was challenging? ‘© What different methods did you use to engage students inthe course? How did you fee! these methods wore received? (© Idoally, the interviewee will describe teaching methods and how they felt delivering the content. Please refer back to previously provided probing ‘questions for Foe 3, ifnecessary.Foci 4: Impact of Training on Performance and Overall Outcome Achievement [Now, let's return othe conversation about training, but fi like to focus on the impact of traning ‘on overall outcome achievement, ‘+ How did your experience wit the traning and preparation program impact the ‘ways in which you performed in the classroom (Le. knowledge of content, content delivery, sive of delivery, level of efficacy, et)? © Helly, the interviewee wall provide mormation about how knowledge and skills gained from taining impacted ther bility to teach the course. Utilize probes below, iPnevessary. + Probing questions ‘© What skis did you lear during training that you wilized throughout the ‘© What did you lean, in ters of course contenvinformation (ie. knowledge to be shared with students) in training tht you dd not know before training? (© How did taining iapact your ability to bean effective instrctor? + Can you provide an example of how your pedagogical practices created ‘opportunities for students to achieve the learning outcomes associated with UNIV 101? © Ideally, the interviewee will provide information shout how their preparation and teaching style enabled them to achieve the cours leaning outcomes Conclusion ‘You've shared a Jot with me today about your experience of teaching UNIV 101 this past, semester, Tam wondering i you have been able to refleston what could have made this experience different, In refleetion, what additonal topics would have been beneficial tobe trained on? Why? “Thinking back on how you were trained for, and how you taught, this course, what would you hhave done differently, if anything? Finally, in what ways, if any, could the taining/preparation program be improved to beter prepare instructors to teach this couse?Appendix NEvaluation Project Budget Item ‘Quantity Amount ‘Transcriber z $600.00 “Amazon Git Cards 10 $250.00 printing 1500 3100.00 $950.00 “Total ExpensesAppendix 0Evaluation Project Timeline ‘ugei5 | Sep-15 Nov-15 [OSI janci6 | Feb-16 Marie | Apeie Mey-16 re-Test Invitation Email, x Email Reminder Post Test Invitation Email Email Reminder Quantitative Data Analysis interview aviation Email Interviews ‘Transcription Data Analysis Presentation to Key StakeholderAppendix PUVa TeY aetuh eee aa gatas ee)