Running Head: MODULE EVALUATION 1

Module Evaluation: CST 361 Overview

Cassandra Humphrey

California State University, Monterey Bay

IST 622: Evaluation and Assessment

Dr. Bude Su

June 12, 2018

MODULE EVALUATION 2

Table of Contents
I. Introduction ............................................................................................................................ 3
II. Methodology.......................................................................................................................... 3
Prototype ........................................................................................................................................... 3
Learners ............................................................................................................................................. 4
Tryout Conditions ............................................................................................................................... 5
Process ............................................................................................................................................... 5
Pre and Post Tests ......................................................................................................................................... 5
Observation .................................................................................................................................................. 6
Feedback....................................................................................................................................................... 7

III. Results ................................................................................................................................... 7

Entry Conditions ................................................................................................................................. 7
Instruction .......................................................................................................................................... 8
Outcomes ........................................................................................................................................... 8
Recommendations ........................................................................................................................... 10

IV. Summary............................................................................................................................. 11
V. Appendices........................................................................................................................... 12
A. Demographics Survey................................................................................................................... 12

B. Observation Checklist .................................................................................................................. 12

C. Pre-Test ........................................................................................................................................ 13
D. Post-Test...................................................................................................................................... 14
E. Pre and Post Test Rubrics ............................................................................................................. 14
F. Pre and Post Test Results .............................................................................................................. 15

G. t-Test: Paired Two Sample for Means .......................................................................................... 15

H. Feedback Survey .......................................................................................................................... 16
I. Archival Data ................................................................................................................................. 17
MODULE EVALUATION 3

I. Introduction
In the CS Online program, students are subjected to a streamlined, fast-paced course load.

Many times, their classes overlap, and students may find themselves taking a final for one class

and attending an orientation for another within the same weekend. However, a couple of courses

require more drawn-out preparation. CST 361 is the first of these courses, and students are

expected to begin preparation about two months before the course begins. Due to the workload in

their current classes, many students find it difficult to attend synchronous information sessions so

far in advance, and view counts of recorded sessions have been consistently low. To combat this,

an asynchronous module was created in the hopes that students would be able to get background

information about the course and prepare independently.

This test aims to determine whether the prototype is functional and usable, while

investigating knowledge gain, user reactions, and areas for improvement in terms of design,

delivery, and content.

II. Methodology

Prototype

The prototype used for usability and effectiveness testing was an

informational module created with Adobe Captivate. The module, titled CST 361 Overview,

introduced the users to the information required to begin preparation for the computer science-

based Service Learning course. It reviewed background information such as why the course is

required, how it is unique, and what steps future students will need to take to prepare before the

course begins.

While the module is fairly locked down in terms of navigation, users are required to

perform interactive tasks such as clicking buttons and answering quiz questions via several
MODULE EVALUATION 4

formats such as multiple choice, multiple selection, short answer, and drag and drop. An average

learner is expected to spend around seven minutes on the module but may take as long as ten

minutes if they replay or review slides.

If the module proves to be an effective instruction tool, this prototype may be used as a

supplement to face-to-face preparation information sessions for future CST 361 courses.

Learners

This module is designed for CS Online students who are about halfway through the

program. According to archival demographic data of the first three CS Online cohorts collected

by Claudia Carter, a majority of CS Online students are male, with the percentage ranging from

69% to as high as 88% depending on the cohort. Most students fall in the 26-35 age range, with

only one cohort having a majority of 18 to 25-year-old students, and 70.6% of students reported

working 40 hours per week.

In the hopes of providing the most realistic testing results, the thirty-three students of

Cohort 6, who are scheduled to take the course in the spring of 2019, were invited to participate

in the module usability and effectiveness testing. However, only three students volunteered, so

the breadth of test subjects expanded to include two fully online MIST students and one student

from another college for a total of five test subjects. In order to keep the subjects as close to the

CS Online demographics as possible, they needed to have taken at least one online college class,

fall between the ages of 18 and 55, and have had minimal previous exposure to information

about CST 361.

Three respondents were male and two were female, 100% of testers had taken at least one

fully online college course in the past year, 60% were between the ages of 25-34, and 60%

worked thirty or more hours per week (See Appendix A).

MODULE EVALUATION 5

Tryout Conditions

The tryout was conducted in the same format for each tester. In order to reflect CS Online

students’ environments as closely as possible, the testing was recorded via Zoom while the

volunteers completed the module in their homes on their personal computers. The participants

used a range of operating systems and browsers. Most used Windows computers, and Chrome

was the most popular browser, but Safari and Firefox were used by one student each. Users were

asked to use which browser they were most comfortable with to alleviate any stress or issues that

may have arisen from using a less familiar browser environment.

Although the volunteers were in different environments: at home alone, walking in the

door from work, or taking care of a young child, this is fairly accurate of CS Online student

experiences. Everyone was able to successfully complete the module, even with the varying

contexts that are common for CS Online students.

Each volunteer was given a link to the Google Form Demographic Survey and the

module, which was hosted on the developer’s ITCDland account. The developer created two

more Google Forms to record responses for the pretest questions and final feedback. However,

these questions were asked verbally and transcribed by the developer as users answered in an

attempt to get unfiltered responses.

Process

Pre and Post Tests

The pre and post tests were conducted in two different formats. Although initially more

difficult to quantify and compare, this was done deliberately to prevent students’ posttest scores

from being unintentionally affected. The pretest was open-ended and was conducted verbally in

an interview style. Users did not have access to the questions beforehand to discourage prior
MODULE EVALUATION 6

research. This way, knowledge of the main topics of the course were reviewed without responses

being swayed by guessing, or by preemptively tipping students off to specific areas of interest in

the model. The posttest was conducted as an end of the module quiz, where students were

required to independently answer multiple choice, multiple selection, short answer, and drag and

drop questions relating to the material.

The pretest responses were graded and weighted based on keywords that aligned with the

posttest answers. Posttest responses were weighted based on the number of necessary responses

required to answer the question correctly. The number of responses mirrored the number of

keywords in the pretest. Care was taken to make the grading criteria as closely aligned as

possible. Each test was out of thirteen total points. All questions for a particular topic received

the same amount of points on both tests. For example, students were questioned on the hours

requirement in both tests and received one point if they were able to respond with the correct

number. See Appendices C, D, and E for a full report of questions, responses, and rubrics.

Observation

The usability and effectiveness test was conducted fully online and synchronously on

Zoom.us in meeting rooms created by the developer. Each test occurred as a one-on-one session

between May 21 to June 5, and all tests occurred on weekdays after 5pm. The total time for each

test ranged from fifteen to thirty minutes, and consisted of a pretest, demographics survey,

module test with embedded posttest quiz, and a feedback survey. Each session was recorded and

posted to the developer’s Zoom cloud account, with copies added to Google Drive as backup.

In each session, the volunteer was given a short brief on the purpose of the module and the

components of the evening’s test. The developer administered the pretest and then shared the

demographic survey link with the user for them to fill out while the developer prepared the
MODULE EVALUATION 7

module. Typically, demographics information comes first, but in this situation, the developer

knew the participants personally and knew they fit the target characteristics before beginning.

Demographics in this case were used for analysis and grouping after the test, not for screening

participants.

After the initial forms were completed, the developer shared the link to the learning

module with the user and confirmed what operating system and browser was being used. The

user shared his or her screen with the developer during this test, and the observation checklist

was filled out for each session (see Appendix B). The developer wanted to observe how students

interacted with buttons, navigation, and other components of the module, so they were not given

any specific directions on how to progress through. One student questioned how to move

forward from the introduction slide but was able to figure it out herself without intervention. The

only task that was formally requested of the tester was turning on the closed captions to ensure

that the accessibility feature worked on different browsers and operating systems.

Feedback

When the module was completed, the user was permitted to stop sharing their screen, and

the developer transcribed the users’ spoken responses to the feedback survey (see Appendix H).

This was a time for participants to give final thoughts on their experiences with completing the

module. They were able to ask additional questions or give feedback on the module’s design,

content, or other components.

After all data was collected and the testing was finished, the developer thanked the tester

for attending, and the Zoom session was ended.

III. Results

Entry Conditions
MODULE EVALUATION 8

The predicted and observed entry conditions were consistent. All of the testers had

minimal knowledge of the course, as outlined in Appendix C. Each one had little to no

knowledge of the specifics of the course, although some were able to summarize the general idea

of helping a nonprofit or similar entity. All users completed the module within the predicted

timeframe, with one tester who had no experience with the course taking the full ten minutes, as

she chose to review before the quiz. All others took around seven or eight minutes to complete

the module.

Instruction

Overall, the course instruction worked as intended. All users were able to navigate

through the course, review previous slides if they chose to, all pages loaded properly on all

browsers, and all interactive elements functioned as intended.

The only discrepancy between intended and observed usage was with an ungraded

“Check Your Knowledge” question. This multiple-answer question appears right after a slide

that outlines the three main requirements for a service learning project. The directions ask

students to select the three requirements and check their answer before moving on. Four of the

five students chose three options, but one student only chose one. Adding in validation that

prevents users from moving forward unless they choose three requirements should prevent this

from happening in future versions.

Outcomes

Based on analyses of pre and post test scores (see Appendix F), it can be concluded that

the module did contribute to users’ knowledge. Their pre and post test scores were calculated

based on their respective rubrics (see Appendix E), and then this data was used to calculate a t-

Test: Paired Two Sample for Means summary (see Appendix G) to illustrate the effect the
MODULE EVALUATION 9

learning module had on students’ knowledge of CST 361. The absolute value of the t-Stat,

t(4)=26.5, is well above the critical value of 2.132 (.05 alpha level, df=4).

Although great care was taken to keep the pre and post test questions and rubrics as

similar as possible without unintentionally giving away answers or highlighting upcoming topics

before the users had a chance to do the module, the difference in formatting between the pre and

posttest may have impacted results.

At such a small scale, however, it is important to note potential for bias in results before

generalizing to project CS Online cohort results. As shown in Appendix A, all users that

participated in the module rated themselves neutral, interested, or very interested in the subject

matter before testing. In the feedback survey, all respondents reported being either interested or

very interested (Appendix H). This positive attitude towards the course or content may have led

to higher scores as well. In terms of enjoyment and usability, all students found the course easy

to navigate, all reported that they felt ready to begin preparation for CST 361, and one user was

very satisfied with the overall look and feel of the module, stating, “I thought it was very clear,

not over-complicated visually but still pleasing to look at. I liked the color scheme, graphics, and

animation” (see Appendix H).

However, it is not expected that all students will enjoy this module. As observed from

past run CST 361 courses, the topics and requirements of the class can be controversial to

students based on their personal backgrounds and prior experiences. Therefore, the goal is to

keep the module objective, providing background, requirement, and preparation information in a

straightforward and unbiased manner that can be completed quickly. While it is hoped that the

module is visually pleasing and interesting to students, the main goal is for it to deliver

information to users with a wide range of attitudes in a functional way.

MODULE EVALUATION 10

Recommendations

Based on developer observations and user feedback, there are three main

recommendations for future improvements.

First, updated versions should provide clearer navigation controls. Currently, the

narration verbally (or via closed captions) instructs users on how to move forward on the first

slide. An arrow appears and points to the next button as well, but several users requested more or

clearer instruction on navigation, while others would have liked the slides to progress forward

automatically.

One student’s hidden toolbar would appear and cover the navigation buttons when they

moved their mouse down to click the next arrow, so providing alternative ways to navigate such

as integrating arrow key commands or allowing for automatic progression may make this

component more intuitive.

Second, updated versions should replace the computer-generated narration with a human

voice. One tester stated that the current voice over made the module more challenging to

complete, saying, “As I was listening to it, I was kind of deciphering what they're saying. It made

it a little hard to follow in some parts” (see Appendix H).

Finally, more information on community partners will be included in future versions.

After the module, students had questions regarding course timing and appropriate projects. While

the module is intended to be sent to students with a message explaining an overview of cohort-

specific dates, more information on types of projects and partner communication in the module

may help clarify the common questions that arose during testing. Currently, users are not

exposed to a project scenario until the quiz, although they are given the requirements during the

module. Adding in summaries of past projects may make requirements clearer.

MODULE EVALUATION 11

IV. Summary

As the analysis, observations, and student scores show, the learning module proved to be

functional and effective in introducing users to CST 361’s unique requirements. Requests for

future improvements were minor and doable, and the test shows that users will be able to

successfully review the module even with minimal knowledge about the course.

It is recommended that testing is repeated with new users after the module is updated,

preferably with individuals with lower interest levels to eliminate potential bias.

However, based on the data collected here, this module could potentially be completed by

students well before they need to begin preparation for CST 361, ensuring a smooth transition

into the course. Because some students still had specific or personal questions after reviewing the

module, it is recommended that the faculty still holds a face-to-face session before the class

begins to answer more in-depth and unique questions.

MODULE EVALUATION 12

V. Appendices

A. Demographics Survey

1. Have you completed at least one

fully online college course?

2. What is your age?

3. How many hours do you work per week?

4. What is your current level of interest in CST

361?

B. Observation Checklist

Checklist Yes No Comments

Did the module function on the user’s stated

operating system and browser?
Was the user able to hear the audio?

Was the user able to view the captions?

MODULE EVALUATION 13

Was the user able to navigate through the module

as it was intended?
Did the user get stuck on any section or page?

Did the certificate display at the end of the module

on the user’s stated operating system and browser?

C. Pre-Test

1. What is the goal of CST 361?

• This is the one with tutoring in the name, right? It is a real hands-on real-world
experience with a company.
• It’s Service Learning. I don’t know anything else.
• I don’t know too much about it. I believe it’s the community service one. Besides that, I
don’t know too much. It’s meant to be for an organization or a nonprofit that works for
free. It’s supposed to be a display of what we’ve learned so far.
• I don’t know.
• I don’t know.

2. Do you know anything about service hours requirements?

• Not yet.
• No, not yet.
• No, I don’t know anything about it yet.
• No.
• No.

3. What do already you know about Service Learning projects?

• It involves working with groups like nonprofits, schools, the YMCA, hospitals, or
businesses.
• I’m probably going to work with somebody who helps the community with technology.
• Absolutely nothing.
• Nothing.
• I remember hearing about this class before, but I don’t know what I’ll be doing.

4. Have you heard about any course requirements, specifically required documents?
• I don’t know any specifics, but my impression is that it is a big project.
• No, I haven’t.
• It’s a project that has to be done for a nonprofit. It has to be a free project. But besides
that, I haven’t heard anything.
• No.
• No.

5. Do you know of any steps you will need to take to prepare before the course starts?
• No. Potentially ideas of how to help a company locally.
MODULE EVALUATION 14

• I haven’t been told definitively.

• Just kind of preparing and thinking about where I want to do my project in terms of area
around me so I have somewhere to go and offer my services.
• No,
• No.

D. Post-Test

1. In 361, you will? (Multiple Choice)

• Use your skills in computer literacy, multimedia design, and technology to assist
nonprofit organizations and community agencies in “Bridging the Digital Divide.”
• Work in middle or high schools to help students deepen their understanding of
technology principles, techniques, and methodologies. You will also study how issues of
social justice and equity affect mathematical literacy and why it is important for everyone
to be mathematically literate.
• Examine students' and communities' health, particularly as it relates to service and social
responsibility, community and social justice and multicultural community building and
civic engagement.

2. You must complete [] hours at your Service Learning site (Fill in The Blank)
30

3. Required Forms (Multi-Answer Multiple Choice)

• Learning Agreement
• Activity & Time Log
• Partner Evaluation
• Student Evaluation

4. Is this partner appropriate? #1 (True/False)

No

5. Is this partner appropriate #2 (True/False)

Yes

6. Steps to Prepare (Drag & Drop List)

• Search for community partners in your area
• Contact 2-3 organizations asking if they are in need of help with technology
• Choose which organization and project will be the best fit for you
• Contact the instructors for organization and project approval

E. Pre and Post Test Rubrics

Pre-Test Rubric
Question
# 1 2 3 4 5
MODULE EVALUATION 15

Community Nonprofit or Learning

Service Number similar Agreement Search
Keywords Tutors (30) CS Related Time Log Choose
Partner
Interactions Evaluation Contact
Student
Evaluation Approve
Points 1 1 3 4 4

Post-Test Rubric
Question 1 2 3 4 5 6
Points 1 1 4 1.5 1.5 4

F. Pre and Post Test Results

G. t-Test: Paired Two Sample for Means

t-Test: Paired Two Sample for Means

Pre-Score Post Score

Mean 1.2 11.8
Variance 1.7 0.7
Observations 5 5
Pearson Correlation 0.733358798
Hypothesized Mean Difference 0
df 4
MODULE EVALUATION 16

t Stat -26.5
P(T<=t) one-tail 6.02596E-06
t Critical one-tail 2.131846786
P(T<=t) two-tail 1.20519E-05
t Critical two-tail 2.776445105

H. Feedback Survey

1. Do you feel prepared for CST 361?

2. Was the module easy to navigate?

3. What is your interest level now?

4. Is there anything you’re still unsure of in terms of requirements?

• For one of them, are there formats that you use when you contact agencies that you
would recommend? Just kind of reaching out? One other one. I know we had gone over
projects a little bit, but hearing some other example projects would be nice, too.
• I think the main question that I would have in this module is: Is this going to be a similar
timing as the current classes we have where the week before the current course ends, the
new course module opens? I feel like due to the course's setup, I feel like this module
should be released a class early to give people what's going to be expected two classes
from them, so they can take the time to look up the partners and have that prework done.
Also, is this a solo project?
• No, I don't think so, but will this be available for me to look back at later? Will I be able
to go back and look at it?
• No, it was very straightforward.
• No.

5. Do you have any final comments?

MODULE EVALUATION 17

• I thought the formatting was good with asking questions right after the material, that was
really nice. And then the interactivity, being able to toy around with the questions was a
nice addition.
• I mean, I don't see anything that can really be improved. It's pretty well explained in the
slides and everything. My main thing is just scheduling when this will be released in
correspondence to the course.
• Sometimes I forgot that I had to push the next button to move on. I wish it continued
automatically.
• Overall, it was perfect. At the beginning of the module, I wasn't sure how to move
forward.
• The part about the arrow was a little confusing. Maybe be more clear on how to progress.
I thought it was very clear, not over-complicated visually but still pleasing to look at. I
liked the color scheme, graphics, and animation. I like a human voice. As I was listening
to it, I was kind of deciphering what they're saying. It made it a little hard to follow in
some parts.

I. Archival Data