Running head: INSTRUCTIONAL EFFECTIVENESS MODULE

Instructional Effectiveness Module Using PowToon

Claudia Lpez-Gallagher and Nawara Nassan
California State University- Monterey Bay

IST622 Assessment and Evaluation

Professor Bude Su
June 10, 2016

INSTRUCTIONAL EFFECTIVENESS MODULE

TABLE OF CONTENTS
Page
I. Introduction.... 4
II. Methodology.... 5
a. Prototype ............ 5
b. Learners .................

c. Tryout Process .................... 6

1)

Pre- & Post-Test...

.......
6
2)
Observation ...
.....
6
3)
Questionnaire....
..........

d. Tryout Conditions .......... 7

III. Results ........ 7
a.
b.
c.
d.

Entry Conditions...........
Instruction ............
Outcomes .........
Recommendations ...........

7
8
8
10

IV. Summary......... 12
V. Appendices....... 14
A.
B.
C.
D.
E.
F.
G.

Invitation to Participate ........

Pre & Post -Test ....
Invitation to Continue with Part 2 of the Educational Module ......
Instructional Module Screenshot......
Usability Report Questions...............
The Usability Report Results ...
Grading Rubric... ..........

14
15
17
17
18
18
21

INSTRUCTIONAL EFFECTIVENESS MODULE

VI. References .............. 22

Introduction

INSTRUCTIONAL EFFECTIVENESS MODULE

PowToon is a software for creating animated presentations with entertaining characters,

objects, sounds, and effects. In the past, professionals presenting to an audience have been
restricted by specific software with limited capabilities. For those in need to produce a high
amount of presentations on a regular basis, there was a shortage of resources and creativity.
PowToon provides options, tools, and creative outlets to craft videos or presentations which will
keep the audience engaged.
The instructional module presented focuses on giving the learner reliable guidance on
how to use PowToon. It is prepared with interactive sections to maintain the learner involved and
captive throughout the lesson. Our goal was to observe a change in responses of participants that
would become knowledgeable and skilled to use this software in the future. According to
Margaret E. Gredler in Learning and Instruction: Theory into Practice: the evidence that
learning has occurred is an observed change in behavior, typically based on a comparison of preand post-instructional performance. (p.145). In her discussion of Ganges Concepts of
Capabilities, she argues that learning results in humans acquiring skills, knowledge, attitudes,
and values. This study will provide the scientific evidence to show an observed change in
participants skills and knowledge while creating videos and presentations by using PowToon.
We predict in this study that participants will score higher on the post-test after viewing our
instructional module on how to use PowToon. They will be able to acquire skills that will help
them create a video or presentation based on the steps presented in the instruction.

Methodology

INSTRUCTIONAL EFFECTIVENESS MODULE

Prototype
The lesson prototype that was chosen for this study is a screen capture designed to show
the steps from start to end on how to create a simple PowToon video or presentation. When we
decided to use PowToon as our mode of instruction, we recognized that the video alone would
not suffice. For this reason, we chose Zaption. Zaption is a video-based program that provides
tools and interactions that can be added to any video. It provides comprehension checks
throughout the video itself that allow for a more engaging instructional experience for the
learner. For example, learners have to answer multiple choice questions about signing in options
into PowToon, choose a true or false statement describing a function of PowToon, or click on all
related answers to a question linked to functionality in the program. Consequently, with this tool
we were able to create a better instructional tool to address our objectives in a direct manner.
Learners
The target audience for this project consisted of participants with different professional
backgrounds. We chose individuals who possessed the necessary factors for learning (Stolovitch
and Keeps, 2011, p.40). In their book: Telling Aint Training Stolovitch and Keeps presented
key ingredients for learning: ability, prior knowledge, and motivation. Participants in this study
had the ability to learn, prior knowledge of using technology (computers, emails, presentations),
and motivation to help in our study and learn a new skill. They were familiar with creating noninteractive presentations and did not need any prerequisite skills to learn PowToon. We
recognized that learning how to create an interactive presentation with animation can be a useful
tool at their work. Participants will acquire the necessary skills needed by watching the
instructional module provided.
Tryout Process
Pre and Post-Test. Our initial contact with participants was an invitation sent by email
or Facebook Messenger that included details of our project (Appendix A). Individuals that

INSTRUCTIONAL EFFECTIVENESS MODULE

expressed interest in being part of the activity were then contacted with a link to the pre-test
(Appendix B). Both pre- and post- tests were created using Google Forms. Our invitation was
easy to understand, and it listed a short explanation and the steps required by participants. We
divided the steps to part one, the pre-test. Once the pre-test was completed, part two was sent
(Appendix C). Part two contained the remainder of the links. It included the links to the
instructional module, the post-test, and the usability test (Appendix E). Additionally, the steps
provided approximate time participants would need to complete each step. This process was
intended to help them determine if they can invest the time to complete the process.
Observation. Before presenting the prototype to our target audience, we ran the initial
product sample by a younger audience where we were able to observe in person the reactions of
the participants and determine the ease of use. Based on this unstructured observation, we made
a few adjustments to the prototype. For example, we added more interactive sections in the video
to avoid long segments with no reciprocal action. Additionally, we replaced some notes with
questions and in some cases, we changed the type of questions. Our observation was to collect
some exploratory data for improvements on the product. The subjects were not aware of the
observers task; the environment was kept natural and informal.
Questionnaire. The questionnaire was also created using Google Forms. The questions
were based on a five-point Likert Scale. Participants were asked various questions about their
experience through the instructional module and their experience as a whole. When creating the
questionnaires, we took into consideration the guidelines and rules discussed in Evaluation in
Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change by
Russ-Eft, D., & Preskill, H. (2009) to ensure quality and clarity of survey. For example, we
avoided double-barreled questions, leading and loaded questions, and negatives. We made sure
our questions had a single purpose and were simple, clear, and short.
Tryout Conditions

INSTRUCTIONAL EFFECTIVENESS MODULE

We contacted participants by email and Facebook Messenger. The tryout conditions were
expected to be favoring a complete response in a short amount of time. Because of logistical
reasons we decided to complete our project remotely. We also viewed it as a positive to have
participants complete the task in the comfort of their home or environment of their choice. It
would also allow flexibility in terms of time and schedule.
Results
Entry Conditions
Participants taking the test have no prior experience using PowToon. All 11 users are
computer literate and can access the internet. The initial invitation to participate was sent via
Facebook Messenger or personal email. The link included an invitation to participate
(Appendix--). Initially, the request was sent to approximately 20 users. Participants reported
difficulties with viewing the instructional module on their SmartPhone. A specific Zaption
application would have to be downloaded to view our project on a phone versus a desktop/laptop
computer. There were no other unusual entry conditions observed.

Instruction
This module was designed to be completed remotely. It is an interactive and self-paced
instruction designed to be used by participants at a place of their choice and convenient time. The
only possible comment that can be made on observed behavior would be in our practice trial with
the younger audience. We anticipated the need for interactivity and discovered more was

INSTRUCTIONAL EFFECTIVENESS MODULE

required after these observations. One participant had to be dropped because she completed the
pre-test but was unable to complete the rest of the project, which brought our participant number
from 12 to 11.
According to the usability report (Appendix F), 10 out of 11 users agreed that the
instruction was clear. 9 out of 11 agreed that the lesson informational structure is appealing. 10
out of 11 agreed that they will use PowToon as a tool if needed in the future.
Outcomes
The analysis of data retrieved from pre- and post-tests indicated that participants reached
a significant learning outcome as a result of viewing the module on How to learn PowToon.
This data concurs with our initial prediction of possible outcomes. Both pre- and post-tests were
based on dependent samples as they represent an example of a repeated measure. To
demonstrate: we conducted a pre-test before participants viewing the instructional module. Then
we conducted a post-test. Therefore, we chose the t-test: paired two sample for means option.
The null hypothesis is a directional hypothesis: H0: 1<=2, which also means that the research
hypothesis is directional as well 1<2. Based on the numbers from our research data, we are
comparing the absolute value of the t-Stat with one-tailed critical value. (Table1) The t-Stat result
is bigger than the critical value (t=-5.78 > 1.81). Therefore, we reject the null hypothesis and
support the research hypothesis. Thus confirming participants performed better on the post-test
after viewing the instructional module on How to use PowToon.
Information collected using the pre-test provided additional information on the
participants. There were ten females and one male that completed all the parts of the module.
Our sample was not a random sample of contributors; it was a convenient sample due to time and
space constraints.

INSTRUCTIONAL EFFECTIVENESS MODULE

Table 1
t-Test: Paired Two Sample for Means

Pre-Test

Post-Test

Mean
Variance
Observations

14
4
11

Pearson Correlation

0.2494337479

Hypothesized Mean
Difference

df

18.27272727
4.018181818
11

10

t Stat
P(T<=t) one-tail

-5.776554342
0.00008928603116

t Critical one-tail
P(T<=t) two-tail

1.812461123
0.0001785720623

t Critical two-tail

2.228138852

Table 2
Descriptive Statistics

Alpha (for confidence interval)

.5%
Pre-test Results

Post-test Results

Count

11

11

Mean

14.

18.2727272727

4.

4.0181818182

Variance

INSTRUCTIONAL EFFECTIVENESS MODULE

10

Standard Deviation

2.

2.004540301

0.1428571429

0.1097012105

Minimum

9.

14.

Maximum

16.

20.

Range

7.

6.

Median

15.

19.

Percentile 25% (Q1)

13.5

18.

Percentile 75% (Q3)

15.

19.5

Sum

154.

201.

Mode

15.

19.

Coefficient of Variation

Recommendations
Based on the pre- and post-tests, instructional module, and its usability participants gave the
following feedback as short answers. One participant emailed additional comments that are included
in this list.
Pre-set question allows true and false to both be selected (checkboxes are used instead of radio

button)
Timeline question does need to be re-worded to indicate shortening/lengthening time, moving the
start or end time within the slide. As it appears now, the question seems to imply the text box, not
the display time.
Animation question allows both true and false to be selected (checkboxes are used instead of
radio button)
Edit recording allows yes and no to both be selected (checkboxes are used instead of radio
button)
Adjust timing allows true, false, Im not sure to all be selected
The use of radio buttons was to offer variety throughout the module. Zaption allowed for
multiple answers to be chosen, and this could have been offered as a possibility to the

INSTRUCTIONAL EFFECTIVENESS MODULE

11

participants. Improvements to these questions included clarifying the options when answering
the question.
Scrubbing question does not provide feedback response: correct or
Incorrect.
"My PowToons" folder question states pick one or more, but uses radio buttons so only one can
be selected. It also does not provide feedback.
I seemed to get all my responses correct as I went through the presentation, but my grade at the
end was 0 of 10.
Steps when designing Powtoon projects include: All apply, but I can only select 1 (radio button
rather than checkbox) - or missing "All of the above" option.
These comments were directly related to the usability of our instructional module. Being
that we used Zaption to present the instruction online, it is possible that some functions may not
have worked properly during the participant's experience. Regardless of experience, these
comments are valuable for future implementation of the module.
A Timeline is: question, the option #2 should read: The section in the project (bar) where you
can manage the text display duration and when the time at which it will appear within the
slideshow.
Scrubbing is quickly viewing the progress of a slide by moving the playhead manually. This
should be reworded to include moving the playhead forward or backward within the slide.
Needs an attention grabber and screen was too small which may lose some interest.
I think the breaks for questioning were important but perhaps too many. So it made it very
difficult to pay attention to the graphics and also the content. The questions made it difficult to
absorb the information or focus on items of personal interest.
I wish it was a little bit slower.
Sending via email (instead of using Facebook Messenger)
This section of the comments by participants was related specifically to their overall
experience. Zaption allows for the video to be full screen and to slow down or speed up. Two of

INSTRUCTIONAL EFFECTIVENESS MODULE

12

the above comments are related to these two options in Zaption. These comments allow us to
conclude that the navigation bar may have needed prior training before the starting of the
module.
Summary
The instructional module used in this study was intended to provide participants new
skills to benefit them in the future when creating videos or presentations. The studys purpose
was to support our hypothesis that participants will score better on their post-test after watching
the instructional module. Based on the pre-test and post-test, analysis of data, and survey
answers it was established that our assumptions were correct. Participants acquired new skills
and demonstrated a change in behavior after viewing the instruction on How to Create a
PowToon. It was also noted based on the usability report, that 10 out of 11 participants agreed
that the lesson was enjoyable.
Furthermore, we observed that, for one participant, the interactive sections were too
numerous. Another participant felt that the screen was too small which may have caused loss of
interest. Taking into consideration all suggestions and recommendations to improve for future
projects, we recognize that learning is a change in behavior. The data of our analysis confirmed a
change in participants, which is what we set out to achieve by creating the instructional module.
The process of finding participants, creating tests, formulating the data and analyzing it was an
educational process. Moreover, we recognized that it is a very critical process as it needs to
provide valid and usable data for accurate results.

INSTRUCTIONAL EFFECTIVENESS MODULE

13

Appendix A
Invitation to Participate
Hello friends,
We need your help!
My class partner and I need users to help us create an evaluation report for our current class.
Your help is appreciated. This is all done online. You will learn how to use a software to create
an animated video or presentation. You were specifically picked because of your professional
and educational backgrounds and we believe you will enjoy this program and the new
knowledge.
Your part consists of the following steps:
1. Respond to demographics questions and take a pre-test about a software we will present to you
(5-8mins)
2. Watch the instructional video (10:54mins)
3. Take a post-test (same questions as pre-test) (5mins)
4. Respond to a usability report. (5 mins)
This is all simple, quick, and will greatly help us finish our project and hopefully get a good
grade. We will send you the information in steps. Once you complete each, we will send you the
next. We have only till Tuesday to complete this portion (testing).
Please respond via email to confirm your willingness to participate. You can reach us via email
or respond to Facebook message.
Nawara : nnassan@csumb.edu
Claudia: clopez-gallagher@csumb.edu
Thank you for your time.

INSTRUCTIONAL EFFECTIVENESS MODULE

14

Nawara & Claudia

Appendix B
Pre and Post-test
Please enter name
Please enter your email
Gender

I can choose different characters from the

Sound section
True
False

What is PowToon?
A software to create music
A software to create animated presentations
and videos
A software to create PowerPoint presentations
Im not sure

The maximum amount of each slide in

PowToon is:
45 seconds
1 minute
20 seconds
25 seconds

How can you start a PowToon account?

Using a Facebook account
Using a Google account
Using LinkedIn
All of the above
None of the above

Scrubbing is quickly viewing the progress

of a slide by moving the playhead manually
True
False

Powtoon has preset templates to create a

project?
True
False

One of the options for Text Effects is

handwriting effect which is a handwriting text on the screen
True
False

My Styles section includes different

backgrounds
True
False

When designing PowToon projects, you

can:
Add animation
Choose items and drag to slide
Assign entrance and exit animation to

INSTRUCTIONAL EFFECTIVENESS MODULE

15

different items
Adjust timing of items on slide by using the
timeline
All of the above
I can choose different characters from the
Sound section
True
False

Starting with a good plan (storyboard,

script, or outline) is recommended with a
PowToon project.
True
False

It is recommended to record narration

first, then to manage items on the timeline
True
False

To check your progress, you can preview

your scenes by clicking:
Preview button
Auto-save
Sound button
Images button

You can edit an audio recording in

PowToons
True
False

To upload your project to YouTube click

on:
Sound button
Images button
Export button
Auto-save

It is a good idea to save your project often

because PowToon does not auto-save
your work
True
False

Where can you find your PowToon

projects?
My Styles tab
My PowToons tab
My Images tab
My Characters tab

Adding images to a project can be done by:

Adding personal images
Using images from PowToon
Uploading images from the internet
All of the above

You can use the actions button to:

Play
Duplicate
Delete
Export to video or get embed code
All of the above

Adding music to a project can be done by:

Adding personal music files
Using music from PowToon
Uploading music from the internet
All of the above

INSTRUCTIONAL EFFECTIVENESS MODULE

16

Appendix C
Invitation to Continue with Part 2 of the Educational Module
Thank you for completing the Pre-Test. Now part 2 consists of 3
things.
1. Please watch this video:
https://www.zaption.com/lessons/5748bef783451ede381e4fa3 [1]
2. Please complete the Post-Test (same questions as the Pre-Test):
https://docs.google.com/a/csumb.edu/forms/d/1ZdrCpIyEJMX0bfzc1kd_F3KTyzU0DUQZEy720ljVBs/edit?usp=sharing
3. Please answer a quick final questionnaire:
https://drive.google.com/open?id=1iBAE3rYumuZ9x5BdxQRrIUnqzfrut2hw--- ftqVj6Tw[2]
Thank you so much for your time!

Appendix D
Instructional Module Screenshot

INSTRUCTIONAL EFFECTIVENESS MODULE

17

Appendix E
Usability Report Questions
Instructions in the presentation are clear

Navigation through instruction is clear

Navigation through the instruction is intuitive

to follow

Lesson information structure is appealing

Graphics are informative

Audio narration adds to learning experience

Audio quality is clear

You will use this educational tool if/when

needed
Appendix F
Usability Report Results

INSTRUCTIONAL EFFECTIVENESS MODULE

18

INSTRUCTIONAL EFFECTIVENESS MODULE

19

INSTRUCTIONAL EFFECTIVENESS MODULE

20

INSTRUCTIONAL EFFECTIVENESS MODULE

21

Appendix G
Grading Rubric
Category

Score

20 Pre-Test Questions

20

20 Post-Test Questions

20

Total

40

INSTRUCTIONAL EFFECTIVENESS MODULE

22

References
Gredler, M. (2009). Learning and Instruction: Theory into Practice. Upper Saddle River, New
Jersey: Pearson Education, Inc.
Kirkpatrick, D. L. & Kirkpatrick, J. D. (1994). Evaluating training programs: The four levels.
San Francisco, CA: Berrett-Koehler.
Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations a systematic approach: a
systematic approach to enhancing learning, performance, and change. New York, NY:
Basic Books.
Stolovitch, H. D. & Keeps, E. J. (2011). Telling aint training (2nd ed.). East Peoria, IL: Versa
Press, Inc.