IMA 2 - Usability Testing Report

1

IMA 2 - Usability Testing Report

www.khanacademy.org



Group 1
Abeer Mohammad, Cayla Repass & Pelagia Mazarakis








June 9th 2014








IMA 2 - Usability Testing Report

2
I Introduction
Khan Academy is a non profit educational website started in 2006 by Salman Khan.
Khans mission is similar to a MOOC (Massive Open Online Course); to provide a free
education for anyone anywhere. Khanacademy.org features thousands of educational resources
including excersize problems, micro lectures, and video tutorials on mathematics, science
history, health, economics, astrology, ecology, medicine, finance, and civics- to name a few.
Khan Academys resources are utilized by teachers and students from preschool to college.
In order to test the usability of Khan Academys website, our team conducted a usability
test following three of Nilsens (2003) five usability categories: learnability, errors and
satisfaction. On three separate occasions, we examined three separate school-aged students as
they navigated through five different tasks on the site. We collected data from task observations
and online surveys. The task observations were conducted on three separate computers in three
separate settings. One was observed in a private home, one was done via Skype, and one was
observed in a classroom setting. Based on the findings, the overall website usability was
evaluated and assessed.
II Methodology
A. Sessions
EME 6208 IMA-2 Group 1 gathered usability testing participants in various manners.
One was a fourth grade male student visiting one of the team members home, one was a fifth
grade female student from a researchers class, and one was a female student who is a relative of

IMA 2 - Usability Testing Report

3
one of the researchers. Each individual session lasted approximately 20 to 25 minutes. For each
usability test, the team member read a script and thanked the participant for helping in testing out
the website. Participants were told this was not a test and to not worry about making mistakes.
In addition, the participants were encouraged to share their thoughts aloud as they searched
through the new website following the prompts of each administrator.
B. Participants
Spyros is a fourth grade student at Sunset Hills Elementary School; Alexa is a fifth grade
student at Pasco Elementary School; Rawah is a seventh grade student at Beecher School, New
Haven. The tests were conducted during the week of June 2nd, 2014. There were two female
participants and one male participant.
C. Evaluation Tasks
The participants were timed as they navigated through the website and attempted to complete
five tasks (see attachment B -Observation Form).
1. Students click on their appropriate grade level under the math link.
2. Student is able to complete the Math Pretest activity.
3. Student is able to follow the directions for submitting their answer for each question asked.
4. After completing the Math pretest, the student is able to start the practice skills.
5. Student is able to click on Id like a hint or Stuck? Watch a Movie to help guide them in
answering the question.

IMA 2 - Usability Testing Report

4
During the task completion, notes were taken with specific attention given to the participants
errors and emotional reactions while completing the tasks. After the tasks were completed, each
administrator had each participant rate Khan Academys website by using an 8 item
questionnaire (see attachment C - Post Satisfaction Survey). The questionnaire surveyed the ease
of use, navigation and learnability of the website. The observers also closely watched how each
participant reacted to the website during each task, noting both positive and negative reactions.
III Results
A. Task Completion Success Rate
All three participants completed the five tasks successfully. The first task required the
participants to click on the appropriate grade level under the math link. All the participants,
regardless of their grade level, completed the task with ease and within a few seconds. As for the
second and third tasks, though the engagement time in the activity (i.e. Task 2) varied among the
participants, they all were able to complete the two tasks, Math Pretest activity and following the
directions for submitting their answers, successfully. In completing Task 3, both Spyros and
Alexa who are at 4th and 5th grade, respectively, experienced some confusion moments on how
to proceed after completing the Math pretest. However, both participants sided with Rawah, who
is at the 7th grade, and were able to accomplish the task with ease. Finally, the three participants
successfully completed Task 5 in which they had to click on Id like a hint or Stuck? Watch a
movie help icons.

Task Completion Rates

IMA 2 - Usability Testing Report

5
Participant Task 1 Task 2 Task 3 Task 4 Task 5
Spyros
Alexa
Rawah
Success 3 3 3 3 3
Completion
Rates
100% 100% 100% 100% 100%

B. Learnability: Time taken to successfully complete the task
Task 2 required the participants to complete the Math Pretest activity, which required the
most time (mean = 135 seconds). For most of the participants, the Math Pretest took a substantial
amount of time to complete. Task 3 was involved with Task 2, which required the participant to
follow the directions for submitting their answer for each question asked during the pretest. The
next task that required more time was Task 5 (mean = 11.7 seconds), which stated that the
student is able to click on Id like a hint or Stuck? Watch a movie to help guide them in
answering the questions with the practice skills. Some participants had trouble locating the hint
link or the movie link to help guide them. Furthermore, Task 1 and Task 4 took the least amount
of time to complete (Task 1 mean = 4 seconds and Task 4 mean = 4.3 seconds). Task 1 involved
the participants clicking on the appropriate grade level under the Math link and Task 4 involved
beginning the practice skill after completing the pretest.

Time on Task (in seconds)

IMA 2 - Usability Testing Report

6
Spyros Alexa Rawah TOT Avg.
Task 1 3 5 4 12 4
Task 2 7 319 79 405 135
Task 3 N/A N/A N/A - -
Task 4 3 6 4 13 4.3
Task 5 10 17 8 35 11.7

C. Errors
The highest amount of errors was Task 4 (3 errors), which was to begin the practice skill
after completing the Math Pretest. The second lowest amount of errors was Task 3 (1 error) and
Task 5 (1 error). Task 3 involved following the directions for submitting their answer for each
question asked and Task 5 involved clicking on the Id like a hint or Stuck? Watch a movie
to help guide them in answering the practice skill questions. In addition, there were no errors
reported for Task 1 (click on their appropriate grade level) or Task 2 (complete the Math Pretest
activity). There were no critical errors due to every task was able to be completed. All of the
errors were non-critical as they did not hinder the successful completion rate.
Errors
Spyros Alexa Rawah TOT Avg.
Task 1 0 0 0 0 0
Task 2 0 0 0 0 0
Task 3 0 1 0 1 0.34
Task 4 2 1 0 3 1

IMA 2 - Usability Testing Report

7
Task 5 0 1 0 1 0.34

D. Satisfaction
To collect information about the participants satisfaction with the site, an online
satisfaction survey was administered. The results of the survey show that, overall, the
participants were in agreement about being satisfied with this website. Even though one
participant was neutral, it seemed all participants would like to keep utilizing the website. In
addition, all participants agreed (either agree or strongly agree) that the website was well
organized, easy to use and the information was accessible, which also explains their
disagreement with question three (i.e. Found it difficult to keep track of where they were in
website).
Post-Task Overall Satisfaction Questionnaire

Strongly
Disagree
Disagree Neutral Agree Strongly
Agree
Thought Website was easy to use

2 1
Would use website frequently

1 2
Found it difficult to keep track of where
they were in website

3

Thought most people would learn to use
website quickly

1 2
Can get information quickly

2 1
Homepages content makes me want to
explore site

3


IMA 2 - Usability Testing Report

8
Sites content would keep me coming back

1

2
Website is well organized

1 2

IV Impressions, Ratings & Recommendations
A. Impressions
Unanimously, Group 1 was satisfied with Khan Academys website. All participants as
well as administrators would use the site again and/or recommend the site as an useful
educational tool. All participants were able to navigate Khan Academy with ease and found the
hints and movies helpful and quick to find. In addition, based on our observations, all
participants were relaxed during completing the tasks which indicates the friendly atmosphere of
the website. This website is definitely one we would recommend to other instructors and/or
parents to utilize with their students/children to practice Math, Reading, and Science skills. In
addition, the Id like a hint or the movie icon was a great aspect to incorporate on the website
to give extra help to those who need it.
B. Ratings
Group 1 rated Khan Academy on the following areas: Learnability, Errors, and Satisfaction. We
rated the areas using a 1 to 10 scale range (10 being highly recommended and 1 being least
recommended). As shown in the chart below, the website scored very well in all three areas.
Errors and satisfaction scored slightly below learnability due to the fact that there were a few
errors indicated during the observation. This would result in a slightly lower score for
satisfaction. However, overall, the websites average in all three areas indicates it is a well

IMA 2 - Usability Testing Report

9
established site that is user friendly and an effective method to practice Math, Reading, and
Science skills.
Learnability Errors Satisfaction
Cayla 9 8 9
Pelagia 10 9 9
Abeer 10 10 9
TOTALS 29 27 27
AVERAGE 9.7 9 9

C. Recommendations
Overall, Group 1 was very satisfied with Khan Academy website. However, there are a
few recommendations to improve the overall ease and use of this website. Our recommendations
for this website are upon our observations and task completion forms and include the following:
1. Clarify the Getting to know you Activity which is actually a pretest to determine user prior
knowledge. In addition, they need to alert users that the pretest can be accessed only one time.
2. Provide a clear display of where the user is when navigating through the website, especially
for early grade levels.
3. Provide simplified instructions (e.g. a pop-up message) on where to begin with the initial
practice skills after completing the Math pretest as our usability report revealed that the highest
amount of errors was in this area.

IMA 2 - Usability Testing Report

10
4. Clarify what the energy points are used for and what the different badges mean that you
receive after mastering a practice skill.
5. Personalize the website and create a virtual spokesperson who guides users through the
activities, especially for early grade levels.
V Conclusion
In conclusion, all administrators and participants viewed Khan Academy favorably. Khan
Academy is a fun, effective method for users of any age; from preschool to college. Hints and
videos which are embedded in each problem, provide excellent scaffolding support. The website
also provides instant feedback and the users progressed is continuously displayed and tracked on
the dashboard. Errors were minor and all participants found Khan Academy a pleasant
experience. We believe that once the learners become familiar with the website, they will use it
more frequently as a source of educational practice.








IMA 2 - Usability Testing Report

11

Attachments
Attachment A - Task Scenario
Observer Script
Thank you [Name of Participant]... for your participation in this activity. Today, we
will be exploring a website called Khan Academy, which involves Math, Reading, and Science
skills.
After this activity is complete, we should be able to see how well this website works and if we
want to use it with other students like yourself. The purpose of this activity is to find out how we
can improve the website. As you are exploring, please tell us what you are thinking and feeling
as you use the site. However, remember we are not testing you; it is a test to see if the website
works properly.
You will be given five activities to complete. Before you start, we can answer any questions you
have. However, we cannot answer questions during the activity. But remember this is not a test
for you. I want you to try your best.
Do you have any questions before you start?
Lets begin exploring!

Attachment B - Observation Forms
Participant 1: Spyros (4th Grade)
Observer Name: Pelagia Mazarakis
Tasks Start Time End Time Errors Comments
Student clicks on their
appropriate grade level
0:00 0:03 None Spyros signed in with Facebook and
immediately selected appropriate grade level

IMA 2 - Usability Testing Report

12
under the Math link. 4
Student is able to
complete the Math
Pretest activity.
0:03 0:10 None Spyros did make errors on actual getting to
know you test but none in finding and
completing the task of getting to know you
Student is able to follow
the directions for
submitting their answer
for each question asked.
0:10 0:12


None Spyros stared lesson on unit sense and made
no errors in submitting his answers. He did it
quite naturally.
After completing the
pretest, student is able to
start the practice skills.
0:12 0:15 2 Spyros immediately (within 1 min) began the
practice skills but at first was confused how to
click on the buttons thus making 2 errors.
Student is able to click on
Id like a hint or
Stuck? Watch a movie
to help guide them in
answering the question.
0:15 0:25 None When Spyros was not familiar with a math
question he first clicked on Id like a hint . If
that was not sufficient to solve the problem he
would then click on the Stuck? Watch a
movie button.

Participant 2: Alexa (5th Grade)
Observer Name: Cayla Repass

Tasks Start Time End Time Errors Comments
Student clicks on their
appropriate grade level
under the Math link.

5 seconds None Alexa knew exactly what to do after logging
into the website. She clicked on the 5th grade
link.
Student is able to
complete the Math
Pretest activity.

5 minutes and 19
seconds
None Alexa was able to complete the pretest in a
timely manner with no errors.
Student is able to follow
the directions for
submitting their answer
for each question asked.

1 Error Alexa knew where to submit her answer each
time. However, for the math problems she did
not understand, Alexa did not see the link for
I havent learned this yet.
After completing the
pretest, student is able to
start the practice skills.

6 Seconds 1 Errors Alexa did not know what to do after the
progress window appeared when she
completed the pretest. However, she knew she
had to complete each practice skill in order.
Student is able to click
on Id like a hint or
Stuck? Watch a movie
to help guide them in
answering the question.

17 seconds 1 Errors Alexa said the hint helped her better
understand the problem. For the Second
Practice Skill, she needed to watch the movie
on how to understand the problem. However,
after watching the movie, she did not know
how to go back to the original problem.

Additional Comments:

IMA 2 - Usability Testing Report

13

Alexa mentioned that she did not understand the badges and points you received after completing a practice skill. However, she could find the
skills she mastered and the skills she needed extra practice to master.

Participant 3: Rawah (7th Grade)
Observer Name: Abeer Mohammad
Tasks Start Time End Time Errors Comments
Student clicks on
their appropriate
grade level under the
Math link.

0:00

0:04

None
Rawah immediately clicked on her
appropriate level (Grade 7)
Student is able to
complete the Getting
to Know You
activity.

0:04

1.23

None
Rawah had some difficulties in solving
some of the test items and used the I
havent learned this button couple of
times.
No errors related to the Getting to
Know You activity, i.e. starting and
completing the task, were observed.
Student is able to
follow the directions
for submitting their
answer for each
question asked.
Rawah spent this time
observing the summary of her
Getting to Know You activity



None

In Getting to Know You activity and in
all other tasks, Rawah successfully
submitted answers and no difficulties
were observed
After completing the
pretest, student is
able to start the
practice skills.
1.34 1.38 None After completing the pretest and
reviewing her summary results, Rawah
started the Complementary and
Supplementary task by clicking on the
START button.
Student is able to
click on Id like a
hint or Stuck?
Watch a movie to
help guide them in
answering the
question.

1.46

2.10

None
Once the Complementary and
Supplementary task started, almost after
.08 seconds, Rawah clicked on Stuck?
Watch a movie button to get help.
However, after .64 seconds, Rawah
stopped the video and went back to the
task finishing it successfully. She had no
problems in navigating back to the task.

Attachment C - Post-Task Overall Satisfaction Questionnaire
Participant 1: Spyros

Strongly
Disagree
Disagree Neutral Agree Strongly
Agree
Mean
Rating
Percent
Agree
Thought Website was
easy to use

x

Would use website

x


IMA 2 - Usability Testing Report

14
frequently
Found it difficult to keep
track of where they were
in website

X

Thought most people
would learn to use
website quickly

x

Can get information
quickly

x

Homepages content
makes me want to
explore site

x

Sites content would keep
me coming back

x

Website is well organized

x



Participant 2: Alexa

Strongly
Disagree
Disagree Neutral Agree Strongly Agree Mean Rating Percent
Agree
Thought Website was
easy to use

X

Would use website
frequently

X

Found it difficult to
keep track of where
they were in website

X

Thought most people
would learn to use
website quickly

X

Can get information
quickly

X

Homepages content
makes me want to
explore site

X

Sites content would
keep me coming back

X

Website is well

X


IMA 2 - Usability Testing Report

15
organized

Participant 3: Rawah

Strongly
Disagree
Disagree Neutral Agree Strongly
Agree
Mean
Rating
Percent
Agree
Thought Website was easy to
use

X

Would use website frequently

X

Found it difficult to keep track
of where they were in website

X

Thought most people would
learn to use website quickly

X

Can get information quickly

X

Homepages content makes me
want to explore site

X

Sites content would keep me
coming back

X

Website is well organized

X


References
Khan Academy Website. (2014). Retrieved from https://www.khanacademy.org/
Nielson, J. (August 25, 2003). Usability 101: Introduction to Usability. Retrieved online January
3, 2014 at: http://www.useit.com/alertbox/20030825.html