Usability Test Plans

Testing Plan #1 Tammie Montgomery

Document Overview
The myCMU Go Mobile Manual is an electronic document to provide students a guide for the
Central Methodist mobile web site to interact with the course management system and other
school services. The document can be printed for reference.
Goals & Objectives
We intend to conduct field and observable lab tests evaluating CMU students as test subjects
using smart phones or tables as devices. A personal computer or laptop can be used in lieu of a
mobile device if the student does not have one available. The lab test will be conducted in a class
with 5 or more students. Field and lab tests will have the same tasks to complete. Lab tests will
include observations from the test proctor.
Quantitative Goals
We want to be respectful of time but also provide enough tasks to test the viability of the manual.
We anticipate that the total test will take no more than 30 minutes to complete including the
questionnaire. The following will be measurements of the test:
1. Time to completion: Time to find the correct place in the document, time per task and
total time to completion will be measured. If the document is successful, the total time
per task will be minimal for the majority of the students.

2. Error rate: Measure the number of errors that occurred including connection failure and
page not found. We will record errors caused by devices and/or ISP issues even though
they are beyond our control. We could use them to create FAQs for recovery which
should increase the satisfaction with the document.
3. Number of lost paths: Measure the number of times the document took them in the wrong
direction. If many have issues that take them down the wrong path or they have to repeat
missed steps, then the document will need to be revised to be clearer on the instructions.
Qualitative Goals
This manual should be a useful guide to assist the students in getting to the information within
the system as quickly and efficiently as possible. A Likert scale can be used to measure how
agreeable or disagreeable the student is with their experience. We will also try to ascertain the
comfort level of the student with the device they are using to possibly explain the time to
completion if there is a significant difference. Ultimately, we would like to know if the student
would recommend the document to their classmates because this would let us know they found it
to be a useful document.
Testing Procedure
1. Select qualified participants:
a. Lab environment: one class of 5 or more CMU students. This will be prearranged
with the instructor.
b. Field study: other students in our class as well as CMU students we know.
2. Orient the participant: Thank you for agreeing to participate in our study. The purpose is
to test the usability and improve our manual. You will be asked to perform several tasks
and record specific aspects of your experience. Once you have completed the tasks, we
would appreciate you completing a short survey. All data will be reported in aggregate
and your identity will remain confidential.
3. Provide participant with the URLs:

a. The manual is TBD

4. The URL for the myCMU Go Mobile site http:// compass.centralmethodist.edu/go
5. Brief instructions will be given on how to access both at one time and switch back and
forth for most devices. Not all devices will be the same.
6. Provide participants with the Usability Test Task Sheet (see attached). For the field test,
email this to each person and suggest they print it out before starting the test.
7. Provide link to post-test questionnaire
8. Retain data for evaluation

Usability Test Task Sheet (DRAFT)

Complete the following tasks using the myCMU Go Mobile Manual. With each task, record the
information requested. Once completed, return to your team contact.
Date of test: ______________

Start time: _____________

Device used: Cell phone

Tablet

Laptop

Connection type: WiFi

Phone network

Desktop Computer __________

Task 1:
Completed

Check your institutional announcements.

Not complete

Approximate time to complete the task: _______ min ________sec

Number of errors received during this task: __________
Number of times you took the wrong path and had to go back: ____________
Task 2:
Read a discussion board post for one of your classes.

Completed

Not complete

Approximate time to complete the task: _______ min ________sec

Number of errors received during this task: __________
Number of times you took the wrong path and had to go back: ____________
Task 3:
Check on the status of your financial aid.

Completed

Not complete

Approximate time to complete the task: _______ min ________sec

Number of errors received during this task: __________
Number of times you took the wrong path and had to go back: ____________

End Time: ____________

Total time to complete: _______ hrs _______min

Usability Test Questionnaire

Thank you for completing the usability test. Please take a few minutes to complete the
following questions. The results will be used to improve the instructions provided in the
myCMU Go Mobile Manual.
Select the response that best describes how you feel about each statement.
5 Strongly agree, 4 Agree, 3 Neutral, 2 Disagree, 1 Strongly disagree

I was comfortable using

this device
The document helped me
complete the task
When I had an issue, the
document helped to resolve
the problem
I would recommend this
document to my friends
The document was easy to
understand
Overall, I am satisfied with
my experience

Strongly
Agree 5

Agree 4

Neutral
-3

Disagree
-2

Strongly
Disagree
-1

Please provide any additional comments on how we can make this a better document for you:

Testing Plan #2 Courtney Lane

Document Overview
This document describes a test plan for conducting a usability test during the development of
myCMU GO Mobile. The goals of usability testing include establishing a baseline of user
performance, establishing and validating user performance measures, and identifying potential
design concerns in order to improve the efficiency, productivity, and end-user satisfaction.
Goals & Objectives
Quantitative Goals:

Success Rates Did all of the participants successfully navigate the instruction manual?
Task Time How long did it take the participant to successfully complete the task?
Error Rates How many errors occurred during the time elapsed? (i.e. connection failure,

page not found, failure to load,etc.)

Satisfaction Questionnaire Ratings Did user experience prove to be satisfactory?

Qualitative Goals:

Pathway Observations Did users utilize the available resources of the manual to

navigate with ease?

Encountered Problems Which sections of the user manual were more problematic?
Recommendations What suggestions does the user have to enhance the usability of the

manual?
Answers to Open Ended Questions This segment will allow users to answer proposed
questions to encourage reliability and validity.

Testing Procedure
1. Participant Selection Participants will be current or potential Central Methodist
University (CMU) students with private access to smart phone and/or tablet to ensure
internal validity.
2. Orientate participants and gather usability testing consent forms.
3. Provide participants the URL to myCMU GO Mobile site.
4. Provide participants with a list of destinations to navigate to (i.e. financial aid awards,
course announcements, gradebook, discussion forums, etc.).
5. Conduct post-test questionnaire. Post-test questionnaire will consist of five closed ended
questions and five open ended questions, as well as a mini-demographic survey.
6. Conclude with post-test satisfaction questionnaire.
7. Compile data received and prepare for the usability test report.

Evaluation Measurements
Testing results should correspond with previously mentioned goals and objectives, and are as
follows:
Success Rates

Exceptional: 1 attempt
Passing: 2 attempts
Failing: 3 attempts

Task Time

Exceptional: < 30 seconds to completion

Passing: 30-60 seconds to completion
Failing: 60 seconds to completion

Error Rates -

Exceptional: 0-1 errors

Passing: 2-3 errors
Failing: 3 errors

Satisfaction Score -

Exceptional: 10 points
Passing: 6-9 points
Failing: 0-5 points

Usability Test Consent Form

In this usability test:

You will be asked to perform certain tasks on a computer.

We will also conduct an interview with you regarding the tasks you performed.

Participation in this usability study is voluntary. All information will remain strictly confidential.
The descriptions and findings may be used to help improve the Kinect application. However, at
no time will your name or any other identification be used. You can withdraw your consent to the
experiment and stop participation at any time.

If you have any questions after today, please contact Courtney Lane at c.lane3908@gmail.com

I have read and understood the information on this form and had all of my questions answered.

______________________________

__________________

Participant Signature

Date

Post-Test Questionnaire
For each of the questions below, circle the response that best characterizes how you feel about
the statement, where 5 = strongly agree, 4 = agree, 3= neither agree or disagree, 2 = disagree,
1= strongly disagree.

agree

neither
agree or
disagree

disagree

strongly
disagree

The manual was easy to comprehend.

I completed most tasks in 30 secs.

I did not encounter many errors.

strongly
agree
The user manual helped me navigate
the website.

I will recommend this manual to

5
4
other users.
Please provide a response for the following questions:
What did you like about this manual?

What did you not like about this manual?

What was easy to understand about this manual?

What was difficult to understand while using this manual?

What things would you change about this manual?

What questions arose during the utilization of this manual?