To: C.

Ferriter
From: Jesse Shuman
Re: instructions and Usability Report
Date: 03/7/2018
I created a set of instructions over how to properly tie a neck tie. I choose this topic because
I’ve been to countless events in my lifetime where a tie was required. I can remember the first
time I had to tie a tie on my own for a social event. I spent the whole night looking up tutorial
videos and trying to follow confusing diagrams to get the tie right before the next morning. I
decided to enlighten the ones who still don’t know or may be interested to know for the future.
After I created the instructions I also made a set of questions for the subjects to answer before
they attempted to follow the directions. I also formed another set of questions for after they
completed the task. I took four rounds of trials to gather notes over the ease and difficulties the
subjects experienced over the task. I used these notes to improve the instructions. With the
revised set of instructions, I did three more trials. I received more notes and recommendations.
Methodology
I chose to use retrospective think aloud protocol. In this strategy, the subject can focus on the
task at hand then rather than diverting their attention to answering questions. Explaining their
thought process like they would in concurrent protocol could affect their ability to complete to
task. As where in retrospective thick aloud, all the questions and feedback are saved for once
the trial is completed. “May add little value to usability tests beyond what can be derived from
users’ observable behavior” (Hertzum, 2016, p. 3). I agree with Hertzum that not much more
can be pulled from classic thinking aloud that you couldn’t observe yourself. It is easy enough to
see when a subject is stuck on a step, without them saying anything. “RTA method, on the
other hand, proved significantly more fruitful in revealing problems that were not observable”
(Haak, 2003, p. 11). Haak said retrospective think aloud is more can provide more feedback
then concurrent. Both concurrent and retrospective are useful in the right setting. If I had
chosen to do a longer trial concurrent think aloud might would be a better method so the
people wouldn’t forget a thought they had mid-trial. Since my instructions were short and
relatively simple, I decided retrospective protocol was the best option.

Results and Analysis

Pre-test
I typed up a pretest for the subjects to answer before the attempted to follow the instructions.
This helped me get a survey of what range of people I was testing and gave me a better
understanding of their formality with tying ties. The results of the pre-test didn’t change from
the draft instructions and the revised instructions because they are asked before the subjects
looked at the instructions. The first question was “Have you ever been to an event wear ties
have been required for the men?” This question let me know the relevance of neckties for my
audience. For all seven trials the answer was a constant yes. From this I gathered that all the
subjects knew what a tie is and what scenarios one would wear one in. The second question
was “Have you ever successfully tied a tie?” This question was designed to determine the level
of experience the subject held in actually tying a tie. Only two out of the seven subjects answer
the question with a yes. From this I knew the majority of the people doing this experiment have
no experience with the task and will need very detailed step by step instructions. The third
question was “Can you complete the task without the instructions?” This was my final question
of the pretest. I used it the discovery the subject confidence level over the following task. Only
one person answered this question with a yes. This was to be expected considering most of the
subjects said they have never done it before.

Post Test (draft Instructions)

The post test was my way of executing the retrospective think aloud protocol. I used it to help
identify problems and gather information that helped me improve my instructions. I split the
results from the draft instructions and the revised version of the instructions because the
results were very different from each other. The first question was “Were the instructions easy
to follow?” This was a yes or no question to get a general idea of the understanding of the
instructions. Two out of Three people said no, meaning the directions were not easy to follow.
It was abundantly clear this draft needed work. The second question was “Was there one step
that was particular harder than the rest?” In the short answer question, I was asking for a
specific step that the subject might have been stumped on. I also followed the question with
why this question was confusing. Steps four five and six were brought to light. Step four was
reported for the terms thick side and thin side being confusing. Also, that I said “while
continuing to hold the fold” at the end of the sentence when it would have been more helpful
in the beginning. In step five, the word clock wise threw off one subject. They suggested I find
another way to word that direction. In step six, a subjected wasn’t sure where to go from this
step so offered the advice that it might need to be more specific. That concluded all the
feedback for that particular question. My next question asked “Do you think you can
successfully tie a tie on your own now?” This was another way to determine the success of the
instructions on the subjects. Only one person said they still could not tie on their own. So
difficult to follow or not the instruction was an overall success but had plenty of room for
improvement. My third question was “Are you more likely to wear a tie now?” The idea behind
this question was determine how much learning this task has affected the subject’s life. All the
male subjects answered yes to this question and all the female subjects said no. I also asked for
brief recommendations to further improve the instructions. All subject mentioned adding
photos to help one follow the process easier. Another recommendation was giving to be more
specific so someone who has no experience can make sure they are completing the steps
properly.

Post Test (Revised Instructions)

After editing my instructions, I did three more trials using the same post questions. For
question one, 67% claimed the directions were easy to follow. Only one out of the three
subjects said the instructions weren’t easy to follow. In the second question, one person
answered yes that the word clockwise was confusing in step five. This being the second time I
have received that comment I would definitely rephrase that step. However, the rest of the
subjects answered no for that question. For my third question, two people claimed they could
now tie a tie on their own, one said maybe after more practice. This could have something to
do with using different subjects this trial. The result for my last post-test question was the same
as the results from the draft instructions. All the males said yes, and all the females said no. The
only other recommendations I received from this test was to remove the word clockwise.

Retrospective think aloud protocol provided more than enough feedback to improve upon the
instructions. With the revised copy of instructions, 67% could complete the task. The pictures
were a big success and helped improve the usability rate. Some of my word choices could be
seen as a failure, considering it continued to complicate the process on the revised instructions
for the subjects. With all the data I have gained from usability testing, I’m sure I could perfect
the instructions with a few more alterations.

Discussion and Justification

After the first four trials I got plenty of feedback to work with from my usability test. From the
first set of trials the majority of my feedback stated that just by adding pictures along the steps
would drastically improve my instructions. That was the first and probably most successful
revision made. The pictures I added to the instructions were of myself tying the tie. I wanted
the steps to be easily visible, so add contrast I wore a white shirt with my most flamboyant tie.
Another I made was the confusing phrasing and misspelling. Once I updated them I did another
three trials with different people and received more information to further better the
instructions. The feedback on this test was more positive then the first test. The majority of the
subjects said it was easy to follow and didn’t given any further recommendations. One reported
to check my spelling and remove the word clockwise to simplify my language. I used this to
make a few last changes to my instructions before I submitted it.

References
Haak, M. V., & Jong, M. D. (2003). Exploring two methods of usability testing: concurrent versus
retrospective think-aloud protocols. IEEE International Professional Communication
Conference, 2003. IPCC 2003. Proceedings. doi:10.1109/ipcc.2003.1245501

Hertzum, M. (2016). A usability test is not an interview. Interactions, 23(2), 82-84.

doi:10.1145/2875462

Geisen, E., & Bergstrom, J. R. (2017). Usability and Usability Testing. Usability Testing for Survey
Research, 1-19. doi:10.1016/b978-0-12-803656-3.00001-4
Appendices
Pre-test
1. Have you ever been to an event wear ties have been required for the men?
2. Have you ever successfully tied a tie?
3. Can you complete the task without the instructions?

Post-test
1. Were the instructions easy to follow?
2. Was there one step that was particular harder than the rest?
3. Do you think you can successfully tie a tie on your own now?
4. Are you more likely to wear a tie now?
5. Any further recommendations?