Scenario Tasks

Task 1: Answer the following question in the quiz, as shown on the slide.
Task 2: Indicate that you wish to answer the question that your professor has posted
verbally to the entire lecture.
Task 3: Indicate that you have queries/questions about a particular slide that has
passed, while the lecture is still in progress.

Observations
Scenario Task 1:
Our first user, Hei Wai performed the task smoothly. She pressed the button ‘5’ as the
answer to the quiz question as shown in Figure 1.4. She briefly mentioned that this
means she answered the question. She took a few seconds to realize the change in
feedback light (green to blue) when we showed her how the clicker would respond to
her action. The sketches flashed are as shown in Figure 1.2 and 1.3. The delay in
response time is due to the limitations of paper prototyping which is the lack of
colour. She had also mentioned that she thinks this change of colour in the feedback
light means her response is accepted.
Our second user, Sheng Rong also performed the task smoothly. She pressed the
button ‘2’ as the answer to the quiz question in Figure 1.4. She mentioned that she did
not clearly understand the change in colours of the feedback light but subsequently
figured that it could represent that her answer has been received by the system. The
limitation of the testing method used is also reflected here.
Our third user, Jamie had queries on how to perform the task at first. She asked
whether she could randomly press any button to answer the quiz question. She
realized that the quiz question in Figure 1.4 had options 1 to 5, so she clicked button
‘2’ as her answer. She was confused about the blue light but subsequently felt that the
change in colours of the light was not a concern or a problem. She felt that the screen
was a good addition as it showed the answer that she has picked for the question.
There were two usability problems. There was a few seconds delay in the response of
first two users due to failure to recognise change in the colours of the feedback light.
This suggests that the colour change or blue colour choice may not be intuitive to the
users. This may be because it is inconsistent with familiar systems’ feedback light for
response acceptance. However, the short delay and the users’ subsequent
understanding without prompt suggest that this is a minor usability problem.
Furthermore, the time delay may be inflated by the serial position of the tasks and
paper prototyping limitation. There is a need to follow-up with high fidelity
prototyping which provides real time interaction, and changing task sequence to
certify the need for change in feedback light. Secondly, the feedback light may be
redundant, as perceived by the third user. This is because the small screen also
provides feedback to the response selection, resulting in multiple sub-feedback
mechanisms. It occurs to us that the feedback sequence of the UI may be too long,
thus there may be a need to shorten the sequence. We can consider the removal of the
blue feedback light as irrelevant features increases user’s cognitive load and impacts
the design aesthetics.
Also, we realized that all three of them did not notice that in the slide (Figure 1.4),
there were two distinct sections in the toolbar (Figure 1.5) and that the responses
section was activated while the other function was deactivated.
Scenario Task 2:
Our first user, Hei Wai performed the task without much difficulty. She said that the
hand button looks most relevant in this task and thus she pressed the hand button.
When the hand button lighted up in green, she said that this is an indication that it is
in the usage of hand button mode. Her response indicates that the system effectively
communicates the design model to the user, resulting in the correct user mental
model.
Once again, Hei Wai did not notice that the change in activation responses function
for the quiz questions to the hand/participation function on the toolbar (Figure 2.4).
Our second user, Sheng Rong, also performed the task smoothly. She pressed the
hand button as she said it felt most intuitive and relevant to perform task 2 using this
button. She also understood that when the hand button lighted up, it indicated that this
button was in use. She briefly mentioned her concerns on how the professor would be
using this piece of information. Like Hei Wai, Sheng Rong did not realize there was
the toolbar to feedback to students about responses and queries and whether which
mode is currently turned on or off.
Our third user faced some problems with this task. Initially, she did not know how to
use the clicker to perform the task. However, she said she would raise her hand in
normal situations when she wishes to answer a professor’s question. Following this,
she thought carefully and looked at the clicker prototype again and pressed the hand
button. After she was presented with the final sketch showing the blue feedback light
(Figure 2.2), she expressed some confusion about whether it represented that
responses were received or an indication that one is picked to answer the question.
There were two usability problems. The first would be the non-visibility of the toolbar
in the slide (Figure 2.3). Users fail to notice the change in activation from the
response mode to the participation mode. As explained earlier, this is likely to be a
result of the use of paper prototyping sketches whereby we only use pencil
descriptions, thereby being unable to capture users’ attention and to reflect the real
time changes in such situations. The second problem is brought up by the third user,
which is a change in the purpose and function of the hand button, resulting in mode
error. She suggests distinct feedback lights to indicate the change in function. This
solution may be feasible if the number of feedback light colours is not too large
whereby users will have to learn them by rote, increasing their cognitive load and
usability problems associated with their usage. Follow-up testing is required to verify
this. An alternative solution would be to inform users about the function of feedback
lights. This can be done via the instruction cards that come with the clicker which are
usually placed on the back of the clicker. This is a convenient and easy way to
provide help documentation without complicating the design of the clicker.
Scenario Task 3:
Hei Wai had once again performed the task smoothly. She pressed the hand button as
she felt that it was intuitive as students would normally raise their hands before
posing a question to their professor, suggesting that the UI design matches the real
world. After pressing the hand button, she paused when the word ‘Slide’ appeared on
the screen of the clicker as seen in Figure 3.2. She mentioned that this was probably
to guide users to key in the slide number. When the feedback light was changed to
blue (Figure 3.4), Hei Wai wondered if this response is captured by the system
automatically or by the lecturer manually. As mentioned in the previous two tasks,
she did not notice the toolbar again or the change in functions in the toolbar.
Sheng Rong did not perform the task as we have expected. She pressed the 0/J button
to perform this task. She did it because she felt that this was an underused button and
could likely be used for this task. She felt that the hand button did not appear intuitive
as it did for the second task. We then presented her with the error sketch (Figure 4).
She was confused at this moment and wondered out a loud that it should not be the
hand button because she had already used it for the second task. Thus, the usage of
multiple modes reduces the execution of tasks associated with the button. She hopes
that there can be a separate button for the two different tasks. She decided to give up
doing the task after the error occurred. As mentioned in the previous two tasks, she,
like Hei Wai, did not notice the toolbar again or the change in functions in the toolbar.
Jamie performed the task relatively smoothly. She pressed the hand button as she felt
it was intuitive, as she would raise her hand before asking a question. After pressing
it, the word ‘Slide’ appeared and she was a bit confused about which slide number to
key in. However, she realized that she could figure out the slide number based on the
current slide or even from her notes. After keying in the slide number, when we
showed her the sketch with the feedback light in blue (Figure 3.4), she realized that
this change in feedback light was an indication that her responses was received by the
system. She questioned the ability of the system to cancel or change queries sent since
she was able to change her responses to the quiz question.
In all, there were two usability problems expressed by our users. The first is the
occurrence of mode error in the hand button. It is used for two different tasks –
sending queries and participating in class, resulting in error. After the performance of
the second task, users have difficulty in associating the hand button with the third
task. This may be because of the lack of labels in the hand button to indicate that the
button serves more than one mode. Furthermore, the other buttons on the UI do not
serve more than one function. Inconsistency may have resulted in the delayed or
inability to recognize the modes of the hand button. There are two possible solutions.
We can consider having two distinct buttons for participation and sending of queries –
where each button serves one function only, consistent with the design of other
buttons. There should be no user confusion or mode error but there is a space
constraint in the current UI thus we do not want to reduce the size of current buttons
in order to be able to add more buttons. This goes against the grain of an aesthetic and
minimalist design, reduces relative visibility of buttons by increasing competing
stimuli. Moreover, with careful consideration, the hand button does not represent a
direct step to performing the two different tasks (Tasks 2 and 3). In fact, it represents
only one function – that is to signal to the professor, matching the real world where
raising our hand gets the attention of our teacher/professor, regardless of whether we
want to answer questions or ask questions. Therefore, we may consider the second
solution which is to add a small label above the hand button to indicate multiple
modes.
The second problem is the violation of user control and freedom. Removal of
responses is not inherent in the original design. However, users were able to overwrite
responses to quiz questions which provide user control and freedom to some extent.
One solution would be the addition of a button that enables user to undo their action.
An alternative capitalizes on the consistency of other buttons which is to allow users
to overwrite a submission with an unsubmission. For instance, users can perform the
same task sequence for query sending or participation again. However, the system
will prompt the users to remove the previous submission. A drawback to this solution
is the occurrence of mode error from the multiple functions the buttons serve. Follow-
up studies are needed to test the effectiveness of the proposed solutions.
Overall, usability problems identified in this low fidelity prototyping procedure need
to be considered in the context of its limitations. The lack of real time interaction and
feedback as well as colours may have caused users to fail to notice details such as
change in activation of modes displayed on the clicker and the slide. High fidelity
prototyping is needed to confirm the findings in this testing.