12/12/2010

Why Evaluation
Does the Interface Meet the Usability Requirements?
Effective: completeness and accuracy Efficient: speed Engaging: Error tolerant Easy to learn

Evaluation
Dr. Ann Nosseir Stone et. al. ch 20-27 Benyon et. al. ch 12 & 21

Why Evaluation (cont.)

Exploring Other Concerns in Evaluations Why users are unable to complete tasks easily. Is the UI developed for all levels of users? Are all design features acceptable to users?

The Activities of Usability Evaluations

The Process of Usability Evaluation Is Iterative Techniques for Usability Evaluations
User Observations Inspections of the User Interface
Conform to usability standards?

Other Evaluation Techniques

Variations of user observation or inspection

12/12/2010

What Happens in a User Observation Evaluation Session? Welcome participant, explain purpose, make participant comfortable Ask participant to complete tasks while you observe and record Following completion of tasks, as for participants views, or to complete post-test questionnaire Thank participant.

Creating an Evaluation Strategy

What Is the Purpose of This Evaluation?
Does system meet usability requirements/concerns Qualitative Usability Requirements Desired features

Creating an Evaluation Strategy

Quantitative Usability Requirements/Usability Metrics
Explicit measures used: percentages, timings, or numbers are used.
It should be possible for the users to load any page of a web site in 10 seconds using a 56K modem. It should take no more than two minutes for an experienced user (one who has domain knowledge and has undergone the prescribed level of training when the new system is introduced) to enter a customers details in the hotels database

The users on an e-shopping site should be able to order an item easily and without assistance. Railway clerks work in extremely noisy environments, so any warning messages to them should be visually distinct and highlighted on the screens.

12/12/2010

Creating an Evaluation Strategy

Prioritizing Usability Requirements and Concerns
The usability requirements most important to the success of the system are given priority. Assign values to the five dimensions of usability, the Five Es.

The Five Es.

The Five Es.

High priority for online museum placed on engaging

Creating an Evaluation Strategy

What Type of Data Do I Want to Collect?
Quantitative data
Numeric content

Qualitative data
Non-numeric content

12/12/2010

Think of
What Am I Evaluating? What Constraints Do I Have?
Money Timescales Availability of usability equipment Availability of participants and the costs of recruiting them Availability of evaluators

Documenting the Evaluation Strategy

Choosing Your Users

Who Is a Real User?
Users who reflect the different skills, domain knowledge, system experience

Choosing Your Users

Ideas for Participants Offering Incentives
thank you letter, pay for out-of-pocket expenses, samples, gifts

Users Working Alone or in Pairs Number of Participants Recruiting Extra Participants

Recruiting Screeners and Pre-test Questionnaires

12/12/2010

Creating a Timetable
Decide the Duration of the Evaluation Session 30-90 minutes Create an Evaluation Timetable sessions, evaluation, reporting

Preparing Task Descriptions

the tasks the participant will perform while interacting with the prototype during the evaluation

Where Will You Do the Evaluation?

Field Studies users own environment Controlled Studies other than users environment

Preparing to Collect Evaluation Data

Timing and Logging Actions
Automatic Logging of Keystrokes and Mouse Clicks Specialist Logging Software for Usability Evaluations Choosing a Logging Product

12/12/2010

Think-Aloud and Offering Help

Using Cognitive Walkthrough Questions
Is there anything there that tells you what to do next? Is there a choice on the screen that lines up with what you want to do? If so, which one? Now that youve tried ti, has it done what you wanted it to do?

Taking Notes When Observing Users

Conducting Post-Session Discussions

Retrospective protocol Post-session interview / debrief

Questionnaires
Advantages
Cant forget to ask a question All participants see the same questions Ability to collect quantitative data

Disadvantages
Difficult to design Must predict topics the users will need Closed questions dont give reasons why the users answered the way that they have.

12/12/2010

Using Technologies to Help Recording

Video and Audio Recording Eye-Tracking Equipment Practitioners Choice of Technology

with
Facilitator
Note-Taker

Roles for Evaluators

Equipment Operator Observer Meeter and Greeter

What to Do If a Participant Does Not Agree to Be Recorded

Recruiter
The Lone Evaluator

Usability Defects (characteristics)

Reviewing the Data to Identify Usability Problems

Irritates or confuses the user Makes a system hard to install, learn, or use Causes mental overload for the user Causes poor user performance Violates design standards or guidelines Reduces trust or credibility of the system Tends to cause repeated errors Could make the system hard to market

12/12/2010

Working with Quantitative Data

Tabulations, charts, and rankings for visual rep. Descriptive statistics: mean (average), median (middle value), mode (most common value) Inferential statistics: tests of statistical significance yielding probability.