Usability Doc Lukebishop

Usability Test Report Document / www.ybe.
com / September 2008
Usability Test Report Document / www.ybe.com

conducted and written by Luke Bishop

September 2008 Qantm College, London Interface Design Amir Dotan
Usability Test Report Document / www.ybe.com / September 2008
Table of Contents
Executive Summary User Proles Methodology Rationale Testing environment Procedure Analysis Pre-scenario 1 Pre-scenario 2 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Survey results Interview results Summary and Conclusion Expert Review Appendices A - Pre-test questionnaire B - User release form C - Post-test interview D - Post-test survey E - Scenarios / Notes References / Test script 21 22 32 33 34 35 36 - 42 43 3 4 4
4 7 8 9 10 12 13 16
Executive Summary
A usability evaluation of the main pages of www.ybe.com was conducted in September 2008. The overall purpose of this test was to observe and measure the usability of FlyBes most important and essential functionality - its ticketing booking function, as well as other key factors. The FlyBe site is primarily intended for use by the general public in Great Britain and Europe to book and purchase air travel tickets; a secondary purpose if for booking hotel accommodation and car hire. This report document is a compilation of the methodology used to conduct the test, the data gathered during the test, as well as recommendations for addressing and correcting the usability issues found during user testing. In a broad sense, users generally characterized the FlyBe web site in two distinct ways: 1) the site was functional to an adequate degree and adopted a conventional approach to airline web site design, but suffered from a lack of consistency across several measures, and 2) felt that the sites aesthetics offered very little to the overall experience, and in some instances, detracted from it. The primary user task of booking air travel is well supported with a highly visible Book A Flight functionality panel on the home page, positioned in an area of highest trafc and visibility 1. The secondary user task of booking additional hotel accommodation and car hire are supported via icons in the upper navigation, and from boxed content areas - both located on the home page. The upper icons are located on subsequent pages throughout the site. Although not primary functionality, the site further supports users interested in investment, employment and corporate information via links found in the global navigation. However, there is opportunity for improvement in several areas:
Visual design: the site is cluttered, confusing and too busy. Information could be presented in a less cumbersome manner. Streamlining: some pages are launched in child browser windows, and also lack corporate branding and navigation; additionally, some paths take the user out of the main booking site. Location within site: the site does not provide clear, consistent visual cues that inform the user where she is at all times. Home page content: visual elements on the home page are not weighted in importance elements compete with each other for attention and creates a distracting experience. Repetition of navigation: multiple links to the same page occur throughout the site. Labeling: the site suffers from inconsistent use of navigation terms and category heading names. Mental models: portions of the site present information to users in a way that seems illogical, or is unexpected. Error recovery: error messaging is not always placed ideally, and recovery from user error could be improved, in the Account Settings, for example.
No issues were deemed to be critical, while approximately 60% of the issues were of medium severity. Several issues were deemed to be of high severity. Many of the issues of
3
medium severity can be xed by improving navigation, headers, language and page layout.
User Proles
Information collected in the pre-test questionnaire prole data:
(Appendix A)
revealed the following user
user ID
age
gender
profession
education
hours on computer p/week
purchases on internet
frequent yer member
U1 U2 U3

user ID
48 45 46

M M M

Nursery teacher Interior designer Advanced Rolfer

University graduate University graduate Post-graduate

0-10 hrs 11-25 hrs +26
Y Y Y
Y Y Y
types of purchases
types of computer activity
U1 U2 U3

airline / train tickets, books airline / train tickets, home merchandise airline / train tickets, books
email, professional research, word processing email, social networking, word processing email, professional research, word processing,
User demographics collected from the pre-test questionnaire.
Methodology
Rationale
Materials for usability testing scenarios, tasks and questionnaires were derived from the expert review conducted on www.ybe.com. Based upon ndings from the review, scenarios, tasks and questions focused on the effectiveness, error frequency/severity and user satisfaction of the following areas: booking functionality, the global navigation, account registration, frequent yer account set up, and FAQs/Q&A. Each of these ve areas are core features of the site, yet each exhibited potential usability issues ranging from cosmetic to critical, and therefore were selected for further investigation via user testing. Cost constraints permitted usability testing of only a portion of the booking process - users were not required to complete ticket purchases, thus incurring personal charges.
4
Testing Environment
All usability tests were conducted during a 48-hour period between 01 - 02 September, and transpired at the same location in an effort to ensure all users experienced identical testing conditions and environment. Testing occurred on a MacBook laptop computer with remote mouse device, using the Firefox 3.0 browser and a broadband connection (8MB). Audio and video for each testing session were recorded with ClearLefts Silverback screen recording software (Silverback allows the insertion of cue points during testing via remote control, so facilitator(s) may quickly and easily refer to specic points in a video timeline). Each user signed a release form authorizing the use of this recorded information for the purposes of this study (Appendix B). The average duration for each session was one hour, including time to respond to pre-test questionnaire, as well as the post-test interview (Appendix C) and survey (Appendix D).
Procedure
First, testing procedure and expectations were explained to users and any questions users had initially were answered. Secondly, users were asked to complete a demographic questionnaire. Thirdly, each user was presented with 7 scenarios/tasks (Appendix E), of which the rst two were exploratory in nature. The user was asked to perform each task as he normally would outside a test setting, one at at time, while the facilitator observed and documented user behavior. Users were encouraged to think aloud to provide additional details about their actions and decisions, and instructed to verbally inform the facilitator when they believed they had completed the task. Lastly, users were interviewed about their overall experience using the web site, and also completed a short likert-scale survey. Users were intentionally given initial exploratory exercises in order to habituate themselves to the elements of the sites interface, as well as to the unusual activity of thinking aloud while performing a task. Also, tasks that were deemed to be relatively easy during the expert review (for instance, S1, S2) were deliberately chosen to be performed rst so that users did not feel immediately discouraged by tasks that could possibly be overly complicated (S4).
Analysis
User Proles
All users tested are university graduates (U3 has post-graduate experience), use a computer at least 10 hours per week, and all have used the internet for booking and purchasing travel, particularly tickets for air travel. All users had previous experience using airline web sites, therefore issues uncovered during usability testing are not generally attributed to users unfamiliarity with conventional air line web site functionality; survey results indicate that users considered www.ybe.com to be a conventional and standard air line site.
Scenarios
Overall, users were able to complete all tasks (only one user was unable to complete a single task). Time-to-complete gures reect a sense of uniformity regarding how users address a particular issue; 4 tasks were completed in under 3 minutes, while all users were timed between approximately 6-8 minutes to complete Scenario 4. During testing, users were asked to perform the following: Pre-scenario 1: explore the FlyBe home page, scrolling but without clicking. (PS1) Pre-scenario 2: explore the entire FlyBe web site, scrolling and clicking allowed. (PS2) Scenario 1: nd a complete list of ight times for a given date. (S1) Scenario 2: update account details (street name + post code only). (S2) Scenario 3: nd the cheapest fare for a particular ight route on a given date. (S3) Scenario 4: set up a frequent yer account. (S4) Scenario 5: nd the maximum weight allowance for checked-in baggage. (S5)
All Scenarios have been given a priority rating: high (urgent or important x - recommend addressing immediately) medium (relatively important x - to be addressed only after high priorities are xed) low (least urgent/ important x, recommend addressing when time and resources permit)
Detailed Findings
Fig 1. Time to complete for tasks (S1-S5) *
U1
U2
U3
S1
1.53 1.21 0.57 3.37 3.01 2.58 1.35 2.14
S2
S3
3.48 6.05 3.12 8.26 8.11
S4
S5
1.38 2.02
Time in minutes
* Note: PS1 was not subject to time; PS2 was constrained by a 4-minute time limit by the facilitator - therefore these gures are not reected in the chart above.
1. Discussion of PS1 - Please give me your initial reactions to this (home) page. Feel
free to explore this page as you normally would. You can scroll around with your mouse, but please don't click on anything just yet. Measurements of PS1 are largely qualitative - users were asked for opinion related to the FlyBe home page layout, use of graphics, colours, and what element they would click on rst. user
U1 U2 U3
user comments
too much information design is old school - nothing new here page is cluttered - too many colours, too many boxes
rst click
Destinations link (in global navigation) Destinations link (in global navigation) Destinations link (in global navigation)
recommendations
Introduce a more minimal approach to the home page design, highlighting the most key areas. Users were often confused by the large amount of information and lack of creativity in design. Remove boxed content on page as these areas can accessed via the global navigation.
Users were asked to provide their initial response to the FlyBe home page.
Every user indicated they would click on the Destinations link rst in the global navigation when performing this exploratory task, strongly indicating that users come to the site for a single purpose. Users did not indicate a strong interest in discovering boxed content areas on the home page. All users seem to feel that the home page was visually uninteresting as well as overly populated with information. The post-test survey indicated that users did not nd the home page attention-grabbing; two users strongly agreed that pages in general contain too much information.
Fig 2. Users generally felt the FlyBe home page was too cluttered.
2. Discussion of PS2 - You have a few minutes to freely explore this Web site. You
may go anywhere you would like to go on the Web site, but please remember to speak aloud as you do so. I will tell you when your time is up. (allow 3-4 mins.) Users were almost exclusively interested in Flybe destinations and the cost of fares. While users were curious enough to mouse over the global navigation links at top (g 3) , only one user clicked on one of these links (Hotels) but then could not return to the main site without difculty (mini-site was displayed in a child window). Users did not access the links describing FlyBe corporate information, Benets, Car Hire, Hotel and the Site Map. User comments during this exploratory time include: I no longer know where I am! How do I get back to Home? (to the FlyBe main site) - U3 Theres too many things to click on. - U1
Fig 3. Users moused-over the global navigation but generally did not select links under these headings.
PS2
recommendations The site may benet from a reduction in the amount of text information and navigation links; unvisited links such as Site Map and About FlyBe could be conventionally located in the footer, thereby contributing to reduction of clutter and overloading the user with too many choices. Pop-up windows should not be used. Users seem focused on content related to destinations; visually weight the site in favour of more Destination content, presented with greater frequency.
3. Discussion of S1 - Youre planning to y from Aberdeen to Birmingham on 9

September - where would you go on the Web site to view a complete list of ights on that date? priority
low
issues
Users did not access the Timetable.
recommendations
All users located ight times from the booking panel on the home page and completed this task faster than all others.
Users do not access the complete list of ight times Expert Review item 1.3 indicates there may be an available. issue with displaying the Timetable in a child window. As no users accessed the Timetable, remove this link from the global navigation.
Fig 4. All users accessed the list of ight times via the booking panel from the home page.
4.Discussion of S2 - An account has been created for you to use on the site. Please
update your account details by changing your street address and post code. (provide
detail to user).
priority
medium
issues
The Find Address functionality spawns a c h i l d w i n d o w, a n d displays incomplete and confusing system feedback.
recommendations
Provide informative and accurate system feedback during the Find Address ow; the system generates an empty window - user did not know if the update had completed. The system fails to successfully update the user account using this functionality, requiring x. Remove blank window in this process, and return user to Account page. Display messaging at top of Account page that states status of account update (updated / not updated).
Fig 5. User updates contact details via Find Address functionality.
10
Fig 6. User updates contact details via Find Address functionality; system displays an empty page.
User comments: I dont know whats happening now - did it update? It says Done at the bottom.... Is it waiting for me to do something? - U1 It didnt work (account update) - Ill try it again. - U1 It didnt work again - I guess Ill update it by hand. -U1
11
5. Discussion of S3 - On the FlyBe web site, where would you go to nd the cheapest
fare for the ights on 19 September, from Aberdeen to Birmingham - returning the 19 October? priority
medium
issues
recommendations
Air fare booking functionality The lack of a calendar did not appear to be l a c k s a c o n v e n t i o n a l problematic, although further testing may be calendar embedded in the necessary. travel date selection process. U1 selected a system-prohibited travel date (return The system allows users to dates that precede the departure date); the system choose travel dates that should not allow the users to choose a return date cause the site to generate an that precedes the departure date. error. System should be more intelligent during the date selection process. For instance, if the user chooses the last day of a given month to depart, the system should display the following month at the top of the list when user selects the return date.
Fig 7. System generates an error message; can be prevented by providing logical date selection. 12
6. Discussion of S4 - You would like to ensure that you will earn the frequent yer
points for this ight - set up a frequent yer account using the email and password provided. priority
high
issues
recommendations
All users were unable to Simplify Frequent Flyer set-up process: complete the task in fewer than 4 attempts. provide the user the option to create a Frequent Flyer account during the time of initial Account Users choose circuitous registration (check box). paths which require m u l t i p l e c l i c k s , l i n k s the system should know when the user is logged appear below the fold. in, and pre-populate Frequent Flyer account form with user account details, including user email System requires address. Captcha can be removed completely if additional registration the user is already logged in under these process, even when user conditions. is logged in. Introduce descriptive page headings such as, A m b i g u o u s h e a d i n g s Frequent Flyer Account Registration. confuse users - unclear which account is being created. Captcha codes are unclear requiring users to enter them multiple times before succeeding. All users are required to supply a different email address than the one found within their Contact Details, confusing users. U2 was unable to complete the task due to poor descriptive language - U2 believed he had created an Frequent Flyer account by default, when he had not.
13
Fig 8. The site would benet from clear page headers; Captcha technology is often frustrating for users. 14
Fig 9. Users were confused by unclear instructions regarding Frequent Flyer account creation.
User comments: It says I dont need to do anything because I already have a customer account. - U2 Im thinking, like, I want to kill somebody. - U3 Im confused - dont know which account Im signing up for. - U3 If was doing this on my own, and not being tested, I would have left this site. This is just ridiculous. - U1
15
7. Discussion of S5 - You want to bring luggage along with on your trip to Birmingham,
but you want to make you sure your baggage doesnt exceed the maximum weight allowance. Find out what the maximum weight allowance is for checked-in baggage in Economy class. priority
medium
issues
recommendations
A l l u s e r s a c c e s s e d Remove the redundant Q&A link visible on all Q&A via the link in the pages. global navigation (Information -> Q&A). Enlarge text in the Q&A section; menus items are already presented in a scrollable, drop-down Text in the Q&A section menu, so loss of real estate will not be an issue is too small. of enlargement. Answers appear in bold Display Answers in order of most-viewed by type. users. No logical, meaningful Group questions and answers in meaningful a r r a n g e m e n t o f chunks, use meaningful header titles for Q&A Answers. page.
Fig 10. Users did not see/use the Q&A link available on every page.
Fig 11. Users found Q&A text too small; the all-bold style decreased readability.
16
User comments: Where is it? (the Answer). The text is so small - plus, its all in bold - I cant see what Im reading. - U1 I really feel information about baggage should be at the top of this list - not whether or not I can take Christmas crackers on board the ight. I mean, how often does that happen? Once a year? - U1 Is this bridesmaid dress question really the most important thing about baggage? Why is this rst in the list? - U3
Post-test Results
Survey results
In order to determine users over-all sense of satisfaction, users were asked to complete a survey using a likert scale [ranging from 1 (strongly disagree) to 5 (strongly agree)]. Users were asked to respond to 31 positive statements regarding their experiences on the FlyBe web site. Results are interpreted as percentages in order to represent users general opinions of the sites visual elements, graphics, use of colour, layout, ease of use and usefulness of content (refer to Appendix D for complete list of questions). Overall, approximately 48% of user responses (Fig 12) reected a negative opinion (1 & 2) regarding the use of visual design elements in the interface. U2 had the most favourable opinion of the site (Fig 14), with only 23% of responses falling within a negative range. U3 had the least favourable opinion of the site, as 61% of responses fell within a negative range (Fig 15). U1 and U3 responded negatively (1 or 2) to the sites usefulness of content. All users responded negatively (1 or 2) when asked if the sites information was easy to read. All users responded with a 3 with regard to ease of use of the sites navigation.
17
Fig 12: Total number of user responses.
31% 29% 19% 19% 1%

1 (strongly disagree - 19% ) 3 (undecided / ambivalent - 31% ) 5 (strongly agree - 1% )
Note: only one statement received a 5;
2 (disagree - 29% ) 4 (agree - 19% )
Question #27 : The site has a clear purpose: 5
(U3)
Fig 13: Total responses - U1.
31% 17% 17% 34%
1 - strongly disagree 3 - undecided / ambivalent 5 - strongly agree
2 - disagree 4 - agree
18
42% 35% 23%
35%
32% 3% 3%
26%
19
Interview results
Users were asked 11 post-test questions, samples of which appear in full below (see Appendix C for complete list of questions). Q1. What are your over-all impressions of the site? Frustrating experience, unclear information, too much information. - U1 Standard airline site, theres really nothing amazing here. - U2 It should be less cluttered, have something at the top or side to let you know where you are (labels, headings), it should to more intuitive. - U3
Q4. What are three things you like best about the site? The central list of destination options on the home page, the ability to compare cost of different travel dates on the same page, its easy to book travels dates. - U1 The name of the company, destination animation on the home page, all the options are available on the home page. - U2 The pages load quickly, the log-in is easy, it has good deals on fares. - U3
Q6. If you could make one signicant change to the site, what would it be? Tidy it up a bit - its visually too confusing. - U1 Fix the Frequent Flyer situation - its a disaster. - U2 Remove all the boxes on the home page. - U3
Q7. Would you return to this site on your own in the future? Why or why not? Yes - to compare pricing with other airlines. - U1 Yes - because its easy to choose ights, and they have lots of destinations. - U2 Yes - I travel a lot and would come here to compare pricing. - U3
Q11. Please comment on the Frequent Flyer account set-up process. It was completely demented; I would never spend that much time being told no. - U1 I was confused by the language, I went round and round trying to gure out how to set up an account, only to nd out that I already qualied for one. It should say more clearly that if you hold an account that youre already a Frequent Flyer member. - U2 Pretty straight-forward, but the system didnt give me good feedback about what was going on. - U3
20
Q12. Please comment on the content found on the FlyBe web site. I generally liked the content, found it to be fairly useful. - U1 Generally pretty concise and specic, fairly standard. - U2 The content is undifferentiated - it all looks the same. It was depressing. They could have some more pleasing colours - the colours are awful. - U3
Summary and Conclusion

While usability testing did not uncover any catastrophic issues with the FlyBe web site, testing did point to a range of issues which diminished usability. Testing indicated that the site would benet from a minimal home page design which is less cluttered and distracting, and uses greater focus to help users locate the sites primary functionality. Testing indicated that multiple paths to the same page was either confusing for users, or was unused and redundant. While users generally felt positively with respect to the site content (such as inexpensive fares and multiple destinations), users generally registered a negative response with regard to locating content and navigating the site. Users felt the site was encumbered by too much visual information that competed equally for attention, and also suffered from a lack of clear, easily readable text. Timed sessions indicated that users were able to successfully complete a task relatively quickly, although all users suffered from multiple errors and unclear system feedback when attempting to register a Frequent Flyer account. Unclear information prevented one user from completing this task. By implementing a handful of key measures - which are largely visual in nature - to address the usability issues mentioned above, the FlyBe web site could greatly improve its usability, improve user condence and increase the frequency users return to the site.
21
EXPERT REVIEW
Review Methodology
The site review was conducted by a single user on 31 August, 2008 and lasted approximately 4 hours. The site was evaluated against Jakob Nielsens standard usability guidelines or heuristics (1990). The evaluation was limited to review of the primary site functionality, and to top level navigation.
Detailed Findings
The following tables contain detailed ndings for a total of 20 usability issues. Each issue has been categorized by the following metrics: Severity - the level of confusion experienced by the user, or the failure of the system. Location - the page name and/or link on which the issue occurs. Issue - the usability issue uncovered, including examples as appropriate. Recommendations - the recommendation to be applied to the usability issue. A severity rating appears at the end of the report, (p 30).
VISIBILITY OF SYSTEM STATUS

# 1.1 severity medium location multiple pages issue recommendation
M a n y p a g e s l a c k a Add a header title at the visible header describing top of pages lacking the page name; the headers. users may not know what page she has arrived at. The site lacks visual cues Introduce distinctions in in the navigation global and local regarding user location. navigation, visual indicators differentiating the page the user is currently visiting from other pages.
1.2
medium
all pages
22
# 1.3
severity medium
location Timetable
http:// www.ybe.com/ timetableService/ timetable.do
issue Timetable launches in new window containing small table lacking all references to ybe.com including navigation and branding.
recommendation Display Timetable within standard page template of site, keeping navigation and branding visible.
Note: Including the Timetable functionality within the Booking panel on the Home Page could reduce this issue, and increase ease of use. This is also widely practiced among other airline sites. Curiously, this feature is embedded on the Contact Us page: http:// w w w. y b e h o t e l s . c o m / contact.php
1.4
medium
Timetable
http:// www.ybe.com/ timetableService/ timetable.do
Clicking on the information icon i launches a new, ill-tting browser window partially obscuring both closing functions at top and bottom, and unnecessarily displays scroll bars.
Display additional timetable information in parent window, possibly in opening panel using javascript.
CONSISTENCY AND STANDARDS

# 2.1 severity medium location multiple pages;
https://www.ybe.com/ en/rewards4all/ https://www.ybe.com/ cam/initFindBooking.do http://www.ybe.com/ ightInfo/1book.htm http://www.ybe.com/ map/default.htm http://www.ybe.com/ customerRelations/ index.do? layoutType=full
issue Site lacks language consistency between global navigation links and initial information appearing on corresponding pages. For example, Q&A in the global navigation links to a page entitled Support Form, and Careers links to a page with header of Introduction, etc.
recommendation Match labels of global navigation links with corresponding page labels.
23
# 2.2
severity medium
location Flybe Destinations

http://www.ybe.com/ destinations/default.htm
issue
recommendation
Header appears in red block C h a n g e t o s i t e background with smaller standard blue block text. header, with larger white text. There is a visual relationship between the upper navigation icons (top right) and the colored bounding boxes for each section on the Home page; users may believe think that boxes encircled in purple may relate to Agent Log-in, or that boxes in yellow may relate to either My Flight or Frequent Flyer icons. Potentially confusing. Remove colored bounding lines from boxes on Home Page.
2.3
medium
Home Page
2.4
medium
Hotel page
http://www.ybehotels.com/ home.php? &languageCode=en&count ryOfOrigin=IE
Clicking Hotel links takes Add navigation option user outside main site to to return to main hotel mini site, does not FlyBe site. provide option to return to main Flybe site. breaking site convention; an Place an asterisk (*) asterisk appears by all by the phone number required elds, but not by options. the phone numbers - one of which is required. Users may scan only for required elds for speed, missing this option forces the page to reload. Redundancy in navigation R e m o v e links: Q&As and FAQs redundancies; either link to the same page. remove links in Also true for Car Hire, Hotel navigation, or remove and FlyBe Credit Card links redundant sections and corresponding sections on on home page (the
Home Page.
global navigation could be distilled down to fewer options, as much content are available below in boxed areas).
2.5
medium
Account registration
https://www.ybe.com/cam/ initAccountRegistration.do
2.7
low
Home Page
2.8
low
Hotel page
http://www.ybehotels.com/ home.php? &languageCode=en&count ryOfOrigin=IE
Localization options include Remove Ireland as an Ireland, but uses English option, unless Gaelic language. is used.
24
# 2.9
severity low
location Home Page
issue
recommendation
The FlyBe logo on the top Deactivate the logo left of the page is clickable, link. so users may think that they are not already on the home page.
LANGUAGE / MATCH BETWEEN SYSTEM AND REAL WORLD

# 3.1 severity medium location Timetable
https://www.ybe.com/ timetableService/ timetable.do
issue Additional Timetable information is presented as an error message in the browser, is written unclearly and chunked into confusing sections. Technical text appears rst, followed by clunky explanatory text. The Destination pages include an empty box at bottom entitled Other Destinations, which is contrary to the lengthy list of destinations displayed on the Site Map. Users are required to click Back to the Site Map to view list of destinations.
recommendation Display Timetable within standard page template of site, keeping navigation and branding visible. Use clear explanations of timetable. Remove technical language. Allow users to select other destinations from the Other Destinations box; perhaps populate this box with similar destination as the one initially selected by user
(for example, if the user selected Birmingham, then display all UK destinations).
3.2
medium
Destination pages
http://www.ybe.com/ destinations/Belfast.htm
If no additional destinations can be added here, then remove the empty box from each page.
25
# 3.3
severity medium
location Q&A
http://www.ybe.com/ customerRelations/ showAnswer.do
issue The phrase ...check the top Q&A below... is confusing because of the two locations Q&A and FAQs. Also, top and below used in close proximity can be confusing. The term Information in the global navigation is too general and uninformative. The term Outbound in the booking functionality is slightly technical in avour, and less conventional across the industry.
recommendation Rephrase to a less confusing statement, such as, check the rst answer below.
3.4
medium
Global Navigation
Re-name this label, or provide additional adjective, for example, Passenger Information Substitute the less conventional term Outbound for the more widely used term Departure (as used on
major airline sites such as British Airways, Virgin Atlantic, Air France Lufthansa).
3.5
low
Home Page
USER CONTROL
# 4.1 severity low location all pages issue recommendation
The site does not allow Provide search functionality search functionality, on each page. which many users prefer/are used to vs. browsing (most major
airline sites offer search Britishairways, Lufthansa, Virgin Atlantic, United).
26
ERROR PREVENTION
# 5.1 severity high location Home Page
http://www.ybe.com/ default.htm
issue Booking system allows user to select a Return Date previous to the Outbound Date. An error message informs user of the anachronous return date.
recommendation Allow the system to prohibit the selection of dates which pre-date the outbound date. For example, if the user chooses 31 Aug, the drop-down in the Return Date should only display from September. The system should recognize if the user is logged in, use the account information it already holds, eliminating this unnecessary and repetitious registration step.
5.2
high
Rewards4All
https://www.ybe.com/ en/rewards4all/
The site does not use user account log-in information in the Rewards4All sign-up process, and requires users to complete another form, using different email and password combinations to their original account log-in details; requiring two sets of log-in details will greatly deter users.
5.3
medium
Account Registration
https://www.ybe.com/ cam/ initAccountRegistration. do
The password security Allow more exibility in requires a combination the choice of password. of capital and lowercase a l p h a n u m e r i c characters. The reviewer used a lowercase alphanumeric 10-digit password, which the system rejected. All links appearing in left Remove links or grey-out margin are active but not and deactivate links. clickable; user is presented an error message prompting user to log-in to access these links. The system provides No recommendation r e a l - t i m e e r r o r - needed. prevention messaging to the user when lling out forms.
5.4
low
Log-in
https://www.ybe.com/ cam/login.do
5.5
Account Registration
https://www.ybe.com/cam/ initAccountRegistration.do
27
HELP AND DOCUMENTATION

# 6.1 severity medium location Q&A / FAQS issue The FlyBe site provides FAQ information, but it is presented in very small font and cluttered closely together. The answers do not seem to be weighted in a logical order. recommendation Increase font size, and improve readability layout; answers should be displayed in order of most frequently asked by users.
ERROR RECOVERY
# severity location Rewards4All
https://www.ybe.com/ cam/ processAccountRegistr ation.do
issue The site does not provide adequate error recovery during the Frequent Flyer sign-up process. The process is complex and error messaging is not wellplaced. As a result, error recovery is poor.
(see Error Prevention)
recommendation Ideally, remove this task entirely and provide users the option to sign up for the Frequent Flyer program (as a simple check box) during initial Account Registration. Otherwise, if a user is logged-in, pre-populate the form with user account information - remove the Catcha functionality. Display subsequent error messaging at the top of the form, as many implicated input elds appear below the fold. If the user has not logged-in, then require login and then follow above.
7.1 critical
28
AESTHETIC AND MINIMAL DESIGN

# severity location all pages issue recommendation 8.1 high The FlyBe site is blighted R e d u c e a m o u n t o f by an o v e r - a l l horizontal bars throughout cumbersome sense of the site. visual design; an over abundance of coloured horizontal bars for navigation and headers make the site difcult to scan. The use of primary colors throughout is unfortunate, although this may be difcult to change as the logo incorporates these colours. Over-population; poor Include more white space emphasis and focus of t o c l u t t e r e d p a g e s , v i s u a l e l e m e n t s . A l l especially the home page. elements - barring the Booking panel - seem to be weighted the same in terms of importance. The use of red is distracting and is used for different categories/labels. The use of yellow/gold Change either background text on a red background color, or text color to a is difcult to read. combination which reduces visual stress and adds contrast.
8.2 medium
Home Page
8.3 low
Destination pages
https:// www.ybe.com/ destinations/ default.htm
29
ACCESSIBILITY
# 9.1 severity high location all pages issue recommendation The FlyBe site is not review HTML to remove HTML standards compliant validation errors; add alt a n d s u ff e r s f r o m 2 0 5 tags for images. validation errors in its coding. Other accessibility issues exist such as an absence of <alt> tags for images in the code. However, CSS validation was successful.
all pages
Severity Rating
(each rating can include one or more of the following denitions) Critical An emergency condition that causes the customer's system to fail or causes customer data to be lost or destroyed; A catastrophic usability issue that is likely to cause frequent data integrity errors; A key feature needed by many customers is not in the system; There is no work-around to these problems; High A serious condition that impairs the operation, or continued operation, of one or more product functions - this can be avoided, but only with difculty; The software does not prevent the user from making a serious mistake. The usability problem is frequent, persistent, and affects many users. There is a serious violation of standards. Medium A non-critical, limited problem (no data lost or system failure). Moderate problem nding a work-around; The problem causes users moderate confusion or irritation. Low Minor questions and/or confusion about the product; There are minor inconsistencies that cause hesitation or small aesthetic issues like labels and elds that are not aligned properly; Cosmetic issues; Minor irritation to users, work-around easily found.
30
*
Noteworthy, but a non-issue; does not require a x nor recommendation.
31
Appendices
Appendix A: Pre-test Questionnaire
General
Name of participant: Gender: M / F Which of the following best describes your age? [18 to 25 26 to 39 40 to 59 60 - to 74 75 and older]
Ethnicity / please specify ________________________________
Professional Demographics
What is your current position and title? How long have you held this position? Which of the following best describes your work environment? [commercial business, nonprofit, government agency, self-employed, other _______________ ] Which of the following describes your highest level of education? [GCSE or equivalent, some university, university graduate, postgraduate (MA/PhD), other ________________________
Computer Expertise
Do you use a computer? Y / N (if N, stop) Besides read email, what are typical activities you do on the computer? Approximately how many hours per week do you spend on the computer? 0-10hrs 11-25hrs +26hrs What computer platform do you usually use? [e.g. Mac, Windows XP, etc.] What Internet browser(s) do you usually use? [e.g., Firefox, IE v.__, AOL, etc.] other___________ During the last year, how often have you made purchases from a Web site? If yes, what types of items have you purchased?
Is there anything else you use the internet for?
32
Appendix B: Audio / Video Release Form
Usability Testing - Video/Audio Release Form
I agree to participate in this usability study conducted and recorded by Luke Bishop, for educational purposes in conjunction with coursework at Qantm College, London. I understand and consent to the use and release of the recorded material by Luke Bishop. I understand that the information and material is for educational purposes only and that my name and image will not be used for any other purpose. I relinquish any rights to this material and understand that it may be copied and used (limited to educational purposes only) by Luke Bishop without further permission. I understand that I can leave at any time. I agree to immediately raise any concerns or areas of discomfort with the study administrator (N/A).
Your signature:____________________________________________________
Date:______________________________________________________________
Please print your name: ______________________________________________
Thank you!
33
Appendix C: Post-test Interview
1. What are your overall impressions of the FlyBe Web site?
2. If you had to give the site a grade, from A to F, where A was exemplary and F was failing, what grade would you give it, and why?
3. Name three words or characteristics that describe this Web site. 4. What are the three things you like best about the Web site?
5. What are the three things you like least about the Web site?
6. If you could make one signicant change to this Web site, what change would you make? 7. Would you return to this Web site on your own in the future? Why/why not?
8. What would entice you to return? 9. Are there materials you would like to see added to the Web site? Which ones?
10. Would you recommend this Web site to a colleague? To a friend?
11. Please comment about the Frequent Flyer account set-up process.
12. Please comment about the content found on the FlyBe Web site. 13. Are there any additional comments youd like to make?
34
Appendix D: Post-test Survey

Strongly disagree
1 2 3 4
Strongly agree
5
For our last activity I'm going to give you a short survey of statements about the FlyBe Web site. I would like you to rate your agreement with each statement on a scale of 1 to 5, 1 as strongly disagree with the statement, and 5 as strongly agree with the statement. Please write your response at the end of each statement. 1. The homepage is attractive. 3 - 4 - 1 2. The overall site is attractive. 3 - 3 - 2 3. The site's graphics are pleasing. 2 - 3 - 1 4. The site has a good balance of graphics versus text. 4 - 3 - 1 5. The colors used throughout the site are attractive. 2 - 4 - 1 6. The typography (lettering, headings, titles) is attractive. 2 - 4 - 3 7. The home page's content makes me want to explore the site further. 2 - 2 - 2 8. It is easy to find my way around the site. 4 - 3 - 3 9. I can get to information quickly. 1 - 3 - 3 10. It is fun to explore the site. 1 - 3 - 2 11. It is easy to remember where to find things. 4 - 2 - 2 12. Information is layered effectively on different screens. 1 - 3 - 2 13. The homepage is attention-getting. 2 - 3 - 2 14. Information is easy to read. 1 - 4 - 2 15. Information is written in a style that suits me. 1 - 4 - 2 16. Screens have the right amount of information. 1 - 3 - 1 17. The site effectively communicates the company's identity. 1 - 2 - 3 18. The information is relevant to my professional needs. 2 - 4 - 3 19. The site is designed with me in mind. 2 - 3 - 2 20. The site's content interests me. 4 - 2 - 2 21. The site's content would keep me coming back. 4 - 2 - 2 22. The site has characteristics that make it especially appealing. 1 - 3 - 1 23. The site reflects progressive, leading edge design. 1 - 2 - 1 24. The site is exciting. 1 - 2 - 1 25. The site is well-suited to first-time visitors. 3 - 4 - 3 26. The site is well-suited to repeat visitors. 3 - 4 - 3 27. The site has a clear purpose. 3 - 4 - 5 28. It is clear how screen elements work (pop-ups, scrolling lists, menu options). 3 - 4 - 3 29. My mistakes were easy to correct. 2 - 3 -3 30. The site was easy to navigate. 3 - 3 -3 31. I could find what I was looking for easily. 3 - 4 - 4
[ U1 - U2 - U3 ]
35
36 - Scenarios
Pre-Scenario 1
This is the homepage of a company called Flybe. Please give me your initial reactions to this page. Feel free to explore this page as you normally would. You can scroll around with your mouse, but please don't click on anything just yet. Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, photos, etc.
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Have you ever seen this Web site before? Y / N Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, visual elements.
Without clicking on anything yet, if you were exploring, what would you click on rst?
Pathway(s)
Success (Circle 1) 0 Not completed 1 Completed with difficulty or help 2 Easily completed
Notes/Observations
36
Pre-scenario 2
I'm going to give you a few minutes to freely explore this Web site. You may go anywhere you would like to go on the Web site, but please remember to speak aloud as you do so. I will tell you when your time is up. (allow 3-4 mins.)
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
37
Scenario 1
Youre planning to y from Aberdeen to Birmingham on 9 September - where would you go on the Web site to view a complete list of ights on that date?
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
38
Scenario 2
An account has been created for you to use on the site. Please update your account details by changing your street address and post code (use the following account details): username / email - info@lukebishop.com password - Testing123
When you feel you have completed this task, please say so.
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
39
Scenario 3
On the FlyBe web site, where would you go to nd the cheapest fare for the ights on 19 September, from Aberdeen to Birmingham - returning the 19 October? When you feel you have completed this task, please say so.
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
40
Scenario 4
You would like to ensure that you will earn the frequent yer points for this ight - set up a frequent yer account using the email and password I provide you. username / email - info@lukebishop.com password - Testing123
When you feel you have completed this task, please say so.
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
41
Scenario 5
You want to bring luggage along with on your trip to Birmingham, but you want to make you sure your baggage doesnt exceed the maximum weight allowance. Find out what the maximum weight allowance is for checked-in baggage in Economy class. When you feel you have completed this task, please say so.
User comments: ______________________________________________________________________________________ ______________________________________________________________________________________
Pathway(s)
Notes/Observations
42
References
1 http://www.blackartofwebpublishing.com/FPatternHotSpots
Facilitator Test Script

part I : Intro Hello, (User), Thank you for taking part in this usability test. This test consists of three parts: a background questionnaire and signing a consent form, the usability testing of a web site, and a short survey of your experience using the site at the end. The entire process should take about an hour to complete. It is important to remember that we are not testing youwe are testing the site itself, so there is no right or wrong for you. The audio and video of the session will be recorded, this information will be used for the purposes of this study only and will not appear anywhere else. If you have any questions at any point during the process, it is perfectly ok to ask, but try to engage with the site as you naturally would on your own. Do you have any questions before we start? > (consent form rst) > (pre-questionnaire)
Part II : site test Thank you. Now we will begin the site test. I will ask you to perform 7 tasks to do one at a time. While you are performing each task, I will make a few notes and merely observe, not intervene or comment. Remember, Id like you to try to speak aloud describing what youre doing, why youre doing it, what you are thinking as you are performing each task. Do you have any questions before we begin? > (Post-test interview) > (Post-test survey)
Closing thank you.
43

Usability Doc Lukebishop

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Usability Doc Lukebishop

Uploaded by

Copyright:

Available Formats

Usability Test Report Document / www.ybe.

com / September 2008

Usability Test Report Document / www.ybe.com

September 2008 Qantm College, London Interface Design Amir Dotan

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

revealed the following user

hours on computer p/week

frequent yer member

Nursery teacher Interior designer Advanced Rolfer

University graduate University graduate Post-graduate

0-10 hrs 11-25 hrs +26

types of computer activity

User demographics collected from the pre-test questionnaire.

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

1.53 1.21 0.57 3.37 3.01 2.58 1.35 2.14

3.48 6.05 3.12 8.26 8.11

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

3. Discussion of S1 - Youre planning to y from Aberdeen to Birmingham on 9

Usability Test Report Document / www.ybe.com / September 2008

Fig 5. User updates contact details via Find Address functionality.

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Fig 12: Total number of user responses.

31% 29% 19% 19% 1%

2 (disagree - 29% ) 4 (agree - 19% )

Question #27 : The site has a clear purpose: 5

Fig 13: Total responses - U1.

31% 17% 17% 34%

1 - strongly disagree 3 - undecided / ambivalent 5 - strongly agree

Usability Test Report Document / www.ybe.com / September 2008

Fig 14: Total responses - U2.

42% 35% 23%

1 - strongly disagree 3 - undecided / ambivalent 5 - strongly agree

Fig 16: Total responses - U3.

1 - strongly disagree 3 - undecided / ambivalent 5 - strongly agree

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Summary and Conclusion

Usability Test Report Document / www.ybe.com / September 2008

VISIBILITY OF SYSTEM STATUS

Usability Test Report Document / www.ybe.com / September 2008

CONSISTENCY AND STANDARDS

Usability Test Report Document / www.ybe.com / September 2008

location Flybe Destinations

Usability Test Report Document / www.ybe.com / September 2008

location Home Page

LANGUAGE / MATCH BETWEEN SYSTEM AND REAL WORLD

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

Usability Test Report Document / www.ybe.com / September 2008

HELP AND DOCUMENTATION

Usability Test Report Document / www.ybe.com / September 2008

User comments: