Web browser usability test

Lim Zhen Ling Samantha

Ng Yan Ting
Wong Tak Wee
 Google chrome
• Developed by Google
• Released in September 2008
• Cool features
– Minimalistic design
– Rearrange tabs in different
windows by drag and drop
– No need to delete history:
Incognito mode
– Google Suggest: All-in-one
address bar
– No need for download manager:
All-in-one browsing tab
– Top 9 web sites: Convenient
access to your recent/favourite
web pages
 Google chrome
• Developed by Google
• Released in September 2008
• Cool features
– Minimalistic design
– Rearrange tabs in different
windows by drag and drop
– Privacy: Incognito mode
– Google Suggest: All-in-one
search, history and address bar
– No need for download manager:
All-in-one browsing tab
– Top 9 web sites: Convenient
access to your recent/favourite
web pages
– Highlighted textboxes
 web browser usability testing

• Objectives
• Pre-testing assessment
– Assessment criteria
– Cognitive walkthrough
– Pilot test
• Tasks selection: Final revision
• Procedure
• Results
• Limitations
• Improvements
• Conclusion
 Our objectives
• Improve usability of Google Chrome web
browser for current and future users
• Create positive user experience
 Our objectives
• Improve usability of Google Chrome web browser
for current and future users
• Create positive user experience

• Objectives of HCI research (Shneiderman, 1987)

↓ time of learning
↓ rate of errors
↑ retention over time
↑ user satisfaction
 Web browser assessment criteria
• Web performance
• Security
• User interface design/Usability
 Web browser assessment criteria
• Performance
• Security
• User interface design/Usability
– Intuitiveness
– Convenience
– Accessibility
 Cognitive walkthrough: Intuitiveness
Decreased learning time and
visibility of functions
• Consistency
• Informative feedback
• Direct manipulation of
system
• Interface real estate
management
• Overall look and feel
 Cognitive walkthrough: Intuitiveness
Decreased learning time and
visibility of functions
• Consistency ×
– With older, more familiar
web browsers
Inactive mode

Active mode

√ Minimalist approach  overloading of function buttons  × violate

intuitiveness
 Google Chrome

Internet explorer

Firefox

√ Minimalist approach  overloading of function buttons  × violate

intuitiveness
 Cognitive walkthrough: Intuitiveness
Decreased learning time and
visibility of functions
• Consistency To open/close
– With older, more familiar bookmarks bar:
web browsers Right click

√ direct manipulation with the system inconsistent layout and function × low
‘visibility’ violate intuitiveness
Cognitive walkthrough: Intuitiveness
 Cognitive walkthrough: Convenience
• Shortcuts
• Easy reversal of actions
• Reduce short term memory
load
• Error prevention and
solving
• User locus of control
 Cognitive walkthrough: Convenience
• Shortcuts
• Easy reversal of actions
• Reduce short term memory
load ×
• Error prevention and
solving
• User locus of control

Minimalist design  reorganisation of functions  require learning  ST memory

load increases
 Cognitive walkthrough: Accessibility
• Important or commonly
used functions
– Find “Help”
 Cognitive walkthrough: Accessibility
• Important or commonly
used functions

Minimalist design  remove redundant coding  search time increases

Minimalist design  remove redundant coding  area of access decreases  time
needed for selection increases
 Cognitive walkthrough: Accessibility
• Important or commonly
To close
used functions bookmarks bar:
• Bookmarks bar Right click

To open
bookmarks bar:
Right click

Small area of access  violate Fitts’Law  time needed for selection increases
Cognitive walkthrough  pilot testing  finalise tasks

Tasks Selection : Final

 Task 1
Go to http://www.hardwarezone.com, and
stop the page before it finishes loading.
Press the Esc button on the keyboard
Click on the button
× Low visibility
Task 1
 Task 2
Close the bookmarks bar, and after you have
done so, show the bookmarks bar again.

Right-click on the bookmarks bar to close

 right click on the slit to open
× Fitt’s Law

Go to “Tools”
Task 2
Task 2
 Task 3
Make and export a bookmark

Drag URL address to bookmarks bar  go to

“Bookmarks manager”  go to “Tools”
 click on “Export bookmarks”
Click on star icon to bookmark
Go to “Tools”  click on “Export bookmarks”
× accessibility - muted suggests inactivity; low visibility 
Task 3
Task 3
 Task 4
Save a web page on to the desktop
Select “Document”  “Save page as…”
 Check for file on desktop
 Task 4

Mozilla Firefox
Internet Explorer
 Task 5
Locate the “Help” option.
Go to “Tools”  Select “Help”
Task 5
Participant Profile & Procedures
 Participant Profile
• 6 participants
• 3 males & 3 females
• 1st Time/Non-Google
Chrome Users
• NTU Undergraduates
• 20~26 years old
• Basic level of computer
and internet expertise

Tasks identified are generally common functions used

by participant group.
 Procedures
• Venue
• Tutorial Room 111 and 113
• Quiet, controlled environment
Venue
Venue
 Equipment

• Acer Aspire 4710

• Google Chrome Version 1.0
• MORAE 3
• Debut Video Capture Software
• Sony Cyber-shot Dsc-s600
• www.hardwarezone.com
• Pre/Post-test Questionnaires
• Video Release Form
 Observation Methods
• MORAE 3
– Screen Activity, Mouse Clicks
• Debut Video Capture Software
• Notetaker’s Guide
– Navigation Pathways, Facial Expressions, Success/Failure etc.
• Questionnaire/Feedback Form
– Likert Scale
– Open-ended
feedback
RESULTS
 Findings
Success ratings
  Task 1 Task 2 Task 3 Task 4 Task 5

M1 2 2 1 2 2
M2 2 2 0 2 2
M3 2 0 0 2 2
F1 2 0 0 2 2
F2 1 1 2 1 2
F3 1 2 0 2 2

*Success rating: 0 – Not completed, 1 – Completed with difficulty or help, 2 – Easily completed
 Findings
• Task 3 > Task 2 > Task 1 > Task 4 > Task 5

• Highest completion rate: Task 5

– All participants completed it with ease

• Lowest completion rate: Task 3

– Only 2 participants completed it
• Only 1 of them completed it with ease
• The other completed with some difficulty or help
 Findings
Difficulty ratings
 Findings
• Emotional responses provide information about
the tasks too
– Whether MATCH with self-reported difficulty ratings
AND success rates

• Participants were confused by some of the tasks

– Especially Task 2
– And also Task 1 and 3
 Results analysis
• By:
– Task design – how it affects the performance and
success ratings

– User expectations – how user experience and

knowledge affects performance and success
ratings
 Results analysis
• Task design
– Certain tasks inherently more complicated –
demands a longer action sequence
• E.g. Tasks 2 & 3
– Other tasks are much simpler – shorter action
sequence of 1 or 2 steps only – better
performance
• E.g. Tasks 1, 4, & 5
 Results analysis

Mean success ratings

Participant Task 1 Task 2 Task 3 Task 4 Task 5
ID

Mean 1.666667 1.166667 0.5 1.833333 2

success
rating
 Results analysis
• User expectations
– Mental models developed from use of their
familiar web browsers
• E.g. Internet Explorer, Mozilla Firefox, Safari

– Mental models would then affect performance of

tasks on Google Chrome
• The smaller the discrepancy between Google Chrome
and familiar web browser, the better the performance,
the lower the difficulty rating
 Results analysis
Participants’ familiar web browsers and Difficulty ratings
Difficulty rating

ID Familiar web browser Task 1 Task 2 Task 3 Task 4 Task 5 Mean

M1 Firefox 1 2 4 1 2 2
M2 IE 1 2 3 1 1 1.6
M3 IE 3 5 5 2 2 3.4
F1 Safari 3 5 5 2 3 3.6
F2 Firefox 2 2 1 3 1 1.8
F3 IE 3 4 5 1 1 2.1667
Average – Firefox 1.5 2 2.5 2 1.5 1.9
Average – IE 2.3 3.7 4.3 1.3 1.3 2.6

P.S. Safari is left out of the comparison due to its much lower % of use among web browser
users.
 Results analysis
• Task 1: Stop a web page from loading
– Location of X button
• Firefox - Left
• IE and Google Chrome - Right

– Consistency for IE users: similar location

• IE users can more readily identify and navigate around
that area for “Stop’ option
 Results analysis
• Task 1: Stop a web page from loading
– Difference between IE and Google Chrome
• X button doesn’t double up as the load button,
coloured in red (redundancy effect)

– IE users perform better?

• No!
• 2 participants who completed the task with difficulty or
help
– 1 used Firefox, 1 used IE
 Results analysis
• Task 1: Stop a web page from loading
– IE closer to Google Chrome
• IE users SHOULD give lower difficulty ratings
• IE users’ average: 2.3
• Firefox users’ average: 1.5
• Firefox users give lower difficulty ratings INSTEAD!
 Results analysis
• Task 1: Stop a web page from loading
– IE users:
• NO better performance
• NO lower difficulty rating
– Explained by:
• Pressing “Esc” button can also stop a page from loading
– Hence IE users who employ that action sequence, instead of
clicking on “Stop” button, would not be expected to perform
better because of consistency in design layout
 Results analysis
• Task 2: Open / close bookmarks bar
– Bookmarks bar
• Firefox: right-click on top options toolbar, and
check/uncheck “Bookmarks toolbar”

• IE: no bookmarks bar/toolbar!

 Results analysis
• Task 2: Open / close bookmarks bar
– Firefox users are expected to have an advantage
since of the presence of similar “Bookmarks
toolbar”
• No! - 2 users of Firefox and 2 users of IE completed the
task
– 1 of the users of Firefox completed it with difficulty or help.
 Results analysis
• Task 2: Open / close bookmarks bar
– Firefox closer to Google Chrome
• Firefox users SHOULD give lower difficulty rating
• Task 2
– Firefox users’ average: 2
– IE users’ average: 3.7
• Firefox users INDEED give lower difficulty ratings
 Results analysis
• Task 2: Open / close bookmarks bar
– Firefox users:
• Similar performance as IE users
• Lower difficulty ratings
– Explained by:
• Similar layout design – consistency
– Firefox users recognize Google Chrome’s bookmarks bar and
its associated functions more easily
 Results analysis
• Task 3: Make and export bookmark
– Action sequence
• Firefox
– Make bookmark: click on muted star
– Export bookmark: go to “Bookmarks” option ”Organize
Bookmarks” ”Import and backup”  “Export HTML”

• IE
– Make bookmark: click on yellow star with a plus sign  “Add
to favourites”
– Export bookmark: click on yellow star with a plus sign 
“Import and Export…”
 Results analysis
• Task 3: Make and export bookmark
– Google Chrome: action sequence similar to Firefox’s
• Make bookmark: click on muted star
• Export bookmark: go to “Tools” button  go to “Bookmarks
manager” highlight/click on bookmark  click on “Tools”
option  select “Export bookmarks”

– Only 2 participants who managed to complete the task

• Both Firefox users
 Results analysis
• Task 3: Make and export bookmark
– Firefox closer to Google Chrome
• Firefox users SHOULD give lower difficulty rating
• Task 3
– Firefox users’ average: 2.5
– IE users’ average: 4.3
• Firefox users INDEED give lower difficulty ratings
 Results analysis
• Task 3: Make and export bookmark
– Firefox users:
• Better performance
• Lower difficulty ratings
– Explained by:
• Similar action sequence – consistency
– Firefox users hence more familiar with task
 What are the things you liked about
Google Chrome?

“Sleek look, nice aesthetics..”

“Looks sleek and tidy…”

“…look sleek and clean..”

“Faster speed…”
“Stylish and simple in design…”
 What are the things that you did NOT
like about Google Chrome?

“No gadgets, add-ons..”

“Too difficult to find various functions…”

“…icon not coloured…”

“Slightly cumbersome in its usability, especially with regards to bookmarks..”

 Feedback
• What people liked:
– Mainly looks, aesthetics
– Level of emotion: Visceral

• What people disliked:

– Navigation, usability
– Level of emotion: Behavioral
Would you recommend Google
Chrome to your friends?
3

2.5

1.5

0.5

0
Yes No Maybe
 Usability criteria
• Intuitiveness
– Consistency
– Feedback
– Overall look and feel

• Convenience:
– Shortcuts
– Easy reversal of actions
– Reduce cognitive load
– User locus of control: error prevention and feedback

• Accessibility
 Scope and severity ratings
• Scope
– Local problems
– Global problems
• Severity
1 - prevents completion of a task
2 - causes significant delays in completing a task
3 - causes minor usability problems, but users can complete
the task
4 - minor annoyance - does not significantly impact usability,
but should be corrected if time allows.
Tasks Usability Problems No. Of participants Severity Scope
affected

2, 3, 4, 5 Participants face difficulty in completing tasks 4 1 Global

associated with options classified under
“Documents” and “Tools”
2, 3, 4, 5 Classification of task options was not intuitive 4 1 Global
i.e. participants could not find the export
bookmark option and the functions for
bookmarks bar under bookmarks manager
3 Participants could not find the export 4 1 Local
bookmarks option
1, 2, 3, 4, 5 Web browser window closes without warning 3 2/4* Global

2 Participants were unaware that bookmarks bar 6 2 Local

can be closed and opened via right click

3 Participants could not identify bookmarking 4 3 Local

option
1 Participants could not find the stop button 2 3 Local
without prompt
2 Participants could not identify the bookmarks 6 3 Local
bar without prompt

1 - prevents completion of a task

2 - causes significant delays in completing a task
3 - causes minor usability problems, but users can complete the task
4 - minor annoyance - does not significantly impact usability, but should be corrected if time allows.
* Depending on the task user was performing
4 out of 6 of the participants felt that Google Chrome’s minimalist design
(larger web pages) improved their web browsing experience

 retain its
minimalist design
Improve on its
existing features
based on

RECOMMENDATIONS
feedback and
current usability
problems
Reorganisation of options
Reorganisation of options
Direct manipulation of bookmarks bar
Redundancy coding for “Documents” and “Tools”
option
 Google Chrome

Open these the next time I use Google Chrome

Do not show me this dialog again

Alert window
Bookmarks bar button/options
Stop button
CONFOUNDS AND LIMITATIONS
 Limitations/suggested changes
• Participant selection • Repeat usability testing
– Small sample size – in a different population
– Population stereotypes
• Asians
• University undergraduates
– Not representative of all users – reorder task sequence
– Include practice tests prior to
• Practice effect actual testing
• Testing environment and
procedure – Naturalistic environment
– Discrepancy with the real world
• Computer system
• Usage conditions
– Presence of test administrator
– Obtrusive equipment
 Web browser evaluation
√
 User performance test
 Feature test
 Web compatibility test
 Configuration test
 Security test
CONCLUSION
 • Version tested: early stage of
development
• Minimalist design aimed at the
everyday user
– Differs from Chromium
– Relatively sound product
• Make adjustments to accommodate
users accustomed to browser giants
IE and Firefox
• Google Chrome team: make
adjustments as we go along
Google Chrome with 3D – Google Chrome Blog
– Google Chrome Beta
– Google Chrome with 3D