Test Republic Testing Challenge:: June-July 2009

Name:
Simon Morley

Email:
xxxxxxxxxxxxxxxxxxxxx

Contact no:
xxxxxxxxxxxxxxxxxxxxx

Test Republic name:

Simon Morley

Bug reports (for the above test challenge) are contained on the following pages. Notes are
included in the page following the problem reports. The last pages contain the "raw" notes made
when executing the test.

Time spent on the initial bug investigation: 20 minutes + 15 follow-up / reproduction later
Time spent on bug report writing: 3 hours (majority of time spent due to not using a pre-formatted
report template and review of the report – worthwhile as demonstrated in one of the final notes.)
Issue Report ID: SM00001
Report Heading: Dialogue box for location gives no context for the entry
details

Configuration:
Client-side: HP Compaq nx7000
Windows XP Home Ed version 5.1.2600 SP3
Web browser: Google Chrome 2.0.172.33
Server-side: Web site: http://maps.google.com
Version: Not available

Problem location: Web site: http://maps.google.com

Report author: Simon Morley

Report observation date: 30 June 2009
Report submitted date: 17 July 2009

How/When found: Review of first page of web site.

Reproducible: Yes

Severity: Small (non critical)

Customer impact:
Incorrect results returned. (Potential to enter data in the customer context – not
necessarily the context expected of the web site.)

Description:
On the start page for the site (http://maps.google.com/) there is no indication of the
format required for the entry of the required location.

Temporary solution:
None

Suggested solution:
Introduce a roll-over to give the possibilities for data entry. Include a note to state that the
default location/map on the first/start page is the default country context, i.e. if another
country than the default is intended for search then this should be entered.

Indicate the formats that are recognised: country-specific postal codes, GPS coords, etc.

Expected design & verification cost:

 Minimal-Medium depending on the implementation.
 Minimal: Introduction of a roll-over with basic information for data entry.
 Medium: Tweaks to the page to include additional help symbols/buttons.

Expected benefit to customer:

 Enhanced comfort/security about using the site - especially first-time users
 More return users instead of searching for alternative map sites.
 More recommendations for the site.

Expected benefit to vendor:

 Reduced number of "how-to" questions to vendor
 Enhanced "reputation" for accessibility and usability to a wider audience - wider age and
demographic range of users (e.g. allowing for web newbies / beginners)
Notes on report ID: SM00001-2

1. The first thing I noticed on the page after entering the address was that the map
appeared over the USA. I wanted to try an address in the UK and had one place to enter
this in “Set default location”. BUT, what format should I use? I know that UK post codes
and US zip codes are different, but would the site know this?

2. I guessed that it might – but hadn’t tested this yet. I went back one step – suppose that I
was completely new to using map websites and had never been to the USA I would just
enter the postal code as I was used to using it.

3. So, my first set of tests was about entering postal codes (valid and potentially invalid) in
“raw” format – without any country identifier…

4. I tried a valid UK postal code then followed it with an invalid UK postal code (maybe valid
somewhere else in the world)

5. I did the same with some Swedish postal codes (without any country context) – bingo! I
got an unexpected result.

6. This then led me to think, “Why postal codes?” The page is not giving me any context
information about how the data should be entered and is making assumptions about what
I want to know – which I managed to get the wrong result by!
Issue Report ID: SM00002
Report Heading: Country context should only reflect the default location

Configuration:
Client-side: HP Compaq nx7000
Windows XP Home Ed version 5.1.2600 SP3
Web browsers: Google Chrome 2.0.172.33
Firefox 3.0.11
IE8 version 8.0.6001.18702
Server-side: Web site: http://maps.google.com

Problem location: Web site: http://maps.google.com

Report author: Simon Morley

Report observation date: 30 June 2009
Report submitted date: 17 July 2009

How/When found: Testing of the location data & Review of results.

Reproducible: Yes

Severity: Medium (non critical, liable to provide erroneous results)

Customer impact:
Incorrect results returned. (Potential to enter data in the customer context – not
necessarily the context expected.)

Description:
It was noticed that postcodes from two different countries could be entered successfully
without indicating the country. However, when another valid post code from one of the
countries was entered a non-expected result was returned (result for a third country.)

Reproduction steps (on any of the browsers mentioned above):

Pre-requisite: Clear the browser cache

Step 1: Enter URL http://maps.google.com

Result: Map displayed over USA

Step 2: Enter a UK postal code (in “Set default location”). Enter LS9
Result: Map displays a location over UK.
Note: The default location is now set to the UK.

Step 3: Enter a valid Swedish postal code (in “Change default location”). Enter 11121
Result: Map displays a location over Sweden
Note: The default location has changed from UK to Sweden.

Step 4: Enter a different valid Swedish postal code. Enter 17831

Result: Map displays a location over USA
Note: The default location was Sweden and the search for “17831” did not happen in
Sweden – this is a separate fault. Default location is now over the US.

Step 5: Qualify the Swedish postal code: Enter 17831, Sweden

Result: Map displays a location over Sweden
Temporary solution:
Indicate a note/warning that the required country should be specified.

Suggested solution:
Do not allow ambiguous entries. Where a default location is set (country) then if a country
context is not entered it will be assumed the location data entered refers to the default
country.

If the location data is not found for the default country then an error “not found” response
is returned.

Include an information response to state that the location was searched for in the default
country (<country name>) and that if this was not the intended country for search then
the country should also be entered.

E.g. assuming the default country was set to the USA then the following search/results
might be expected.

Search location = LS9

Result = “Location not found in USA”

Search location = 178 31

Result = Pennsylvania

Search location = 178 31, Sweden

Result = Ekerö, Stockholm

Expected design & verification cost:

 Medium. Retest of valid and non-valid postcode-country pairings.

Expected benefit to customer:

 Clearer and more accessible usage of the site - the customer learns to know exactly what
is expected and how to enter data. This is done in a very low-tech and low-cost way to
the customer so that it is not a deterrent to usage.

 It is much better that the customer learns to enter the location data with location context
(if different from the default) than to receive a mis-leading result (from the customer's
perspective). The customer may not realise that the result is not related to the location
context (country) that they were thinking about.

 Reduced customer confusion and frustration.

Expected benefit to vendor:

 Enhanced customer experience. Happier customers that return (don't search for
alternatives) and recommend the site to their friends.
Issue Report ID: SM00003
Report Heading: No version / revision handling on the website

Configuration:
Client-side: HP Compaq nx7000
Windows XP Home Ed version 5.1.2600 SP3
Web browser: Google Chrome 2.0.172.33
Server-side: Web site: http://maps.google.com
Version: Not available

Problem location: Web site: http://maps.google.com

Report author: Simon Morley

Report observation date: 17 July 2009
Report submitted date: 17 July 2009

How/When found: Review of results from previous testing.

Reproducible: Yes

Severity: Small (non critical)

Customer impact: Difficult to provide accurate feedback to vendor.

Description:
There is no apparent versioning information on the web site - at least visible from the
start page. This means it is not possible to give accurate feedback from the customer on
issues observed. It is also not possible for the customer to track issues and fixes in
replies from the vendor (e.g. issue SM0003 fixed in version x.y.z)

Temporary solution:
Giving precise information on detection date. However, this may be cumbersome or not
always practical for the customer.

Suggested solution:
Introduce a footer with the version information of the website.

Expected design & verification cost:

Minimal

Expected benefit to customer:

 Additional tracking for issues reported and fixes implemented.
 Allows the customer to have a separate tracking notation from the vendor and still have a
common reference point. Two customers may report similar problems – identified as one
root cause and one fix supplied by the vendor – then the customers can track “their”
problems without having to interpret a generic vendor issue.

Expected benefit to vendor:

 Allows for clearer tracking of reported problems- less ambiguity from the customer
(reported in vA.B.C rather than “I saw this the other day…”)
 Enhances the customer “service” and openness from the vendor to the customer –
ultimately, enhances the perception of accessibility.
Notes on Report ID SM0003

1. This report came from a review of the basic data for the report formats.

2. I could give version information for the client-side but not the server-side

3. Google encourages feedback - but if I try something out 2 weeks ago and report it now
how does google know which version of their website was used? Suppose I'm not sure
about exactly when I saw the issue, "I saw this the other day..."

General notes

I first looked at the site over 2 weeks ago. I made some initial notes and stored them in the
"cloud" using google docs. I then looked at them later, from a different PC, made some editorial
changes and re-stored. Then later at the time of writing the report I used another PC (borrowed)
to write this document and stored in google docs.

I didn't use a set template for the problem/issue reports - I took some headings from Cem Kaner's
bug advocacy paper and some from my own previous experience. I have not tried to make the
reports "best-in-class", but merely as examples of presenting information in a clear and
understandable way.

I found a more “serious” fault by reviewing my problem description – this is mentioned under
report SM00002 (Description, Step4, Note), but I did not have the time/space (10 pages) to fit in
an extra bug report. But this was very interesting to – an example of the power of post-analysis
and reviews – whilst writing a description of one problem I noticed another that I hadn’t spotted
originally!
Below are the "raw" notes that I made whilst looking at the site. Readers can see the exact steps
I went through (the above reports are a consolidation of those steps and groupings of problems)
and how I looked at the problems. The notes above give a summary of the steps but it's
sometimes useful to look at the "raw data" - not always as readable but good for reference.

Raw Notes

Notation/clarification (added 17 July 2009):

Sx:
Means test step - it's not really a test case - the test cases could be created from certain test
steps, but for the purpose of this exercise I'm just interested to differentiate between a test step,
an observed result and my comment or additional information for the observation.

R:
Observed result.

C:
Comment or deduction on what was observed.

Px:
Pre-requisite, pre-condition or pre-amble to step Sx.

The below steps and observations were made 30 June 2009. Steps S3-S6 were tried on
browsers Chrome, Firefox & IE8 with the same results.

Chrome - logged out of my google account

S1: Opened a tab with (http://maps.google.com/)

R: Map opened over USA. Scale set to 500mi/500km

S2: Clicked "Set default location" link

R: Dialogue box "Set default location"

C: No information on how the location should be entered: Country, City, GPS coords, Ordnance
Survey map coords, post/zip code or any other variant.

C: Better dialogue is needed, with a pop-up/roll-over for how a location should be entered....

S3: Invalid GB postcode entered (ls9 9qw)

R: Map is blank.

C: No information that the postcode could not be located.

C: The scale changes from the opening page (200ft/100m in this case.) If the map is blank then
the scale should be blank or removed - otherwise this is misleading. It could be interpreted that
something is displayed and that the scale/zoom needs to be altered to see the display

C: Link has changed from "Set default location" to "Change default location". This is misleading -
implying that my invalid entry is now a "default".
S3.1: Invalid GB postcode entered (ls9 9qw). Then press "Satellite" display button.

R: Map displays the message "We are sorry, but we don't have imagery at this zoom level for this
region. Try zooming out for a broader look."

R: Scale is blank.

C: Mis-leading information. Implies that there is something to be seen at a different zoom level.

C: If there is nothing to be displayed then the response message should be different.

S4: Re-enter address http://maps.google.com

R: Satellite image appears over UK (postcode LS9 9) - however, the default location appears as
"ls9 9qw" (invalid).

P5: Settings are Swedish.

S5: Enter a valid Swedish postcode as the location (111 21 - Stockholm Central Train Station)

R: Map is displayed of the areas.

S6: Enter another valid Swedish postcode (178 31 - Ekerö - one of Stockholm's islands)

R: Map displays a location in Pennsylvania, USA

C: No obvious connection to the site's settings and the search algorithm used

S6.1: Enter the country also "178 31, sweden"

R: Correct map displayed.

C: Inconsistent use of the web browser's settings - picking the Swedish location for the postcode
"111 21" and the US location for "178 31" until the country was entered...