Ten Tips of Web App Testing

How to test and launch a world-class application

White Paper
June 2011

White Paper: Web App Testing

White Paper

Web App Testing

How to test and launch a world-class application
Table of Contents Introduction.. - New Challenges, Old Solutions.. - The Crowdsourcing Advantage.. - Testing Types.... 3 3 4 4

10 Tips of Web App Testing. 6 1. Dont Be a Slave to Your Metrics.. 6 2. Know Thy Third Party Apps... 7 3. The Testing Managers Dilemma.. 7 4. Invest in Testers, Then Tools.. 8 5. Protect Users, Save Yourself.. 8 6. Multimedia, Multi-Problems. 9 7. Forfeit the Blame Game 9 8. Be Everywhere.. 10 9. Plugins: Proceed With Caution.. 10 10. Cross-Browser Testing. 10 About uTest.. 12

Why is there never time to do it right, but always time to do it over?

- Colonel Mike Mullane Retired NASA Astronaut

White Paper: Web App Testing

Introduction
New Challenges, Old Solutions for Testing Your Web Application Establishing procedure amidst a constant state of change may seem like an absurdity (and a losing battle), but this is what todays software companies face when it comes to testing their web applications. Aside from an expanding matrix of new browsers, plugins, third party apps, programming languages and more, there are now thousands, if not millions, of new users from all around the globe. Of course, none of this was in the manual! Thus, the purpose of this whitepaper is twofold. While our 10 Tips section will help you stay on top on of the latest trends in web app testing, our introduction will outline the basics of functional, usability and load testing for web-based applications. Along the way, well cover topics such as:

Complexity kills.

It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges, and it causes end-user and administrator frustration. - Ray Ozzie Chief Software Architect, Microsoft

Third Party Apps: Your product is only part of the equation - what about the satellite apps that are orbiting your web application? Cloud Computing: With opportunity comes risk. Theres a ton of upside to cloud computing, but what happens to your web app when it fails to functional properly? Multimedia: How will your web app perform for users with different versions of Flash, Acrobat, Java, etc.? Have you thought about HTML5 and the challenges of managing H.264 vs. Theora? Plugins: While the use of plugins may be slowing, the use of extensions are not. Learn why plugins and their extensions should be a testing focal point. Localization: Obviously, your application needs to work where your users reside. Discover why most software companies underestimate the importance of localization in their web testing. Deadlines: Your web application will never be perfect, so how will you know when its safe to launch? Its called the Testing Managers Dilemma and well show you how to deal with it. Metrics: As one testing expert put it, When your car is about to go off a cliff, its a weird time to be thinking about gas mileage and drag coefficients. Find out how metrics can mislead.

White Paper: Web App Testing

The Crowdsourcing Advantage While the following material is suited for companies that rely on traditional staffing models for testing (i.e. internal QA teams or offshoring firms), this whitepaper will be especially useful for those who leverage some aspect of crowdsourced (or communitybased) testing. To learn more about the growing trend of crowdsourcing - and how companies of all sizes can use it to complement their in-house efforts check out the 8 Essentials of Crowdsourcing whitepaper.

Web Applications: Testing Types

Functional Testing No surprises here. The most frequent and critical task that QA takes on before launch is to perform comprehensive functional testing. This can include - but is not limited to - testing of:

For

a successful technology, reality must take precedence over public relations, for nature cannot be fooled. Richard Feynman Renowned Scientist

Web Forms: This includes sign-up, signin, contact and purchase forms; database queries, checking for cross-site scripting/AJAX vulnerabilities and other issues related to incorrect inputs. Links: Ensuring that all outgoing and internal links are functional and accurate. GUI: Despite back-end test automation, front end problems with web applications are almost always discovered by human testers, including issues related to browser compatibility or CSS flaws. Language: Does your web app properly support Unicode? Can it accept and handle foreign characters? This is a growing concern for developers and should be considered as an essential part of your testing efforts. Security: Security testing including checks for open redirects, cookie tampering, SQL injections and other vulnerabilities that could harm your users (and reputation) should also be performed prior to launch.

Bottom line: Does your web application work when, where and how your users need it to work? Usability Testing Beta testing may be suitable for some large companies (mainly those with name recognition) who want to solicit user feedback, but it is NOT an effective substitute for usability testing, which should include professional analysis of: Graphics: Have you identified and addressed the major GUI bugs in your web app? Are the color schemes easy to understand (in every country of your user

White Paper: Web App Testing

base)? Many of these flaws will not be reported by beta users, but they should be known about and fixed before launch. Feature Set: This can include the intuitiveness of shopping lists, online checkouts and other e-commerce actions; the accuracy of search results, data entry, sign-up forms and other features. Accessibility: Is your application accessible for all users, including the visually impaired? Does your app involve parental or administrator restrictions? Be aware of your legal obligations. Benchmarking: How does your web application stand up to that of your closest competitors? What about your own previous versions? What areas does it compare favorably or unfavorably? Overall Usability: Beta testers - if they report bugs at all - are likely to respond with this-site-sucks or this-site-is-great type of analysis. When launching a web application, professional, non-biased analysis is essential.

Bottom line: Is your web application intuitive and highly usable for your target audience? Load Testing How will your web application hold up when it needs to most? Synthetic load tools are helpful in simulating these types of scenarios, but with the assistance of real testers, you can obtain a complete picture of how your application performs under stress. Load testing should, at the very least, involve one of the following methods:

The

real value of tests is not that they detect bugs in the code, but that they detect inadequacies in the methods, concentration and skill of those who design and produce the code. - Tony Hoare Testing Expert

Live Load Testing: A team of live testers (preferably from where your users reside whether it be North America or a global audience) will test your application simultaneously, enabling you to see how your app performs under real-world, non-simulated load. This process, as its name would indicate, requires NO automated tools. Ideal for web apps that contain Flash, streaming video or other types of multimedia that are difficult to evaluate with simulated load. Simulated Load Testing: Using the simulated load testing tool of your choice (more on this below) you can obtain a realistic snapshot of your web application's performance under peak synthetic usage. Hybrid Load Testing: A combination of the aforementioned methods, this process involves having live testers perform functional testing on your web application while under automated load. This lets you see bugs or performance issues that only show themselves under peak loads.

White Paper: Web App Testing

Bottom line: Will your web application perform as expected under maximum pressure? Will it crash? Will the performance degrade? Web Testing Checklist: A Summary Regardless of whether you are performing functional, usability or load testing, it is imperative that each of these practices achieve maximum testing coverage that matches your user base across: Location: If you have a global user base, why would you only When your car is about to go off a cliff, test your web application in your its a weird time to be thinking about office? Its therefore critical to gas mileage and drag coefficients; extend your testing coverage better to take the right control action beyond your borders if needed. look out the window and steer or use Language: If your application is available in multiple languages, the brake until youre back on course. you must verify that nothing is - Michael Bolton lost in translation (like the CEO, DevelopSense intuitiveness of your content, error messages and core features). Operating Systems: Your app needs to work seamlessly across all the different flavors and versions of Windows, Mac and Linux. Browsers: The days of Internet Explorer hegemony are over. Todays web applications must be tested thoroughly against multiple versions of IE, Firefox, Chrome, Safari and others.

And now, the Ten Tips of Web Testing.

The 10 Tips of Web Testing

1. Dont Be a Slave to Your Metrics When testing a web application, its easy to become overwhelmed by the data thats available to you. It can drag you in all sorts of directions and distract you from more important matters if you allow it to. Dont. Said testing expert Michael Bolton of DevelopSense: Some people enslave numbers. They make numbers work too hard, and too often. Ive seen organizations collect piles of data about defect escape ratios and defect detection percentages. They hire market research firms and calculate the ratio of happy customers to unhappy customers. But the aggregated data doesnt

White Paper: Web App Testing

tell you anything specific on how to make things better for the unhappy customers. In other words, while comprehensive testing will present you with a ton of raw data, the most important information will still come from your testers, managers and customers. 2. Know Thy Third-Party Apps When launching a web application, remember that it wont operate in a vacuum. There are many third-party applications that could interfere with its performance, apps like: Live chat Checkout processes Search plug-ins RSS Feeds Embedded videos or audio players Ad servers or embedded ads Web analytics packages Blogs, forums and message boards Social networking modules or toolbars (like Tweet streams or Digg)

Do any of these apply to your web application? To your users, these are part of your app. 3. Understand the Testing Managers Dilemma As any honest test manager will tell you, it is NOT his or her responsibility to make the final decision about when the product is completed that is clearly the role of the product owner. Testing managers have enough to worry about. Between time limits and budget constraints, theres no shortage of obstacles to must deal with so dont make them the final arbiter of product readiness. Besides, to a good testing manager, their job is never complete. In theory, software testing could go on forever! Despite knowing better, software executives regularly pressure their testing managers into providing an answer, and end up being told what exactly they want to hear. Also, it is okay to fall off the agile wagon every once and awhile. With the short sprint cycles and constant deadlines, its amazing that more companies dont wind up there more often. But just because you cant maintain a truly agile schedule 24/7, doesnt mean that certain aspects of your development (like testing consistently) should ever be abandoned.

White Paper: Web App Testing

4. Invest In Testers, Then Tools While bug-tracking systems and automated test tools are an essential part of any test team, theres a price to be paid for relying on them too heavily literally and figuratively. Consider automated tools, for instance. Automation will never find bugs for use cases that havent been conceived and documented. It simply helps to ensure that old bugs dont resurface and that the main test cases are clear. Conversely, live testers will find bugs that are new or unique to your app - which is probably most of them. In short, ignoring real-world testing at the expense of tools is a recipe for disaster.

I urge you not to use

Speaking of expense, if you dole out big money for elaborate test tools (and expect an even bigger - James Bach ROI), you will feel compelled to keep using it, regardless of its Testing Expert effectiveness. In this regard, software companies would be wise to follow the advice James Bach gave to individual testers (see sidebar). 5. Protect Your Users, Save Yourself To have a safe, secure web application is obviously a no-brainer nobody launches with anything less in mind. Whats not so obvious is the way to achieve this type safety for your web app. To point you in the right direction, here are a few questions to ask yourself before your next launch: Is your web app behaving the way your privacy policy claims it does? Can a tester easily get an app to cough up the private data of another user? For example, if they see user_id=232 in the URL, what happens if they change it to user_id=231? Do they get to see someone elses personal data? Are you sharing personally identifiable information about your users with third-parties like Salesforce.com or Google Analytics? What about the company thats hosting your app? What happens if your web app is cached when it shouldnt be? Does it share the wrong data with people? What happens if you actually want caching? Is it sending out the right things to be cached? Are end users actually seeing a benefit?

expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they dont work. A free tool may be freely abandoned. This gives you flexibility.

White Paper: Web App Testing

Is your site vulnerable to common security exploits like XSS, injection flaws, broken authentication, flawed session management, unvalidated redirects and forwards?

Since these areas are sometimes overlooked by in-house teams, it can be helpful to leverage testers who are skilled in testing, but new to your application. 6. Multimedia, Multi-Problems How much does your web application rely on Flash, HTML 5, Java and other richmedia tools? Even if you answered a little bit, multimedia testing - including load testing - should become a critical component of your testing from this day forward. Of course, many in-house teams will find they have neither the time nor the headcount to achieve testing across all versions of multimedia players. If this is the case, you should strongly consider leveraging a community of professional testers to quickly and easily find the versions you need. 7. Forfeit the Blame Game

Its important to have a tester that can

tell the difference between an enhancement and a true bug. I know, it sounds so ridiculously trivial but finding quality testers who can also understand the nuances of business is key. - Jack Margo SVP of Web Operations, DevShed

Reproducing defects, filing bug reports and verifying fixes are tasks commonly believed to be bottlenecks in the testing process. They are not. While they can comprise close to 20% of a test teams duties - and while testing teams are often blamed for delays in these processes - it is often actually a bottleneck on the development side (where the engineers are competing with their own challenges, obstacles and deadlines). If test teams can work with development (rather than blame them) to improve the quality of the software prior to code completion, then it will improve the speed of the whole system. Buggy software can seem like the test teams fault, but it is important realize that other factors are contributing the pace of each development cycle - so dont take it out on your testing team. Making developers better, helping them understand failures and the factors that cause them will mean fewer bugs to find in the future. Testers are quality gurus and that means teaching those responsible for anti-quality what they are doing wrong and where they could improve.

White Paper: Web App Testing

8. Be Everywhere Does your web application work in New Zealand? What about Singapore? If your web app is localized that is, if your users are dispersed throughout the globe then youll need to test language translation, currency conversion and other locationspecific features. Aside from functionality, it is equally important to run usability testing with users from around the world. Neither in-house QA teams nor outsourcing firms can efficiently complete such a task. It would be (and indeed, has been) time-consuming, costly and impractical. And thus, it has been ignored. However, crowdsourcing enables you to easily leverage a global community of professional testers, letting you handpick the testers you need to fill out your testing coverage matrix. 9. Plugins: Proceed With Caution Plugins remain a significant problem for web application developers. In fact, most of the security issues on the web today are a direct result of Flash and Acrobat bugs. In many cases, people end up with ancient versions of Acrobat reader that can run random exploits simply because they forgot to make the needed updates. Does this sound familiar? Also, while the sheer number of browser plugins has declined in recent years, extensions have gained in popularity. While less common (and usually only found on more advanced users browsers), extensions can dramatically impact how an app will work if they are buggy. As such, they too will require testing prior to launch. 10. Respect The Dead Make Way For The New Its no secret that IE6 is on its way out, but large organizations cant seem to kill it off just yet. As frustrating as it may be, support for older browsers can The internet? Is that thing still around? be critical, especially for B2B applications.

Homer Simpson

Conversely, the next wave of HTML5 is going to introduce a flurry of competing multimedia standards. HTML5 vs. Flash is going to be a big debate in the years ahead, and companies will have to make a very important choice: Either they deal with Flashs security issues and lack of support on mobile, or they opt for HTML5s confusing vendor specific standards (H.264 vs. Theora).

10

White Paper: Web App Testing

Also, CSS3 is slowly making its presence felt. In fact, many web apps already use it for additional layout effects. If you go down this road, how will non-CSS3 browsers deal with this? Cloud computing is another major trend to keep in mind when testing your web app. How will your application behave when different parts of the cloud are misbehaving? What happens to your lead generation efforts, for example, if your CRM system malfunctions? Or if Omniture goes down for an hour? What effect will this have on your users? Consider these cloud issues when testing. While some trends will die before their time, others will persist far beyond whats anticipated. For web application companies, it is critical to be mindful of these trends and remain on the offensive. Then again, you dont have much of a choice.

11

White Paper: Web App Testing

About uTest uTest provides in-the-wild testing services that span the entire software development lifecycle including functional, security, load, localization and usability testing. The companys community of 60,000+ professional testers from 190 countries put web, mobile and desktop applications through their paces by testing on real devices under real-world conditions. Thousands of companies -- from startups to industry-leading brands rely on uTest as a critical component of their testing processes for fast, reliable, and cost-effective testing results. More info is available at www.utest.com or blog.utest.com, or you can watch a brief online demo at www.utest.com/demo.

uTest, Inc. 153 Cordaville Road Southborough, MA 01772 p: 1.800.445.3914 e: info@utest.com w: www.utest.com

Case Study Case Study

12