Automation Testing Tools

Sanjay Patel
Muhammad Rafi
Mahesh Vaghasia

-1-
 Automation Test Tools Comparison

One of the biggest problems in the software industry is that it is impossible to manually
test all possible paths and combinations of paths through a realistically large software
program. No software project has that much time to exhaustively test the software. As a
result, there has growing market for automated testing tools. However, many attempts to
automate software testing are not successful. Simply put, “Automated Testing” means
automating the manual testing process currently in use. This requires that a formalized
"manual testing process" currently exists in the company or organization. Minimally,
such a process includes:
 Detailed test cases, including predictable "expected results", which have been
developed from Business Functional Specifications and Design documentation.
 A standalone Test Environment, including a Test Database that is restorable to a
known constant, such that the test cases are able to be repeated each time there are
modifications made to the application.

The case for automating the Software Testing Process has been made repeatedly and
convincingly by numerous testing professionals. Most people involved in the testing of
software will agree that the automation of the testing process is not only desirable, but in
fact is a necessity given the demands of the current market.

At first glance, it seems easy to automate testing. You can just buy one of the popular test
execution tools, record the manual tests, and play them back whenever you want to.
Unfortunately, as those who tried it have discovered, it does not work like that in
practice. Just as there is more to software design than knowing a programming language,
there is more to automation testing than knowing a test tool.

Research background: As part of our efforts to evaluate several major tools available in
the market, we have evaluated following six tools:

Tool Name Vendors

========= ============
Rational Robot Rational Software
Winrunner 7 Mercury
QA Run 4.7 Compuware
Visual Test Rational Software
SilkTest Segue
QA Wizard Seapine Software

As part of our evaluation, we used some of the tools: Winrunner, and QA Wizard. We
interviewed several people within Motorola and Siemens Medical Solutions who have
been working with some of these tools for years. None of us had any prior experience in
working on any of these tools, so we had to rely a lot on the informal interviews that we
conducted and demos given by some testers in Motorola and Siemens. In Motorola, one
department: CDMA software division extensively uses Robot to test their software. We
also got some of our research materials from IEEE and ACM libraries.

-2-
Questions asked to testers

Following are questions that were asked to testers who have been using some of these
tools inside Motorola and Siemens:
1. How long have you been using this tool and are you basically happy with it?
2. How many copies/licenses do you have and what hardware and software
platforms are you using?
3. How did you evaluate and decide on this tool and which other tools did you
consider before purchasing this tool?
4. How does the tool perform and are there any bottlenecks?
5. What is your impression of the vendor (commercial professionalism, on-going
level of support, documentation and training)?
6. Does the tool perform your entire set of tests and are there pre and posttest
activities that have to be performed manually?
7. How easy is it to interpret the results of the automated tests?
8. How did the use of the tool affect the quality of the testing?
9. Were there any non-technical problems in your organization through introducing
the tool, and how were they overcome?
10. Is the tool now integrated into your work processes and standard procedures? If
so, how much effort and how long did this take?
11. Were there any other benefits or problems in using the tool, which were not
anticipated?
12. Do you feel the tool gives you value for money?
13. Have you saved any money by using this tool? Can you quantify the savings or
improvements?
14. How long did it take you to achieve real benefits? What are the critical factors for
achieving payback?
15. What improvements are needed in the tool? Is the vendor responsive to requested
enhancements?
16. What were your objectives or success criteria when buying the tool? (e.g. run
more tests, more consistent regression tests, improvement in meeting release
deadlines, improved productivity, capacity planning, performance assessment)
17. If you were doing it over again now, would you still purchase this tool? What
would you do differently?

Short overview of each tool

Rational Robot v2003

IBM Rational Robot v2003 automates regression, functional and configuration testing for
e-commerce, client/server and ERP applications. It's used to test applications constructed
in a wide variety of IDEs and languages, and ships with IBM Rational TestManager.
Rational TestManager provides desktop management of all testing activities for all types
of testing.

Rational Visual Test 6.5

-3-
Rational Visual Test 6.5 is a tool based on the BASIC language and used to simulate user
actions on a User Interface. It goes far beyond clicking on controls and generating
keystrokes, however. Visual Test is a powerful language providing support for pointers,
remote procedure calls, working with advanced data types such as linked lists, open-
ended hash tables, callback functions, and much more. Included with Visual Test is a host
of utilities for querying an application to determine how to access it with Visual Test,
screen capture/comparison, script executor, and scenario recorder.

Mercury WinRuner 7

Mercury WinRunner is a powerful tool for enterprise wide functional and regression
testing. WinRunner captures, verifies, and replays user interactions automatically to
identify defects and ensure that business processes work flawlessly upon deployment and
remain reliable. WinRunner allows you to reduce testing time by automating repetitive
tasks and optimize testing efforts by covering diverse environments with a single testing
tool. WinRunner is available for Windows 98, Windows 2000, Windows Me, Windows
XP, and Windows NT platforms.
With WinRunner, you can:
• Create test scripts simply and easily
• Automatically identify discrepancies in data
• Validate applications across browsers
• Automatically recover tested applications from a crash.

Segue Silk Test

Silk Test is an automated tool for testing the functionality of enterprise applications in
any environment. It lets you thoroughly verify application reliability within the confines
of today's short testing cycles by leveraging the accuracy, consistency and time-saving
benefits of Segue's automated testing technology. Designed for ease of use, Silk Test
includes a host of productivity-boosting features that let both novice and expert users
create functional tests quickly, execute them automatically and analyze results accurately.
With less time spent testing, your QA staff can expand test coverage and optimize
application quality. In addition to validating the full functionality of an application prior
to its initial release, users can easily evaluate the impact of new enhancements on existing
functionality by simply reusing existing test casts.

QA Run 4.7

With QA Run, programmers get the automation capabilities they need to quickly and
productively create and execute test scripts, verify tests and analyze test results. QA Run
can help you keep testing in sync with accelerated release cycles, improve the return on
your testing investment and deliver the quality applications end users and management
expect. QA Run uses an object-oriented approach to automate test script generation,
which can significantly increase the accuracy of testing in the time you have available.
As you point and click, QA Run records user actions and system responses into reusable
scripts that test specific application functions. You will create powerful, feature-rich tests

-4-
even if you don't have extensive knowledge of programming languages or application
structure.

QA Wizard

QA Wizard completely automates the functional regression testing of your applications

and Web sites. It’s an intelligent object-based solution that provides data-driven testing
support for multiple data sources. QA Wizard's scripting language includes all of the
features of a modern structured language, including flow control, subroutines, constants,
conditionals, variables, assignment statements, functions, and more.

Evaluation criteria

In order to be able to evaluate each tool, we created several criteria or desired functions
of a testing tool. We evaluated each tool against each functionality to see how each tool
supports that functionality. Following are the criterion against which each tool was
evaluated:

1. Record and Playback: This criterion deals with how easy it is to record and
playback a test. Does the tool support low-level recording (mouse drags, exact
screen location)? Is there object recognition when recording and playing back? Or
does it appear to record ok but then on playback (without environment change or
unique id’s etc changes) fail? How easy it is to read the recorded script.
When automating, this is the first thing that most professionals will do. They will
record a simple script; look at the code and then playback. Eventually record and
playback becomes less and less part of the automation process as it is usually
more robust to use the built-in functions to directly test objects, databases etc. But
it is necessary to evaluate a tool for these criteria. If a tool cannot recognize
objects then the automation process will be very tedious experience.
2. Web Testing: Web based functionality on most applications is now a part of
everyday life. As such, test tool should provide good web based test functionality
in addition to client-server functions. While evaluating the tools for this
functionality, we looked for the tools native support for HTML tables, frames,
DOM, various platforms for browsers, website maps and links. Some of the things
to look for are:
• Are there functions that let us know when the page has finished loading?
• Can we tell the test tool to wait until an image has been loaded?
• Can we test if the links are valid or not?
• Can we test web based objects functions like is it enabled, does it contain
data?
• Can we extract data like title of the page, a hidden element etc. from the
web page itself?

3. Environment Support: How many environments does the tool support out of the
box? Does it support the latest Java release, Oracle, Powerbuilder, Wireless
protocols etc. Most tools can interface to unsupported environments if the

-5-
 developers in that environment provide classes, dll’s etc that expose some of the
applications details but whether a developer will or has time to do so is another
question. Ultimately this is one of the most important criterions to test a tool. If
the tool does not support our application, then it’s worth nothing.
4. Cost: Our selection of any tool is constrained by the available budget. While cost
may not be a significant function criterion to evaluate a test tool, we can never
ignore this while evaluating a tool.
5. Ease of Use: We have evaluated each tool on how easy it is to use each tool. Since
we were the first time users for all of these tools, we were in a very good position
to judge how easy it was to use these tools. For ease of use, we looked for User
interface, out of the box functions, debugging facilities, error messages, layout on
the screen, help files and user manuals.
6. Database Tests: It is essential for a test tool to provide multiple database support.
Nowadays, we have many database systems like: Oracle, DB2, SQLServer,
Sybase, Informix, Ingres etc. We evaluated each tool to see if and what database
systems each tool provided. We also looked at how the tools handle data stored in
the database, for example, how they hold returned data e.g. is this an array, a
cursor, a variable etc. How does the tool manipulate returned data? Can it call
stored procedures and supply required input parameters? What is the range of
functions supplied for this testing?
7. Data Functions: Besides database support, test tools also need to provide a
facility for storing data offline. For this functionality, we looked at all the tool
facilities for creating and manipulating data. Does the tool allow us to specify the
types of data we want? Can we automatically generate data? Can we interface
with files, spreadsheets, etc. to create, extract data? Can we randomize the access
to that data? Is the data access truly random? This functionality is normally more
important than database tests as the databases will usually have their own
interface for running queries.
8. Object mapping: A lot of applications use custom objects. Most custom objects
behave like a similar standard control like: pushbuttons, checkboxes, radio
buttons etc. For application that use custom objects, we need to make sure if we
are able to map (tell the test tool that the custom control behaves like the
standard) control? Does it support all the standard controls methods? Can we add
custom control to its own class of control?
9. Object Identity Tool: While interacting with a test tool, one of the primary means
of identifying objects will be via an ID Tool. A sort of spy that looks at the
internals giving you details like the object name, ID, and similar. This allows us
to reference that object within a function call. The tool should give us details of
some of the object’s properties, especially those associated with uniquely
identifying the object or window.
10. Extensible Language: This is a criterion that some of the testers in Motorola
introduced to us. According to them, a new user will not need to care for this
functionality, but an experienced user who has been using the testing tool for 6
months to one year will need the tools to support language extension so that the
tester can create DLL or extend the language in some way. Some tools provide
extension by allowing us to create user defined functions, methods, classes, etc

-6-
 but these are normally a mixture of the already supported data types, functions,
etc rather than extending the tool beyond its released functionality. Once again,
this functionality was emphasized by some of the experienced testers inside
Motorola.
11. Integration: Tool integration is a very important functionality. Most of the
organizations have variety of tools from different vendors in their software
organization. It’s essential for a testing tool to integrate with other tools. We
looked to see how well does the tool integrates with other tools. Does the tool
allow us to run it from various test management suites? Can we raise a bug
directly from the tool and feed the information gathered from our test logs into it?
Does it integrate with products like word, excel or requirements management
tools?
12. Image Testing: Some application need to test bit map and similar images. Also
when the application has painted controls like those in the calculator application
found on a lot of windows applications, we might need to use Image testing. For
this functionality, we look for; does the tool provide OCR (Optical Character
Recognition)? Can it compare one image against another? How fast does the
compare take? Does the tool allow us to mask certain areas of the screen when
comparing.
13. Test/Error Recovery: Although it is one of the most difficult areas to automate, it
provides foundation to produce a truly robust test suite. The test tool should
provide facilities to handle test/error recovery. We checked the tools to see how
does the tools handle errors, including what happens when the system crashes?
What happens when the database connection is lost?
14. Object Tests: A test tool should provide services for testing object properties. For
object tests, we looked for; can the tool validate several properties at once? Can it
validate several objects at once? Can we set object properties to capture the
application state?
15. Support: Finally, we looked at what kind of support each tools and the vendors
provide. For this criteria, we looked for online support, how fast the service is
available, does the vendors provide software patches free of charge, is there a
ticketing system available for us to open a problem tickets, how soon a problem
ticket is handled etc.

Comparison of the tools based on the above criteria

Record and Playback

WinRunner Has a recording facility allowing you to record in either context sensitive mode or
Tools

in analogue mode. It has rich command line functions that allow you to run the
scripts on start up. WinRunner recognizes two variable types: number or a string.
There is no need to declare variables up front. The scripting language is knows as
TSL (Test Script Language). TSL provides usual array of functions and syntax,
but is also has specific functions for WAP, which are used through an emulator.

-7-
 Has a learn facility for recording and playback. This can be started by clicking the
learn button on the tool bar. There are various ways to playback scripts e.g.
QA Run command line and via the toolbar. It also has a facility: RunAWL. RunAWL is
Compuware’s script execution module that allows you to select scripts to run
without having to open the script editor.
Has two main ways to start recording: one is through it’s quickstart window and
the other from the toolbar->file->New. Silk Test uses an identifier and a tag to
identify each object individually. You can use multi-tags to make this
Silk Test identification more unique. Silk Test uses object-oriented approach. Each object
on the screen is based on a class. Silk Test supports in excess of 20 data types
including: Boolean, long, number, string, semaphore, set, time, window etc. It
supports classes, inheritance, collapsing and opening various parts of your script.
Provides the facility to record and playback scripts. Clicking on the ‘tape cassette’
icon on the toolbar starts the recording session. Each recording session is given a
scenario name if objects are not recognized when you click then a dialog will pop
Visual Test
up asking you to associate the unknown window/object with a know type. It
supports following data types: Integer, short, long, single, double, string, pointer
and variant. Scripts are compiled to P-code and so can be run on other machines.
Provides the ability to record test using a red GUI button; you can also record
(Virtual user) test scripts to play back in LoadTest (another tool by Rational).
Robot uses a language very similar to Visual Basic called SQABasic. It supports
Robot
the following data types: String, Integer, Long, Variant, object, single, double,
currency, user-defined. User defined allows you to build a mixture of data types
similar to a ‘struct’ in C or a ‘record’ is Pascal.
Uses a visual and menu driven approach for creating scripts. It provides an easy to
QA Wizard use paint-and-click approach to scripting. People with very little knowledge in
scripting can record and playback with this tool.

Web Support
To start a web browser in WinRunner, you need to use the web_browser_invoke
function and also enable the WebTest add-in. One disadvantage with WinRunner
is that it does not support Java out of the box. You have to buy a Java add-in. That
makes WinRunner more expensive than other tools. But with the Java add-in,
WinRunner
WinRunner is probably the best among the tools to support web. Following are
some of the functions provided by WinRunner: Web_password_encrypt,
Tools

Web_sync, Web_image_click, Web_link_click,Web_url_valid, bl_get_cell_data,

Web_refresh.
Among the tools, QARunner has worst web testing support. Java support is very
poor as well especially with the version that we tested. Following are some of the
QA Run web functions supported by this tool: AnchorSelect( ), BrowserToolbarCtrl( ),
ImageSelect( ), LinkCheck( ), MouseOver( ) etc. Like Robot, QARunner also
provides a facility to carry out full site checks.

-8-
 You need to enable web extensions to do web testing. Silk Test takes e-commerce
very seriously and supports web testing extensively. Following are some of the
web methods supported: Browser.ClearCache( ), HTMLLink class, HTMLTable
Silk Test class, HTMLPushbutton class, HTMLTextField class, WaitForReady method, this
is a good Function allowing you to wait for a predefined time or until the
application is ready. LoadPage method etc. They have an object browser that
allows you to look at all the classes, their methods and properties.
To start a web browser, you need to use the Webexplore command. This can be
done by writing the URL in the function. VisualTest only supports testing of
Internet Explorer pages (out of the box). Following are some of the commands
supported: WebImageClick, WebFrameExists, WebTableCellCount,
Visual Test
WebTextAreaValue, WebActiveXHasProperty, WebDisplayInfo,
WebLoadReady. These functions are pretty much self-explanatory. The web-
testing capabilities are limited if you require testing on Java-based web sites or
Netscape.
To enable the web-based functionality, you usually have to launch a browser
using StartBrowser function and by enabling the appropriate HTML box in the
extension manager found under Tools->Extension Manager. Following are some
Robot of the commands supported: HTMLDocument for positioning the cursor in web
pages, HTMLImages for clicking an image, HTMLLink for performing a click on
web link, SQASetDefaultBrowser for setting default browser. There are a host of
Java commands to use as well.
Being one of the newer tools, QA Wizard has an excellent web support. It
supports all the browser. It provides very good Java support as well as XML
QA Wizard support; in fact, all the reports in this tool are written in XML. It provides almost
all the web functions supported in other tools. Among the tools, QA Wizard and
WinRunner were the two best for web support.

Environment Support
Out of the box supports, all windows based system. With add-ins, support Java
WinRunner
based applications.
Supports NS-DK, Oracle, PowerBuilder, Visual Basic, Uniface (Compuware’s
QA Run own 4GL), SAP, Java, Netscape, Internet explorer, and most windows
applications
Supports HTML, XML, JavaScript, Java, ActiveX, Windows 2000, Windows 98,
Tools

Silk Test
Windows 95 controls, Visual Basic, Internet Explorer, and Netscape.
Supports Microsoft centric environments e.g. Internet explorer, Windows based
Visual Test
control and similar.
Supports HTML, DHTML, Java, Visual Basic, Visual C++, Oracle
Robot
Developer/2000, Delphi, SAP, PeopleSoft and Sybase PowerBuilder.
QA Wizard Supports HTML, DHTML, XML, Java, and Visual Basic.

Cost
WinRunner Being one of the most reputable test tools tends to be expensive. Price depends on
Tools

number of licenses purchased. Java add-in increases the cost to anywhere from $
4000- 6000 per license

-9-
 QA Run Price is comparable to the price of Robot. Each license cost $2000-3500.
Is on the expensive side. Segue products tend to be more expensive than the
Silk Test
others. Price is a disadvantage here. Anywhere from $3000-5000 per license.
The cheapest of all the tools. You can buy it for close to $1000. You don’t need
Visual Test
multiple licenses, which makes it cheaper.
Robot Depends on number of licenses purchased. Varies from $2500-4000 per license.
QA Wizard Each license varies from $3000-$5000

Ease of Use
Not the easiest, but not that hard either. One very important and easy thing is use
WinRunner of wizard. If you have use wizards in Microsoft applications, then this tool is
pretty easy to use.
It’s very similar to ROBOT, but a bit harder to use than ROBOT. Has very good
QA Run
interface and with one or two use, even a novice can work with this tool.
For a beginner, there is a big learning curve involved in learning this tool. It takes
Silk Test
Tools

some getting used to this tool, not very intuitive.

A little difficult for beginner. It has pointers, some C++ tools, and the interface is
Visual Test
not as slick as the other tools.
Easiest to use. A person without any programming skills could easily learn and
Robot
use this tool. Very good user interface and error messaging.
It is very close to ROBOT for being easy to use. Most of the scripting is done in
QA Wizard
the background. Users can easily create test scripts using click-and-paint method.

Database Tests
You can either test database content via ODBC and db_functions or via a database
WinRunner checkpoint. There is also a neat little function that allows you to import from a
database into a data table.
Provides most database functions. You can accomplish most things either by the
QA Run functions or by creating the correct SQL statement. The dbEOF is very useful for
looping result sets.
Tools

Silk Test Provides very good database support.

Does not support database out of the box. You need to use DLL to accomplish
Visual Test
this.
Database tests are supported via ODBC using following functions: SQLOpen,
Robot
SQLClose, SQLError, SQLRetrieve, SQLRetrieveToFile, SQLExecQuery etc.
QA Wizard Provides support for most database functions.

Data Functions
WinRunner Provides functions for Data-Driven test functions. You can program these
Tools

functions directly to work with an excel spreadsheet or tab separated variable file
or you can use built-in wizards. There is also a data table that you can populate.

- 10 -
 You can use Open functions for working with files Also QA Run provides some
QA Run
additional data functions.
Silk Test uses a script called testcase that is similar to a sub procedure in ROBOT
Silk Test and QA Run. This testcase can have records associated with it for replacing hard
coded data like field data, etc.
Provides functions for manipulating data stored in files. This data can be inputted
Visual Test
into a spreadsheet or database and the data in these stored as CSV files.
Simply the best tool for this function. It provides a resource for data creation
Robot called Datapool. Datapool table allows you to create data for feeding into an
application but unlike lists, arrays, and user-defined records.
QA Wizard Provides support for working with text, spreadsheet, XML and CSV files.

Object Mapping
Object mapping is available via the rapidtest wizard and also the virtual object
wizard. This allows you to map non-standard objects or painted controls to
WinRunner
standard objects as recognized by WinRunner. Rapidtest does not work when you
have webtest facility enabled.
Has a good facility for object mapping. All identified objects use a hybrid of
properties to uniquely identify themselves. These properties are: Type – whether
QA Run
it has derived title or not, Module, Class, Title, and Position. You can get QA Run
to automatically create name for unidentified testing objects.
You can map the custom win to the so-called meta classes i.e., BUTTON,
Silk Test STATIC, and MDI. You do this in the windows declarations dialog. It works as in
other tools in that you map an object to a class you think it behaves like.
Object mapping can be done via two ways. The first is from tools->Options-
Tools

>Recorder Tab. You can select the window/object you want to map. Then you
Visual Test
select the object to map against e.g. pushbutton, etc. The other way to map object
is when you are recording a scenario.
Object mapping facilities can be accessed through the tools->general options
menu. This allows you to include (map) a class against known classes. The
Robot
classes to map include Java, HTML, PeopleSoft Tools and standard windows
controls
You can map objects to pre-defined classes that the tools come with. This tool
comes with most of the standard window controls and objects can be mapped
against these classes. It was a little tough, at first, for us to find how to do that.
QA Wizard
The tool does not provide enough documentation as to how the object mapping
works. We had to play around with it to get this done. Object mapping is quite
weak in this tool.

Object Identity Tool

Uses a facility called GUI Spy. This allows you to find the name and properties of
WinRunner
windows and objects on the screen.
Tools

Does not show as many properties of the object as you would like, but it does
QA Run
store and provide basic details of a window or object on the screen.
Silk Test Uses object ID tool, which is part of the windows Declarations facility.

- 11 -
 Uses the window information utility to capture object details. The utility is quite
neat in that you first select it and point to the window/object and basic details of
Visual Test
the window/object are displayed in the window. You can also browse the object
for more details.
Uses a utility called Inspector. You select inspector and it searches all the open
Robot
applications and displays all related objects and properties in a tree-like view.
You can input the name of the object on the search window to get details of the
QA Wizard object. You have to look for to get the details of an object. Some of the other tools
are more proactive in providing details of the objects.

Extensible Language
WinRunner You can extend WinRunner’s facilities by creating or using DLL’s.
QA Run Like other tools, this tool is also extendable through the use of DLL.
You have the ability to extend the 4Test language in several ways. DLL
Silk Test invocation allows you to call functions stored in DLL’s. It offers inheritance and
abstract classes. You can add your own verification properties to classes.
Declaring a DLL function can extend Visual Test. One neat thing is that you can
Tools

Visual Test use pointers so functions that require variables to be passed by reference can be
used even though the majority is byVal.
Robot’s language can be extended in two ways: One by declaring a function in
DLL and then calling it. You can also extend the language by creating classes in
Robot
languages like: VB and then creating an instance of the class using the
CreateObject function.
QA Wizard It can be extended by using DLL function.

Integration
Since WinRunner is considered as the de-facto test automation tool, more and
Tools

WinRunner
more third party software are popping out that support WinRunner.
QA Run is very strong for integration. Compuware is one of the largest vendors
with a big variety of software product, a lot of which integrate with QA Run. For
QA Run
overall integration, only Rational can match QA Run on the amount of tools that
integrate with QA Run.
Provides integration with some third party products. It integrates with QA
Silk Test
Director, PVCS and some other third party products.
It also integrates very well with some rational products including PureCoverage,
Purify, Quantify, Clearcase, and ClearQuest. It can also be integrated with other
Visual Test
external products by using Visual C++ IDE. There is not much third party
support.
Robot Robot integrates very well with other rational products including LoadTest,
ClearQuest, Purify, Quantify, and Purecoverage. There is also an API that allows
for further integration to run a Java test. Also, a few third party products integrate
with Robot as well.

- 12 -
 It provides and API to integrate with third party products. Since, this is a quite
new tool, there isn’t much third party support available for this tool. In order to
QA Wizard
integrate with other tools, you have to use the API and programmed it to work
with other third party products.

Image Testing
Supports images testing through its bitmap checkpoints. You can adjust the
WinRunner threshold of pixels for comparing images and another closely related facility (get
text) could sometimes get text from these images.
Image testing is handled via bitmap checks. The usual images testing facilities
QA Run are: capturing baseline images, masking, comparing differences between baseline
and actual.
Silk Test supports image testing through the use of its window bitmap tool. You
Tools

Silk Test
can create masks, show differences, zoom etc.
Supports image testing through its screen utility and by using a window compare
when recording a scenario. The screen can add masks to images and you can save
Visual Test
the images to bitmaps, clipboard or an scn (for screen) file. This is slightly more
flexible than some of the other test tools.
Robot Robot supports image testing via its region and window image verification points.
QA Wizard Does not have image testing facility. We could not find it.

Test/Error Recovery
WinRunner does not have error recovery like Silk Test. It does however have a
good exception handling facility. Exception handling can be done at the following
WinRunner
levels: Web Exception handling, Pop up exceptions, TSL Exception, Object
Exceptions.
Works the same way as Visual Test and Robot by using the On Error and
QA Run
associated functions.
Silk Test stands out in this criterion. It has all the usual facilities for handling
errors by the use of the do… except facility. First you write the do statement in
Silk Test your code and this will look to see if any exceptions are raised in the following
Tools

statements. If they are, then exception handling is passed to the lines following
the except statement.
Error handling in Visual Test is almost the same as in Robot. But it’s slightly
Visual Test
better in Visual Test as it has the ability to query error numbers (in a status panel).
Robot does not have a built in error recovery mechanism as Silk Test does.
However, you can use event-driven error handling by using the On Error function.
Robot
This function allows you to capture certain predefined errors using an error
handling procedure that processes the error as you wish.
Does not have any error recovery. Handles some exceptions. It does provide a log
QA Wizard file of the exceptions produced for debugging purpose.

Object Test

- 13 -
 WinRunner uses checkpoints to carry out object tests. The available checkpoints
are: GUI, Bitmap, database. One good thing in WinRunner is the ability to do
WinRunner
multiple checks in one sitting without having to go through a cycle of selecting
from the toolbar again.
It has a facility known as checks, which allows you to carry out various object
QA Run tests. These tests include bitmap, clock, file-aid, form, list, menu, site, table, text,
and window.
In Silk Test, you can verify the state of a given object. This can be achieved by
Silk Test looking at the class of object methods. VerifyFocus in the control class is the
Tools

method to do the test you require.

Very similar to Robot’s facilities. It’s also called verification. You usually record
Visual Test
verifications when recording a test scenario.
Robot’s facilities for doing object tests are known as verification points. The
following verification points are available: Alphanumeric, clipboard, Menu,
Robot Object data, Object properties, Region Image, Windows existence, windows
image, file comparison, file existence, module existence, website scan and
website compare.
It also has a verification and validation mechanism. One unique thing is that when
QA Wizard
you create an object, it validates the object as well.

Support
Mercury, in general, is considered to be one of the best in providing customer
support. You can use their website to get answers from one of their registered
WinRunner
users online very quick. Their helpdesk is also very good at answering and
helping you with your issues.
Tools

Helpdesk is pretty good. They also have one of the largest professional services
QA Run
team of all the test tool vendors.
Their support is pretty bad based on what we heard from some of the people that
Silk Test
are working on this tool.
Not a very good helpdesk support. But there is a good online mailing list at:
Visual Test
http://www.vtindepth.com/mt_info.htm
Some of the testing folks we talked to don’t like the helpdesk. They told us that
most of the time helpdesk people would tell you to refer to the manual. They
Robot
found that some online forums like: http://www.dundee.net.sqa/ is more helpful
than Rational helpdesk.
They claim to provide 24/7 helpdesk support for customers. We had sent one
QA Wizard
question to their helpdesk and got an answer back very quickly.

After evaluating each tool against the criteria, we have created a matrix and allocated a
number to each tool with respect to the particular criterion to see how good the tool
supported the criterion. We score from 1-3. 1 = Excellent support for this functionality, 2
= Good support but lacking or another tool provides more effective support, 3 = Basic
support only. Following is the matrix produced by us:

- 14 -
 Total Score
Image Testing
Record & Playback

Database Tests

Data Functions

Support
Web Testing

Object Mapping

Integration
Ease Of Use

Extensible Language

Test/Error recovery

Object Tests
Environment Support

Cost

Object Identity Tool

Tools
WinRunner 2 1 1 3 2 1 2 1 2 2 1 1 2 1 1 23
QA Run 1 2 2 2 2 1 2 1 1 2 1 1 2 1 2 23
Silk Test 1 2 2 3 3 1 2 1 2 1 3 1 1 1 2 26
Visual Test 3 2 2 1 3 3 3 2 1 2 2 2 2 2 2 32
Robot 1 2 2 2 2 1 2 1 1 2 2 1 2 1 2 24
QA Wizard 2 3 1 1 2 1 1 1 1 1 1 1 1 1 2 20

Summary

While our matrix score makes Visual Test the winner here, we would like to say that you
should use the demo-version of each product to see which meets your requirements.
Based on your requirements, any of these tools could turn out to be the best for your
need. Here are some of the strengths and weakness of each tool to help you make your
selection:

Vendors/Tools Strength Weakness

Rational (Robot, Visual Test)
Mercury (WinRunner)
Compuware (QA Run)
Seque (Silk Test)
Seapine (QA Wizard)
Economical, Lots of supporting Support, Language is a little basic
tools, good extensible language out of the box and a bit confusing
through VB, good data creation with tools breakdown.
facilities, good online community.
Possibly the most popular test tool, No Java support out of the box,
plenty of jobs, good support, good expensive.
online community, Good cross
browser support.
Strong professional services, good Poor online community.
integration with development
products, economical.
Good development language, good Bad helpdesk, expensive.
online community, recovery system.
Strong Java support, Very easy to Not used widely in the market, poor
use. online community.

Tool evaluation and selection is a project in its own right. It can take between 2 and 6
weeks. It will need team members, a budget, goals and timescales. There will also be
people issues i.e. "politics".
Start by looking at your current situation
 Identify your problems
 Explore alternative solutions

- 15 -
  Realistic expectations from tool solutions
 Are you ready for tools?
Make a business case for the tool
 What are your current and future manual testing costs?
 What are initial and future automated testing costs?
 What return will you get on investment and when?
Identify candidate tools
 Identify constraints (economic, environmental, commercial, quality, political)
 Classify tool features into mandatory & desirable
 Evaluate features by asking questions to tool vendors
 Investigate tool experience by asking questions to other tool users
 Plan and schedule in-house demonstration by vendors
 Make the decision

Choose a test tool that best fits the testing requirements of your organization or company.
An "Automated Testing Handbook" is available from the Software Testing Institute
(www.ondaweb.com/sti), which covers all of the major considerations involved in
choosing the right test tool for your purposes.

References

1. www.rational.com
2. www.mercury.com
3. www.seque.com
4. www.seapine.com
5. www.compuware.com
6. www.testing.com
7. www.sqa-test.com

- 16 -