Test Automation for Higher Testing Productivity

Presenter : Nadeem S A
Designation : Test Manager
Email : nadeem@relq.com

RelQ Software Pvt. Ltd

Agenda

ƒ Introduction
ƒ Constraints in Manual Testing approach
ƒ What is Test automation?
ƒ Implementing Test automation for Higher productivity
ƒ Feasibility Analysis
ƒ Tools and their selection criteria
ƒ Test script development methodologies
ƒ Test Automation frameworks
ƒ Case study
ƒ Conclusion
ƒ Q&A

2
Introduction

3
Software Testing

ƒ Necessary to uncover hidden defects and improve the quality

and reliability of software.

ƒ Inadequate testing will be potentially disastrous to the

business.

4
Software Testing
ƒ Is a specialized task

ƒ To test applications of different domains

ƒ Require resources with Domain experience

ƒ To test systems built using heterogeneous environments

ƒ Require resources with different technology skills

ƒ To perform different Test types

ƒ Require resources with different Testing skills

ƒ Consumes Time and resources to plan and execute tests

5
Challenges In Testing

6
Challenges

ƒ How to perform the different types of Tests to get a quality

product ?

ƒ Consistently without cutting down the amount of Tests

ƒ Without impacting cost in a major way

ƒ Without impacting the release deadlines

7
Productivity Constraints of Manual Test Approach

8
Manual Test Approach - Constraints

ƒ Time consuming and depends on the productivity of the

Tester

ƒ As functionality grows per release, the effort and the schedule

required to test the application increases

ƒ If functionality increases and the Test schedule and resources

remain constant, then the Test coverage gets impacted.

9
Manual Test Approach - Constraints

ƒ Difficult to execute Performance tests that require large loads

using Manual Test approach

ƒ Manual testing is taxing on the testers who repeat the same

tests for an extended period of time

10
Why consider Test Automation?

ƒ To consistently achieve good test coverage across all

releases despite functionality increase

ƒ To deliver more in less time without cutting down the tests

11
Test Automation

12
What is Test Automation?

The use of software to

ƒ Set up preconditions for a test

ƒ Initiate and control the execution of tests

ƒ Compare actual and expected results

ƒ Perform the required reporting functions

13
Implementing Automation for Higher Test Productivity

14
 Test Automation Process

System study Application Selection of test

Tool evaluation automation
and confirm navigation
and selection methodology
technical requirement using the tool

Application support Application

/development updates

Review results Review and

Perform tests/ Build test scripts
and report approve
regression testing and test drivers
the defects scripts

Enhancement of
test scripts
Test summary report

15
Importance of Feasibility Analysis

16
Feasibility Analysis

ƒ Automation feasibility is used to evaluate

ƒ Compatibility of Automation tool with application

ƒ Are all GUI objects getting identified ?
ƒ Do I need to have additional plug-ins ?

ƒ The need to use additional technologies

ƒ Need Perl or other scripting languages ?
ƒ Need to store Test data in database ?

ƒ The feasibility of automating the required test cases.

17
 Feasibility Study

ƒ Outcome of Automation feasibility study

ƒ How much complexity exists in implementing Automation for

current and future needs?

ƒ What improvement in Test coverage does Automation provide?

ƒ How much Test effort reduction does Automation enable?

ƒ Whether the right tool has been selected for Automation.

18
Selecting the Automation Tool

19
Selection criteria for Automation tools

‰ TYPE OF TEST
ƒ Functional Testing
ƒ Silk test, Winrunner, QTP, Rational Robot, Test Complete,
Test partner
ƒ Performance Testing
ƒ Load Runner, Silk Performer, QA Load
ƒ Unit Test
ƒ JUnit, NUnit
ƒ Test Management
ƒ Test Director, Silk Central, Quality Center, QADirector

‰ USABILITY
ƒ Ease of use of tool features

20
Selection criteria for Automation tools
‰ COMPATIBILITY
ƒ With the application for current and future needs
ƒ With required OS (Windows XP / NT / 2000 / Unix / Linux)
ƒ With required Browsers (IE, Netscape etc)

‰ LEARNABILITY
ƒ Adequate Training material and technical support
ƒ Online Community

‰ CAPABILITY
ƒ Support for Java / .NET / Oracle Tech.
ƒ Support for Client Server / Legacy Systems

21
Automation – Specialization in Testing

22
Setup best practices
ƒ Capture the requirements by involving end users of the
suite

ƒ Design inline with the requirements

ƒ Select teams with programming skills.

ƒ Code as per identified standards / guidelines

ƒ Develop re-usable scripts

ƒ Maintain version control

23
Set up Best Practices

ƒ Inadequate Testing of Automated scripts

ƒ Causes scripts to break easily

ƒ Create repository of tested and re-usable scripts

ƒ Reduces time to develop automated scripts to test new
functionality

24
Plan for maintenance and use

ƒ Provide user manuals that contain information about

ƒ The features of the Automation suite
ƒ How to use the features of the suite
ƒ How to perform the Test and interpret results

ƒ As functionality grows, scripts may have to be added or

enhanced.
ƒ Train the end-users on how to maintain the suite

25
Test script development

26
Test script development Methods
ƒ Record and playback

ƒ Data driven

ƒ Keyword driven

ƒ Hybrid

27
Record and Playback approach

ƒ User activates the “record” feature

Application Application Application
of the tool.
Version 1 Version 2 Version ‘n’
ƒ User performs the required actions
for the test on version 1 of the
Playback Modify
application.
Record
Playback
ƒ The automation tool generates
scripts by recording user actions.
Test Scripts

ƒ The generated scripts can be played

back on version 2 to reproduce the
exact user actions.

28
Record and Playback approach

Advantages:
ƒ Less effort for automation and quick Returns
ƒ Does not require expertise on the tool

Limitations:
• High dependency on the GUI of AUT
ƒ Scripts contain hard coded data
ƒ Not a recommended approach for developing scripts.

29
Data Driven approach

ƒ Test scripts are freed from hard coded data.

ƒ Test data is got from external files.

ƒ Using the scripting language features of the tool, create a

Library of functions to
a) Perform the application specific tasks
b) Read the test data files
c) Log test results

ƒ Group the functions to develop test cases

ƒ Invoke the scripts using Test drivers

30
Data Driven suite - Architecture

Additional Info:
APPLICATION UNDER TEST

Test Mgmt. tool ƒ Config files : Allow the user to

a) Select the test cases
Test Case
Mgmt tool
Automation Defect Test b) Select the type of test
tool Mgmt tool Results

ƒ Object Repository: Can be an

Driver Config external file storing the required
Scripts Files
properties

Scripts ƒ Data Files: Store the required

data
Object
Data
Files
Repository
Libraries ƒ Result files: Store results in
customized format

31
Data Driven approach

Advantages:
ƒ Data is separated from scripts and stored in data files

ƒ The volume of test data and its combination can be increased for
exhaustive testing

Limitations
ƒ Application must be available to start scripting

ƒ Maintenance of the test script due to GUI changes is high

32
Keyword driven approach

ƒ Identify the Keywords and related data for each task

ƒ Create a Library of functions to the task specific to each

keyword

ƒ Group the keywords to create required Tests in Action files

ƒ Using Test drivers, read the Action files to execute Test

cases

33
Keyword driven approach – Sample Action file

Auto_ID Command Object Data TC_Execute?

AUT0 - 001 TCStart YES

AssignValue %URL%:=https://www.compan
y.com
LaunchApp %URL%
TCEnd
AUT0 - 002 TCStart YES
AssignValue %loginUserID%:=UserID
AssignValue %Password%:=Password
EnterData loginUserID %loginUserID%
EnterData loginPassword %Password%
ClickButton Submit
VerifyResult LoginSuccessfulMsg
TCEnd

34
Keyword driven suite Architecture

Additional Info:

ƒ Action files: Contain test cases

formed using keywords

ƒ Config files : Allow the user to

Select any Action files for
execution

ƒ Object Repository: External file

storing the required properties

ƒ Result files: store results in

customized format

35
Keyword driven approach

Advantages:
• The User can easily create the test scenarios by grouping a set of
Keywords in the Action files.
• The action files can also be a single point of reference for Manual
and Automated Test cases.
• Maintenance is easy as changes in functionality require changes in
Action Files only

Limitations:
• The number of action files increases with functionality.
• The initial effort to design is more compared to data driven approach
• Users will have to be trained on the use of Keywords

36
Hybrid

Scripts developed using a combination of Data Driven and

Keyword Driven approach.

Advantages:

ƒ Use the best features of different methodologies to suit the

project requirements.

Limitations:

ƒ Maintenance is an issue.

37
Test Automation Framework

38
What is a Framework?

ƒ Framework is a logical support structure in which another

software project can be grouped and tested

ƒ Framework comprises support programs, processes, code

libraries etc to help develop and glue together the different
components in a project.

• Can be implemented by leveraging proprietary automation

frameworks and tool sets from leading vendors, supported by
additional programming and processes to maximize the
benefits of test automation.

39
Objectives of Automation Framework

40
Objectives of Frameworks

ƒ To gain long term benefits by ensuring

ƒ Reusability of Automated scripts across projects

ƒ Scalable environment for tests

ƒ Effort for test environment setup is reduced

41
TYPES OF AUTOMATION FRAMEWORKS

42
Types of Frameworks

ƒ Process framework

ƒ Test Framework

ƒ Hybrid framework

43
Process Framework

ƒ Used to automate test environment setup process for multiple

tests having similar Test environment.

Command for Automated

Test 1 Test 1

Command for Automated

Test 2 Automated Test 2
Test
End User Environment
Setup

Command for Automated

Test n Test n

44
Process Framework - Advantages

ƒ Saves effort in test setup process

ƒ Scalable for Multiple types of tests

ƒ Provides an easy interface for end user to execute tests

45
Test Framework

ƒ Focus on creating reusable components that can be used for

similar tests across projects.

Functional Test1
Project A
Functional Test2 Repository
Of
−Test cases
−Test Scripts
Functional Test1
Project N
Functional Test2

46
Test Framework - Advantages
ƒ Re-usability of components across projects reduces
redundancy

ƒ Re-usability of components also reduces the overall time to

develop the Test suite

47
Hybrid Framework

Framework that combines the features of process and test

frameworks

48
CASE STUDY

49
Case Study 1

A US based client, wanted to use Automation to resolve the

following issues.

ƒ Improve the Functionality Test coverage for Localized

(BEFIGS) builds of the Desktop application

ƒ Reduce the effort / time to perform the different tests

Additionally, a Test Automation framework that would enable

end to end testing for localized Desktop application had to be
developed.

50
Case Study 1

Develop test automation suites using Silk test for the following.

ƒ Capturing UI differences in Localized (BEFIGS) builds

ƒ Test installer functionality for localized (BEFIGS) builds

ƒ Capturing Registry and File attribute differences against

benchmark data after installation

ƒ Testing functionality of all modules of localized (BEFIGS)

builds

51
Case Study 1

ƒ Test requirements

ƒ No of Products = 2
ƒ No of Test cases = 1400 test cases each for US and
Localized builds (BEFIGS Languages) = 1400 x 7
ƒ Average No of builds per product for US = 10, BEFIGS = 10

ƒ Challenge
ƒ Manual Execution of all test cases on all environment –
Impossible.

ƒ Solution
ƒ Framework based Automation

52
Case Study 1

ƒ Automation Framework requirements

ƒ Should provide a common Command format for the user to

execute any of the Installer or Functionality or Registry-file
compare or UI Compare suite

ƒ Should allow for testing the suites on any of Win98 or Win

2k or Win XP OS

ƒ Should allow for executing any suite on any build on any OS

for any of the BEFIGS languages

ƒ Should automate the process of test setup environment and

execution

53
Case Study 1

Format of Command to execute test =

Go [Test] [Build] [OS] [Language]

SL COMMAND DETAILS
NO PARAMETERS
1 Go Command to initiate the execution of the test

2 Test The first argument represents the following tests

a) Installer b) Functionality c) UI capture d) Reg-file
compare
3 Build The second argument represents the build that needs to be
tested
4 OS The third argument represents the operating system
(Win2k, Win 98 and Win XP) on which the build to be tested
6 Language The fourth argument represents the language of the
localized build that needs to be tested.
a) Italian b) French c) Spanish d) German e) Brazilian
Portuguese

54
Case Study 1

Process framework implementation

55
Case Study 1

Components of the framework

ƒ Automation suites (Developed using Silk test 6.5)

ƒ Perl scripts

ƒ DOS Batch files

ƒ OS images

ƒ VM Ware

56
Case study 1

Effort to develop framework = 12 Man months

ƒ 3 resources
ƒ 4 months

Effort to develop automation suites = 42 Man months

ƒ 7 resources
ƒ 6 months

57
Case Study 1 - Architecture

Test Repository
1.Test Scripts
2.User Defined
Library
Test Script Test Environment
Test Script
3.GUI files Selection File/
Selection File/
4.Utility Scripts SCM
SCM Test Client
5.Test data files Test Client
Machine cluster
6.Tool Installer Machine cluster

.…
.… Test
database
TestController
Test Controller (optional)
Test Controller Script

Check out scripts

Compile/build
System/application
System/application
Deployment scripts under test
under test
Test Setup scripts
Test data generation-scripts ….
….
Test execution scripts
Report generation script
Report publishing script
Report mailing script Web
Web
Server
Server

58
Case Study 1

Reduction in Effort to execute Test cases using Automation suite

Task Manual Effort Automated Effort Saving in

(Man days) (Man Days) Effort
Functional Test 14 4 71 %

Te st Effort re duction

15
Effort in days

10

0
Manual Automated
Type of Testing

59
Case Study 1

Improvement in Test coverage using Automation

Task Manual approach Automated %

approach Improvement
Functional 20 % 80% 60 %
Test Coverage

Te st cov e rage improv e me nt

%Coverage - localized

100
80
build

60
40
20
0
Manual Automated
Type of Te s ting

60
Case Study 1

Reduction in Effort to Setup Test environment

Task Manual Effort (Minutes) Automated Effort Saving in

(Minutes) Effort
Test Environment 120 20 83 %
Setup

Test environment setup effort reduction

150
Effort in minutes

100
Ef f ort
50

0
Manual Automated
Type of Te s ting

61
Case Study 2

ƒ The Client
– A leading Bank having operations across the globe

ƒ The System
– Online Cash Management System, with support for
multiple branches

ƒ Issue faced by client

– Functionality increases per release, need to ensure
adequate test coverage in the fixed Test schedule ( 4
weeks)
– Need to reduce the Test effort
62
Case Study 2
ƒ Tool
– QTP 8.2

ƒ Methodology
– Keyword-Driven

ƒ No. of Test Cases

– 3,500 ( x 5 branches)

ƒ Scripting languages
– Perl, VB Script

63
Case Study 2

ƒ Automation Suite requirements

ƒ Should be possible to execute the test cases for any of

modules for any of the five branches

ƒ It should be possible to configure and selectively execute

any/all testcase(s) in any of the Action files

ƒ Summary and detailed results of Test case execution should

be available for all Test case executions

ƒ Should be portable on Win2k / XP English and Japanese

OS.

ƒ Results should be emailed to selected recipients

64
Case study 2

ƒ Effort to develop automation suite = 16 Man months

ƒ 4 resources
ƒ 4 months

65
Case Study 2 - Architecture

66
Case Study 2

Reduction in Effort to Execute Test Cases using Automation suite

Task Manual Effort Automated suite Saving in

Effort Effort
Executing 10,000 117 Man days 20 Man days 83 %
Test cases

Test Effort reduction

Effort inPerson-Days

140
120
100
80
Effort
60
40
20
0
Manual Automation
Type of Testing

67
Case Study 2
Improvement in Test Cases execution coverage
Task Initial Coverage Current Coverage Improvement in
coverage
Executing test cases 50 % 90 % 40 %

Improvement in Coverage

100

80
%Coverage

60 Manual
40 Automation

20

0
Coverage
Type of Testing

68
CONCLUSION

69
Conclusion

ƒ Test Automation does not eliminate Manual testing but

supplements it.

ƒ Test Automation requires skilled resources to be implemented

properly.

ƒ A framework based approach for test automation supports

maintainability, extensibility, scalability, ..

ƒ Test Automation Framework significantly improves the

productivity of testing.

ƒ Very useful in tests that are to be repeated in different

platforms.

70
Conclusion

ƒ Using Automation, significant improvement in coverage of

tests during regression cycles is achieved.

ƒ Integrated with a test management solution, provides

complete solution for automated testing, reporting, test
coverage and defect logging.

ƒ Testing team knowledgeable of the application and test cases

is preferred for test automation.

71
Q&A
Thank You

73