11

Automating Functional Testing

using Business Process Flows
By Deepali Kholkar, Nikhil Goenka and Praveen Gupta
Process models promise easy and effective
functional testing
F
unctional testing of large complex business
applications poses multiple challenges.
While the test suite needs to be comprehensive,
test design and execution cycles are constrained
by time, effort and resource costs. An effective
functional test process is one that achieves a
trade-off between these two goals without
compromising on test quality. However,
there is no systematic method available
that helps test design teams come up with
a comprehensive functional test suite or for
selection of tests to achieve an efficient yet
effective functional test.
Syst ems are bui l t t o support an
organizations business processes. Functional
testing is required to be done at several stages
during the development and maintenance life-
cycle, viz., system testing, integration testing,
user acceptance testing, to evaluate the systems
compliance with its functional requirements.
It should test the core functionality of the
system and cover the various usage scenarios
comprehensively.
This requires testing system behavior
under specific conditions that may occur in
its life cycle. However, one may not want
to run the entire test suite repeatedly but
instead apply selection using a combination
of conditions.
Since there is no fixed method for
building the content of a functional test suite,
individual project teams defne their own test
methodology and test cases. The quality and
coverage of these test suites is completely
dependent upon the availability of expertise
and restricted by time.
The onus of building a test suite that
achieves functional coverage including special
cases and of optimizing the tests is completely on
the test design team who currently do not have
any tools to support this. Commercially available
tools offer test management and an automation
platform but do not help generating test content.
Tests are conventionally captured in
document form, which become diffcult to go
through and maintain for large or complex
SETLabs Briefings
VOL 9 NO 4
2011
12
systems. It is hard to get a handle on coverage
and track which functionality has been covered
under which set of conditions, from documents.
The tests, their variations and selection need to
be managed manually. There is no automated
mechanism to verify test documents against the
requirements.
This paper proposes a method for
test design of business applications based
on modeling application functionality as
busi ness process f l ows annot at ed wi t h
busi ness rul es and t est condi t i ons and
using the model for test generation and
test selection. This gives a formal method
to go about creating the test content from
the functional requirements. The structured
model form and visual notation make it
easy to go through and build in the missing
portions, enabling creation of a test suite that
achieves good functional coverage.
Exhaustive test coverage may not be
desirable during every test cycle due to time and
effort cost constraints, hence selection is provided
on the model for test optimization. The model is
verifed using model checking. Sample test data and
scripts to automate test execution are generated.
Since this is a top-down approach
to test design beginning with a high-level
process model as opposed to having just use
cases which are at the leaf level, it can help
developers be cognizant of the various usage
scenarios upfront, avoiding late detection of
gaps and defects when end-to-end scenario
testing is done for the first time during the
system test.
The approach offers some independence
from team confguration changes over time,
since a lot of the necessary domain knowledge
and test specifcations are captured in model
form. It offers the possibility of reuse across
applications.
RELATED WORK
Test generation from dynamic models of
various kinds has been a topic of research for
quite some time. Existing work deals with
generation of test cases from state transition
diagrams, state charts, collaboration diagrams,
sequence diagrams and activity diagrams. Most
of it is intended for unit or integration testing.
The approach described in this paper
is different in using business process models
that depict functionality without going into
implementation detail and in providing a
seamless, end-to-end approach from test design
to automation, usable by business analysts
and test teams. The process or workflow
notation maps easily to the functionality of
most business systems than state transition
diagrams or state charts. The collaboration and
sequence diagrams in many of the approaches
are specifed at a design or implementation
level, whereas test teams do not usually have
access to the system design or implementation.
Constraint solving is also not a new
concept and has been used for generation of test
data earlier. Our approach differs in applying
constraint solving to verify the pre- and post-
condition sequences in each path of the process
model to identify redundant paths and errors
and also to generate test data for each step of
the scenario.
Abdurazi k and Of f ut t descri be a
technique for generating tests from collaboration
diagrams and utilizing them for checking actual
system execution by instrumentation of the code
[1]. The approach is targeted towards unit or
integration testing.
Hartmann et. al., describe test scenario
generation and automation from activity
diagrams [2]. Pre and post conditions are
not processed for either data generation or
verifcation purposes. Transition coverage is
13
targeted as opposed to branch coverage in our
case. Guard conditions are defned for additional
selection and can be manually mapped to
data values. Business rule specifcation is not
supported.
Briand and Labiche describe an activity
diagram based approach to test generation
where each activity is further detailed as a
sequence diagram with Object Constraint
Language (OCL) constraints [3]. Their approach
addresses coverage of all possible test scenarios
by interweaving of activities. This however
leads to a very large number of combinations.
Generation is not automated. The approach
does not attempt to solve constraints for
verifcation or data generation. Test automation
is not dealt with in this work.
Frhlich and Link describe transformation
of use case documents into state charts and
generation of test cases from these [4]. A state
chart is at the level of a use case a higher
level process context is not available. Task
pre-conditions and actions are encoded and
posed as an artifcial intelligence problem to
find test sequences. Transition coverage is
ensured, which is equivalent to task coverage
in process model parlance. Formal specifcation
of conditions, verifcation, data generation and
test automation are not covered. Cavarra et. al.,
describe an approach to test generation based on
state transition diagrams [5]. Test automation or
data generation is not addressed here.
Bertolino and Gnesi discuss a method
to generate test cases for product lines using
the category-partition method by annotation
of textual use-case specification, hence test
generation is not automated [6].
MODELING SYSTEM FUNCTIONALITY
This approach proposes capturing system
functionality as a set of business process
fows, annotated with business rules and test
conditions. If a process model has already
been created in the requirements phase, it can
be reused. The functional test suite must come
from the functional requirements of the system,
by defnition.
Business process fows have been chosen
since it is intuitive to think of functional testing
as testing the business process fows of the
system.
Business Process Modeling Notation
(BPMN) is used as the notation for capturing
the process fows [7]. It is gaining in popularity
as a notation to express requirements and
is understood by all stakeholders, be they
requirement analysts, software developers,
managers or end-users [8]. This makes it
possible for test designers with knowledge of
just the domain or system functionality as a
black box, to specify the model. Knowledge of
internal details of system implementation is
not required.
The frst step is to draw the high-level
business process flows of the system. Each
step in the process fow can be an atomic task
or a sub-process that is further exploded as a
separate process fow. Important branch fows
in the processes that need to be covered in the
tests should be drawn, along with the branching
conditions that are usually the validation of
some business rule. Conditions are annotated
on the process fows using condition names as
labels. Condition names should be readable and
in domain terminology.
If data generation is desired, input
and output parameters of tasks need to
be model ed, whose dat a t ype can be a
simple data type or any entity from the data
model. The data model is required only for
modeling parameters, business rules and
later, for verification of the process model.
14
DATA MODEL
The data model for the application is captured
using the UML class diagram notation. It
captures the various entities, their attributes and
associations. Only entities from the domain model
need to be included, not the classes that are created
during system design and implementation.
The class model for part of an Insurance
system is shown in Figure 1. Business Rules for
the domain/ application are captured as rules
on the entities. These are invariants or conditions
that must hold in every state of the system. They
are expressed as OCL constraints on the object
model [9].
A business analyst or test designer who is
drawing the entity model need only specify the
rules in terms of labels and description in domain
terminology, e.g., PlanHolderMandatoryRule
for stating that a Plan Holder is mandatory
for every Plan; and attach them to an entity.
The development team can later fll in the OCL
constraint for each invariant since it requires
familiarity with OCL syntax.
BUSINESS PROCESS MODELLING NOTATION
BPMN has been accepted as an OMG tandard.
The complete BPMN is not required to be used
by our method. A simple BPMN process fow
diagram depicting the Process Payments batch
process of the Insurance system is shown in
Figure 2. The core BPMN is very similar to the
fowchart or UML Activity diagram notation.
Every process begins with a Start node. The
rounded rectangles are the tasks or activities
being performed during the process.
Each task could be an atomic task or a
sub-process. While there is no rule regarding
granularity of the atomic tasks, a thumb rule
that can be used is for each task to represent a
user-interaction with the system or a use-case.
For example, in the Process Payments process,
the individual use-cases ProcessPayment
and CheckForRejectPayments are tasks while
ProcessRej ectPayments is a sub-process.
The diamond-shaped nodes are the gateways or
decision-points in the process, where the fow
branches or merges. The arrow connectors are
Plan
Plan Payment
Method
Payment Reference
Payment Reference
T
r
a
n
s
a
c
t
i
o
n

D
e
t
a
i
l
s

F
o
r

P
l
a
n
P
a
y
m
e
n
t

M
e
t
h
o
d

P
l
a
n
L
i
n
k
e
d

T
o
Payment Reference Plan
Return DDs Payref
Payref Return DDs
Plan Reject Payments
Plan Reg With
Has-A
Plan Transaction
Plan Payment
Reg Wth Plan
Reject Payments Plan
Reject Payments
Reg With Instruction
Premium History
Plan Basic Transaction
Figure 1: Entity Model for an Insurance System Source: Internal Research
15
called sequence fows and depict the fow of
control in the process. Each sequence fow can
have one or more conditions associated with
it, that enable the fow. We have used these
conditions to express the Test conditions in
the process fows. These are annotated on the
sequence flows as pre- and post-conditions
before and after the tasks respectively. These
need not be just the pre- and post condition
contracts for the task, but are the conditions
that qualify the scenario or branch. Each branch
ends in an End node.
The diagram shows the various scenarios
that can happen during Process Payments, as
branch fows are annotated with the branching
conditions as labels. Most branching conditions
are validations of some business rule as indicated
here. Initially, the condition Rule0002 process
Payments indicates application of Rule0002
which gives the criteria for fetching payments to
be processed. After Task ProcessPayment, the left
branch is the scenario in which no payments to be
processed were found, indicated by the condition
label NotFound_Rule0002Payments. The other
branch shows the condition Rule0003Payments,
which looks for Rejected Payments. If found,
it calls sub-process ProcessRejectPayments
and ends by updating the associated Payment
Reference details.
TEST GENERATION
Models being formal, lend themselves to
automated analysis and verification. The
process model is traversed for generation of
Figure 2: BPMN flow diagram for Process Payments process
Process Payment
Check For Reject Payments
Rule 00002 Process Payments
Not Found_Rule 00002 Payments
Not Found_Rule 00003 Reject Payments Reject Payments
Process Reject Payments
Not Found_Payment Reference To Update Rule 00080 Payment Reference To Update
Rule 00003 Payments
Update Payment Reference
Payment Reference Index Updated
Source: Internal Research
16
test scenarios. Each scenario specifcation is
verifed. The pre- and post-conditions are used
for generation of test data. Test selection is
provided on the model, in order to generate a
pruned test suite.
Scenario Generation
Branch coverage is used as the criteria for
generation, hence all paths in a process fow are
covered using a depth-frst search. Each path or
branch becomes one test scenario for the system.
Each scenario is a sequence of task
tuples, each task tuple consisting of the task
and its specified pre-and post conditions.
Test suite T = {S1,S2.Sn}
where S1,S2.Sn are the scenarios.
Si = {t1,t2.tn}
where t1,t2.tk are the Task tuples
in scenario Si
tj = {prej, Tj, postj}
where prej, Tj, postj are the pre-condition,
task body and post-condition in task tuple tj.
The process flow may involve loops.
Every loop is unrolled once by the scenario
generator al gori thm. All scenarios thus
generated, along with their test data are output
as a test plan document.
Verifcation
Model checking has been used to verify the
process specifcation against the business rules,
detect inconsistencies in the process fow itself
and eliminate infeasible paths, which gives a
great deal of optimization in the number of
scenarios generated. The open-source model
checker Symbolic Analysis Laboratory (SAL)
[10] has been used for this purpose. For
verifcation and test data generation satisfying
the conditions mentioned in the process fow,
the conditions need to be defned using OCL
expressions. Each scenario path is converted
into a specification for SAL where the pre-
and post-conditions are translated into logic
expressions. The applicable business rules and
cardinality invariants from the class diagram
are also translated into constraints.
Changes effected by each task can be
provided very simply in terms of conditions
using OCL and actions for which there is a
simple syntax. Actions can be creation/ deletion
of objects/ associations or setting of attribute
values. This needs to be done only wherever
required, i.e., if later in the fow, the created/
deleted objects are referred.
Post-state of task = Pre-state of task +
changes effected by task
The model checker ver i f i es t he
specifcation for consistency. The pre-condition,
task specifcation and post-condition need to be
consistent with one another at each task level,
and also amongst the entire sequence of tasks
that make up the scenario for the model checker
to be able to fnd a path. Business rules are
verifed amongst themselves for consistency.
Pre- and post-conditions in a path are also
verifed against the business rules to ensure that
there are no conficts between them.
When SAL fails to fnd a solution for
any specifcation, there is a confict. The cause
could be an error in the specifcation or that
the conditions within the scenario do confict
i.e., the path is invalid and can be eliminated.
In the selected case study, 215 scenarios were
generated overall, of which only 130 were valid
and 85 were eliminated as infeasible scenarios.
The error in the specifcation could be either
a business rule being violated by a condition/
action or the conditions in the scenario conficting
among themselves. Errors should be rectifed and
solutions generated for all valid scenarios.
17
Test Data Generation
When the constraint solver fnds a solution to
the scenario specifcation, it fnds a sequence of
states that satisfy the pre- and post-conditions of
each Task in the sequence. Each state contains
values for the various defned data objects and
attributes. The generated data values in each
state conform to the pre-conditions as well as
the business rules. A pre-state is used as the
source of input test data for the task following
it. The initial state or scenario pre-state gives
sample pre-requisite data that must exist in
the system database for the scenario to begin
execution.
The input data generated for each task
needs to not only satisfy the pre-conditions for
that task but also of subsequent tasks in the fow,
so that the system is able to execute the entire
scenario. Also, it should be cognizant of changes
brought about by the preceding task(s), i.e., it must
come from the post-state of the previous task.
These criteria are naturally met by the constraint
solving approach, whereas in the conventional
approach, they have to be ensured manually. This,
coupled with the need to create the scenario
pre-state data makes manual test data creation
time-consuming and complex.
Test Automation
The scenarios are automated by generating test
scripts for them with the generated test data
embedded. The automation has been done
using the Test Drive tool [11] developed in our
lab, which currently generates scripts onto the
open-source web-based automation platform
Selenium [12].
Test Drive has a recorder which helps
record the sequence of UI actions required to
perform each atomic task in the above process,
just like any capture-replay tool. For each
generated scenario, the recorded sequences
for individual tasks are threaded together,
the generated input data is inserted and the
test script is created. These test scripts can
be replayed on any browser using just the
Selenium runtime engine.
Test Selection
The generated test cases are exhaustive and
the number can become really large due to
combinatorial explosion when the process
hierarchy is large and complex.
In such a case, running all tests is neither
feasible nor desirable. The test suite can be
pruned by using preliminary heuristics such
as, certain conditions are not required to be
checked for all possible paths. For example, an
error condition may need to be checked only
once. Selection on conditions is provided, where
conditions to be tested in only one or a specifed
number of paths can be stated. This results
in a drastic reduction in number of test cases.
Support for selection of specifed conditions
and tasks and selection using compound
conditions is being added. Although a very
simple concept, it allows the test team to specify
the functionality they would like to focus on in
a particular test cycle.
TOOL IMPLEMENTATION
The described approach has been implemented
using tools available within the lab and open-
source tools. Figure 3 shows a schematic
diagram of the tool/ approach. The Requirement
model process fows and class diagram, was
created using metamodelers, a modeling
tool developed within our lab that supports
modeling using BPMN and UML. The path
generator was developed using OMGen,
which is Metamodelers scripting language. It
traverses the process fow models to generate
the scenario paths.
18
The Model SAL translator translates
each scenario path into a SAL script. The SAL
model checker verifes and runs each script to
generate data. The test case generator picks
up the scenario paths, generated data and
the individual recorded use-cases to generate
automated test scripts. It also takes as input,
a simple mapping between the attributes in
the domain model and the input felds on the
screen.
INSURANCE APPLICATION: CASE STUDY
The t est generat i on approach has been
successfully tried out on processes from three
real life applications. The results are presented
here. To illustrate the application of the tool,
one of the case studies done on an Insurance
system, is described in detail.
Two online processes and one batch
process were modeled. The batch process
had a long and complex control flow, with
a requi rement document spanni ng over
a hundred pages. The test team had been
unsuccessfully trying to create test cases
manually for it.
The modeling approach helped the
team obtain tests for the batch process with
relative ease. They were able to easily draw
a process model for the complex description;
modeling in fact reduced the complexity of the
test case preparation task, helping visualize the
process flow and its branches. The team could
quickly generate tests which they were unable
to construct manually. Also, they were not
domain experts, but a relatively inexperienced
team.
Another important observation was
that even without the knowledge of this
approach, the test case writer draws a kind
of flow graph to clear their understanding of
the process.
One drawback from their perspective
was having to draw an extra diagram. The
two advantages were considerable time and
effort saved. The process diagram also became
an artifact that was used by the team later for
reference and discussions on coverage.
The Insurance Model
The class model in Figure 1 shows the Product
entity that denotes an insurance product, Plan
denoting a policy that is issued against the
Product to customers and many associated
entities storing different data about the Plan.
The customer, beneficiaries, payees etc. are
each modeled as a Relation of the appropriate
type.
Scenario Path Files
Path generator
Model ---> SAL
Translator
Scenario
Test Scripts
Generated
Data
Requirement Model:
Process flows + Class model
Mapping
File
Test Case
Generator
Test Drive
Recorded Use-Cases
SAL Scripts
Figure 3: Tool Schematic
Source: Internal Research
19
The Process Payments batch process
shown in Figure 2 looks for payments returned
by the bank due to various reasons. After
processing the records, the batch updates
the plan by reversing the premium payment
and generates letters informing the client.
The system tri es to match the returned
premium payment against a specifc item on
the plans premium payment history in order
to reverse it. The main processing happens in
the ProcessRejectPayments sub-process. The
complexity was due to the alterations of various
attributes based on the 150 odd rules present for
this particular batch process.
RESULTS
Data from the test generation exercise is tabulated
in Table 1. The average time taken by the team
to manually create tests was one week per
process. In this respect, our approach compares
favourably. The tool was found to scale up and
handle the large number of rules and flows
generated. It was also successful in pruning the
test suite to a very compact size. 1200 scenarios
were originally generated. After specifying ten
conditions that were to be tested only for a single
fow, the number came down to 215. These were
mostly error and exception conditions.
On verification, 85 infeasible paths
where conflicting conditions occurred were
identifed. These were all found to be redundant
paths where for example, if the flow had a
branch based on variable value=A, B or C, there
was later in the fow another branch based on
a check for the same variable value=A, B or
D. This resulted in several redundant paths
being created with conficting values chosen
for the same variable, which were identifed
by the constraint solver as unsatisfable. The
fnal number of scenarios was thus reduced to
130. The scenarios were verifed by the team
as being a satisfactory set of tests. The metrics
above clearly bring out the considerable time
reduction in test design, but not the total
productivity in test automation. Automation
was not carried out since the implementation
was not web-based and our tool currently
supports automation only for web-based
applications. The batch process had not been
implemented. Hence no data about defects
or execution time and effort are available.
However, the generated tests provided 100%
task and condition coverage and also covered
most of the scenarios.
The test generation and test automation
components have been separately tried on
several real-life projects. The automation
component is being extensively used in a few.
The end-to-end approach has been tried on
several case studies within the lab.
CONCLUSION
The approach aims to provide functional
test teams with a model based method and
framework to design a functional test suite
that gives an assurance on test suite quality
and coverage. The fact that this approach can
be used by business analysts to generate and
Process
Validate New
Application
Capture
Payment
Process
Payments
Batch
Time to
understand
functionality
3 days 1 day 5 days
Number
of Rules
100+ 25 150
Time taken
to model
1 day 2 days 2 days
Number
of Tests
generated
7 42
Total: 1200
Selected: 215
Feasible: 130
Table 1: Results from Insurance Case Study
Source: Internal Research
20
automate tests is its forte. The model and visual
notation capture not just a test specifcation but
a lot of details about the system and becomes
a valuable artifact in itself, besides being used
to fnd errors in the specifcation and drive
selection for achieving optimal test cycles.
It has been tested on case studies drawn
from real projects and was found to scale up. It
was also interesting to fnd that one need not use
the entire process right up to test automation
and data generation to make an impact. Just
modeling and verifcation was found in the
mentioned case study, to provide enormous
benefts in terms of ease of specifcation and
test generation when the team was unable to
create them manually.
REFERENCES
1. Abdurazik, A. and Offutt J., (2000),
Using UML Collaboration Diagrams for
Static Checking and Test Generation, in
the Proceedings of Third International
Conference on the UML, pp. 385395.
2. Hartmann, J., Vieira, M., Foster, H.
and Ruder, A. (2005), A UML-based
Approach to System Testing, Innovations
in Systems and Software Engineering,
Vol. 1, No. 1, pp.12-24.
3. Briand, L.C. and Labiche,Y. (2001), A
UML-based Approach to System Testing.
Software System Model, Vol.1, No. 1,
pp. 10-42.
4. Frhl i ch, P. and Li nk, J . ( 2000) ,
Automated Test Case Generation from
Dynamic Models, in Bertino E (ed.)
Proceedings of the ECOOP, pp. 472491.
5. Cavarra, A. , Davi es, J. , Jeron, T. ,
Mournier, L., Hartman, A. and Olvovsky
S. (2002), Using UML for Automatic Test
Generation, in the Proceedings of ISSTA.
6. Bertolino, A. and Gnesi, S. (2003), Use
Case-based Testing of Product Lines, in
Proceedings of the ESEC/SIGSOFT FSE,
pp. 355358.
7. OMG BPMN 1.0 specifcation. Available
at http://www.bpmn.org/Documents/
OMG_Fi nal _Adopt ed_BPMN_1- 0_
Spec_06-02-01.pdf.
8. Buchanan, I. (2005), Requirements
El i ci t at i on wi t h Busi ness Process
Model i ng Not at i on. Avai l abl e at
http://conferences.embarcadero.com/
article/33351.
9. OMG Obj ect Constrai nt Language
s peci f i cat i on V2. 0. Avai l abl e at
http: //www. omg. org/technology/
documents/formal/ocl.htm.
10. SAL Modelchecker. Available on http://
sal.csl.sri.com
11. Patel, S., Gupta, P. and Surve, P. (2010),
Test Drive - A Cost Effective Way to
Create and Maintain Test Scripts for Web
Applications, in the Proceedings of 22nd
International Conference on Software
Engineering , pp. 474-476.
12. Selenium Open-source Web Application
Testing Platform. Available on http://
seleniumhq.org/.
For information on obtaining additional copies, reprinting or translating articles, and all other correspondence,
please contact:
Email: SetlabsBriefngs@infosys.com
Infosys Limited, 2011
Infosys acknowledges the proprietary rights of the trademarks and product names of the other
companies mentioned in this issue of SETLabs Briefngs. The information provided in this document
is intended for the sole use of the recipient and for educational purposes only. Infosys makes no
express or implied warranties relating to the information contained in this document or to any
derived results obtained by the recipient from the use of the information in the document. Infosys
further does not guarantee the sequence, timeliness, accuracy or completeness of the information and
will not be liable in any way to the recipient for any delays, inaccuracies, errors in, or omissions of,
any of the information or in the transmission thereof, or for any damages arising there from. Opinions
and forecasts constitute our judgment at the time of release and are subject to change without notice.
This document does not contain information provided to us in confdence by our clients.
Authors Profiles
DEEPALI KHOLKAR is a Scientist at Tata Research Design and Development Centre (TRDDC). She
has several years of experience in model driven development and her current area of work is model
based testing and formal specification and analysis. She can be reached at deepali.kholkar@tcs.com.
NIKHIL GOENKA is a Systems Engineer at TRDDC. His research interests include model based
testing and formal specification and analysis. He can be contacted at nikhil.goenka@tcs.com
PRAVEEN GUPTA is a Systems Engineer at TRDDC. His research interests include model based
testing and formal specification and analysis. He can be contacted at praveen2.g@tcs.com.