IBM Business Process Manager testing methodology,

Part 1: General testing guidelines

Allen Chan (avchan@ca.ibm.com)
STSM, IBM Blueworks Live and IBM BPM
IBM

10 December 2014

Jing Jing Wei (weijingj@cn.ibm.com)

Software Engineer
IBM
Jerome Boyer (boyerje@us.ibm.com)
Senior Technical Staff Member
IBM
Peng Guo (guopeng@cn.ibm.com)
Staff Software Engineer, IBM BPM SVT
IBM
Hong Yan Wang (whongyan@cn.ibm.com)
Staff Software Engineer
IBM
Hong Li Liu (lhongli@cn.ibm.com)
Software Engineer, IBM BPM SVT
IBM
This tutorial is part 1 of IBM Business Process Manager Testing Methodology series. It
describes the necessity of IBM Business Process Manager (BPM) project testing, general
testing guidelines and when to test in the IBM BPM project lifecycle. This tutorial shares
good practices of IBM BPM project development and helps you get the big picture of
quality assurance for business process applications. This series of tutorials describes test
methodology for IBM BPM projects. Part 1 describes the necessity of IBM BPM project
testing, general testing guidelines, and when to test in the IBM BPM project lifecycle. Part
2 describes details of various testing that should be included in IBM BPM project tests and
specific information for environments. Part 3 discusses good testing practices, automation, and
testing tools in IBM BPM.
Copyright IBM Corporation 2014
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Trademarks
Page 1 of 13

developerWorks

ibm.com/developerWorks/

View more content in this series

Introduction: Why do we need IBM BPM project testing?

IBM Business Process Manager (BPM) is widely used by enterprises to manage, monitor, and
improve their business processes. Business process management plays an important part in an
enterprise's strategic plan. Therefore, IBM BPM projects demand high quality.
IBM BPM projects are commonly a set of complex applications. The applications can contain
various business processes, data objects, and services that can be stored in different applications
and toolkits and be accessed by the IBM BPM Process Portal, Process Admin Console, REST
APIs, JavaScript APIs, or even exposed HTTP URLs. As requirements change and new features
are added, IBM BPM projects are modified and saved as new versions, called snapshots. Process
instances from different snapshots are allowed to be migrated between versions. At the same time,
nonfunctional requirements such as performance, high availability, and disaster recovery must be
considered during IBM BPM project deployment.

Try the Workflow service

Create long-running, stateful workflows that orchestrate tasks and services with synchronous
or asynchronous event-driven interactions with the Workflow service from Bluemix. Try it for
free!

To make sure IBM BPM projects work well, it is important to ensure the quality of the process
application and ensure that it meets the functional and nonfunctional requirements of the business.
Both test teams and development teams should consider the following goals when developing the
projects:
Produce reliable process-based solutions that achieve the business outcomes.
Provide confidence that the system is configured correctly and the application is developed
correctly for a production deployment.
Establish a baseline to test against the following types of future system changes:
IBM BPM product upgrades
Server modifications
Network changes
Application upgrades
Application server upgrades
Database upgrades
System load dynamics changes over time
Provide visibility into system scalability to allow for budget planning based on anticipated
growth in system load. For example, growth might be anticipated at an annual rate of 10%,
starting with 500 concurrent users.
Troubleshoot performance problems and provide solutions or recommendations where time
allows.

General testing guidelines

Although applications vary by requirements and features, you can apply the following common
testing guidelines to use with your IBM BPM project:
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 2 of 13

ibm.com/developerWorks/

developerWorks

Make sure to have adequate resources for each test purpose.

Make sure that you have automation tools.
Use monitoring and analysis tools.

Use adequate resources for each test purpose

To ensure that the project works well, the testing team must maintain adequate resources and
dedicate a testing environment for each test purpose. Generally, test areas include functional
testing, nonfunctional (or relational) testing, and regression testing.

Functional testing
Functional testing is basic quality assurance that makes sure that software components work as
expected when integrated. Functional testing can be divided into the following test phases:

Unit testing
Integration testing
User acceptance testing
Instance migration testing
Globalization testing
Mobile or browser testing

Unit testing is the first quality guarantee during project development. Use the continuous
integration approach for unit testing to ensure the test coverage for each functional unit.
For individual IBM BPM services, business process definitions (BPDs) and facades, or interfaces,
use a test harness that includes the following 3 sets of data with predictable results:
The good path, also called the happy path
The bad path, also called the unhappy path
The ugly path, also called the system exception path
You cannot ensure that the IBM BPM project works properly under high stress testing, even after
you complete basic functional testing. Always test with some volume instances. Ideally, use a total
of 10,000 instances, and at least 200 in-flight instances in a known state. For performance testing
(as part of nonfunctional testing), you need a higher volume.
User interfaces are important because they impress end users more than other functional
components. Consider involving users in parts of the project that have human intervention,
especially for evaluating the usability of the user interface.

Nonfunctional, or relational, testing

Nonfunctional testing, also called relational testing, is as important as functional testing. It relates
to the project security, reliability, interoperability, robustness, and maintainability. Relational testing
includes the following testing:
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 3 of 13

developerWorks

ibm.com/developerWorks/

Performance testing
High availability testing and disaster recovery testing
Stability and stress testing and endurance testing
Security testing

Use test scenarios for testing that make sense and that you understand how to scale and
extrapolate correctly. For example, have realistic expectations about the complexity of the
application and the realistic response time to simulate real human interaction.
To understand the maximum capacity that the hardware will support, benchmark the IBM BPM
servers with a simple process before you start development and testing. During nonfunctional
testing of your custom process, run the benchmark to compare to the baseline.
To understand how performance of external systems affects the overall process and to account
for it in your process design, write separate testing for all each contact to external systems. For
example, take into account that external systems might not respond immediately when under a
heavy load.
During all tests, include threshold key performance indicators (KPIs) that are monitored. If a KPI is
exceeded, chase down that cause and don't let errors accumulate.
Test performance with a combined manual and automated approach. Use the manual approach for
human intervention, for example, measuring response time for a web page. Use automated testing
for other parts of the project.

Regression testing
New changes and functions are continuously added to IBM BPM projects, and the changes might
bring in new defects. Make sure to include a subset of functional and nonfunctional tests to run
whenever there is a change to the system, or whenever you deploy a new version of an IBM BPM
application.
Adopt a rolling upgrade approach to ensure that any changes to the system are first tested in the
test environment, before you apply them to the production environment and the Process Center.
For more information, see Good practice - Use rolling upgrade when you update IBM BPM on the
IBM Business Process Manager wiki.

Use automation tools

Automation testing plays an important role in IBM BPM project testing. Consider the following tips:
Automation testing can save human resources from repeated regression tests.
Manual configuration is required before business application testing, but it is time-consuming
and error-prone.
It's easy to keep static, reliable test sets as an acceptance standard for checking new
changes.
High automation testing coverage helps guarantee agile project development.
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 4 of 13

ibm.com/developerWorks/

developerWorks

The testing team adopts automation tools and quickly builds automation test packages. Make sure
to choose automation tools that the testing team is skilled in, which reduces the learning curve and
the effort to implement and maintain automation packages.
Human tasks are widely used in IBM BPM projects. Testing teams should choose one or more
automation tools to test web-based technology and mobile-based technology, if applicable.
Choose the static, reliable, well-documented test cases to automate. The automated test cases
should at least cover the main path, and then can be used for regression testing.
The automation results should be detailed so that problems are easy to locate.

Use monitoring and analysis tools

Monitoring and analysis tools can help you monitor the testing and production environments to
make sure they are free of business or system problems. Some can also help to troubleshoot
problems that occur in testing and to analyze system bottle necks.
Finally, some testing, monitoring and analysis tools are required for common testing procedures,
such as performance testing.
IBM BPM itself contains some built-in monitoring and analysis tools such as the Process Admin
Console Instrumentation page, the Process Monitor page, the Performance Admin Console, and
IBM Business Monitor. Use the tools to monitor, analyze business processes, generate reports and
help improve process definitions.
Some monitoring and analysis tools can help you debug the defects or problems that occur in
TM

testing. For example, you can use the Fiddler and Firebug tools to debug problems for coaches.
You also can use Firebug to capture HTTP requests that are used in automation test package
development. For some testing, like stress testing or high availability and disaster recovery testing,
you might need additional monitoring and analysis tools, such as IBM Whole-system Analysis
of Idle Time (WAIT) to capture, analyze, and measure various KPI data. Part 3 of this series
discusses monitoring and analysis tools is more detail.

Testing in the IBM BPM project lifecycle

Agile methodology is recommended when developing IBM BPM applications, so it is important
to also take test and validation into consideration while planning project development cycles. In
addition, whenever you are about to push out a new snapshot or version of a business process
into production, unless it is a new feature, it is crucial to also perform a set of regression testing.
Select representative test cases from your functional and nonfunctional testing. If you are updating
existing active processes to a new version, make sure that you also perform instance migration
tests.
Use the IBM BPM environments properly. IBM BPM supports management and installation
of IBM BPM applications into your own test environment, staging environment, and finally the
production environment. Versioning and snapshot handling is different in Process Center and
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 5 of 13

developerWorks

ibm.com/developerWorks/

Process Server, so make sure to use your test Process Server for integration testing, iterative
testing, and regression testing. Staging should be identical or very similar to production in capacity,
configuration, and security.

Continuous iteration testing

Continuous iteration testing is not just about running tests. It is a broad concept for a range of
activities that include finding defects, gaining confidence about the level of quality, and providing
information for making project decisions. Most IBM BPM projects follow an incremental and
iterative development approach. Testing should not be delayed until the last phases of the project,
such as the unified process transition phase or the user acceptance testing phase. Instead, include
testing during each iteration. Quality assurance is started at the early phases of the project with
test planning and definition that takes place during requirement elicitation and process modeling.
After development iterations are started, the quality assurance team can analyze existing
requirements (described as user stories, acceptance criteria and use cases) to design the tests in
scope for the iteration. For an iteration (i), the tests will be executed in the next iteration (i+1) using
the base code of the first iteration (i). Do not try to do testing on features that are still under unit
testing and development. You might find issues the developers are most likely aware of and are
trying to fix. To make this team effort work well, incorporate the people who are doing the testing as
part of the project team, not as people who receive the code thrown over the wall.
Many discussions happen during the construction of business processes, coaches, and the
information model. Requirements are adapted, and acceptance criteria are fine-tuned. Testers
should not expect a rigid requirements document finalized at the beginning of the project to
develop functional testing from it. Testers have a lot of experience with setting and testing
acceptance criteria and with environment constraints. Developers will find that testers have
valuable knowledge about how systems to be integrated work together. See Figure 1 for all the
phases of continuous interation testing.

Figure 1. Iteration testing

The goal of defining a test is to describe and implement the input to the test. The input can
be values that come from the keyboard or a graphical user interface (GUI), or are read from a
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 6 of 13

ibm.com/developerWorks/

developerWorks

database or spreadsheet. The expected results can include the pre-conditions or the state of the
system and what needs to be true for the test case to run as expected.
Figure 2 represents the organization of project activities by phases and practices, or the work
stream. A traditional IBM BPM project includes more work streams, but the focus of this picture is
on the IBM BPM and testing work streams.

Figure 2. IBM BPM specific continuous iteration test activities

During the development iterations, the process developers and integration developers use unit test
capabilities to make sure that the features and user stories that they are developing work properly.
In parallel, the testers define the tests for the next iteration and are involved in any process
modeling discussions. Testers report at the daily scrum meetings where crucial information can
be exchanged. Testers are fully aware of the iteration backlog, the release backlog, and the test
strategy that is developed at the early phase of the project. Each iteration ends with a formal
demonstration that is presented by one of the business stakeholders, which is call the "playback."
Testers and all stakeholders attend. After the playback is completed, a snapshot is created, and
code is deployed to test servers so that previously defined tests can run in isolation.
This approach can continue for multiple iterations. When the entry criteria for system integration
tests and user acceptance tests are met, those test phases are performed. Figure 2 also illustrates
the fact that it is possible to start system integration testing before the end of development.
When integrating with external systems, developers should complete integration tests as early as
possible. Delayed IBM BPM projects are most often caused by integration issues.

IBM Business Process Manager testing methodology, Part 1:

General testing guidelines

Page 7 of 13

developerWorks

ibm.com/developerWorks/

IBM BPM development roles

Use the agile testing approach during IBM BPM project development so that you can ensure that
quality solutions are delivered for user acceptance validation. In the agile testing approach, subject
matter experts (SMEs), developers, and testers work together as one team in the following ways:

Test cases are authored and reviewed as part of iterations.

Change acceptance and story acceptance are managed within iterations.
Tests are run at the end of each iteration or in the following iteration.
Defects and changes are addressed as part of backlog.
Regression testing is developed as a subset of each iteration.

In a traditional testing approach, the test and development teams work separately, which increases
the risk of project delays in the following ways:
Test cases are written by testers without review during development.
Testing is done at the end of projects.
Testing is misaligned with project changes.
For an agile approach, SMEs, developers, and testers are all involved in IBM BPM project
development. Each role has its own responsibility and interacts with others. Figure 3 shows the
responsibilities of each role.

Figure 3. IBM BPM development roles

The cost to fix a defect depends on the length of the feedback cycle. As shown in Figure 4, the
earlier a defect is found, the lower the cost to fix it. It's important for different roles to interact
closely to discover potential defects earlier in the process.
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 8 of 13

ibm.com/developerWorks/

developerWorks

Figure 4. Cost and length of feedback

An example user story lifecycle in an iteration

Figure 5 shows how roles interact with each other in an example iteration.

Figure 5. Roles in an iteration

In the first interaction, the SME assigns a user story. Everyone reviews the story. Developers begin
to create artifacts, and testers create test cases, based on the user story.
In the second interaction, the SME refines the story details and refers to artifacts and test cases.
Developers and testers update the artifacts and test cases based on the SME's changes.
IBM Business Process Manager testing methodology, Part 1:
General testing guidelines

Page 9 of 13

developerWorks

ibm.com/developerWorks/

In the third interaction, the SME conducts an internal review of the story with a demo and test team
approval. Developers and testers finalize artifacts and test cases.
During the wrap-up period, the SME schedules deployment, manages change, and accepts the
story. Developers support testers to execute test cases.

Conclusion
This tutorial described the overall IBM BPM project test methodology. Read Part 2 of this series to
learn about detailed testing that should be included in IBM BPM project testing.

Acknowledgements
The authors would like to thank Dawn Akuhanna, David Booz, Todd R Deen, Matthew S
Luczkowiak, Pascal Wagner, Dan Eykholt, Weiming Gu, Chris Richardson, and Ekkehard Voesch
for their review and contributions to this tutorial.

IBM Business Process Manager testing methodology, Part 1:

General testing guidelines

Page 10 of 13

ibm.com/developerWorks/

developerWorks

Resources

IBM Fix Central

Technote about collecting troubleshooting data for IBM Business Process Manager products
Technote about the new IBM Support Assistant data collector in IBM BPM V8.5
IBM Support Portal for IBM Business Process Manager Advanced
Good practice - Use rolling upgrade when you update IBM BPM
Fiddler website
Firebug website
Send data to IBM Services when no PMR is open: use the Enhanced Customer Data
Repository (ECuRep)
IBM Software Support Handbook
developerWorks Business process management zone
IBM Business Process Manager documentation on IBM Knowledge Center
Application Integration Middleware Support Blog in the developerWorks community
IBM Business Process Manager good practices wiki in the developerWorks community

IBM Business Process Manager testing methodology, Part 1:

General testing guidelines

Page 11 of 13

developerWorks

ibm.com/developerWorks/

About the authors

Allen Chan
Allen Chan is a Senior Technical Staff Member of IBM Smarter Process, and is
the currently the Lead Architect for IBM Blueworks Live and IBM BPM installation,
Configuration and Migration. Previously, he was the IBM BPM SWAT Technical
Lead where he led a team of experts to help ensure customer success in key IBM
BPM deployment and rollout scenarios. He is passionate about ensuring customers'
successful use of IBM products.

Jing Jing Wei

Jing Jing Wei leads the IBM BPM end-to-end SVT team, working on customer
scenario in-house testing. She acts as an IBM BPM SVT technical leader in the
IBM BPM lifecycle, security, integration and web services areas. Previously, she
worked with the IBM BPM and Monitor SWAT team, supporting customers and
business partners across the world with IBM BPM architect reviews, IBM BPM project
development, and problem determination.

Jerome Boyer
Jerome Boyer is an IBM expert on Enterprise Business Rule Management Systems
in IBM BPM, SOA and Complex Event Processing deployments. As a Senior
Technical Staff Member, Jerome is the lead IBM ODM and IBM BPM solution architect
in IBM Software Services for WebSphere (ISSW). Jerome is the author of "Agile
Business Rule Development", published by Springer in 2011.

Peng Guo
Peng Guo worked in WebSphere Adapters team for 3 years, and then moved to the
IBM BPM SVT team, focusing on lifecycle testing. He was responsible for automation
implementation of the SVT scenario for several releases. He is currently working on
IBM BPM end-to-end testing and customer scenario in-house testing.

IBM Business Process Manager testing methodology, Part 1:

General testing guidelines

Page 12 of 13

ibm.com/developerWorks/

developerWorks

Hong Yan Wang

Hong Yan Wang is a system verification tester for IBM Business Process Manager at
the IBM China Software Development Lab. She has more than 10 years experience
in software development and testing. She mainly focuses on end-to-end testing and
HADR testing.

Hong Li Liu
Hong Li Liu works in the IBM BPM QA & Automation team in IBM WebSphere. In
her current role as a software engineer, she tests the IBM Business Process Manager
product. She writes automation code to make her testing more efficient, and she has
solid Java, shell, and testing experience.
Copyright IBM Corporation 2014
(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

IBM Business Process Manager testing methodology, Part 1:

General testing guidelines

Page 13 of 13