Professional Documents
Culture Documents
Whitepaper
June 2011, by Dr. Harald Goebel Andrew Kashulin Heinz-Jrgen Scherer Dr. Boris Zinchenko
Table of Contents Abstract Motivation Blueprinting an ERP Solution Requirements for Testing Approach for Model-Based Testing Test Case Generation Algorithm
Execution Strategies for Test Cases Adaptive Test Case Generation Important Considerations
Interfaces and IT Systems Work Products and Deliverables Organizational and Resource Assignments
3 3 4 5 6 7
10 11 11
11 12 12
12
12 12
13
13
14 14
15
15 16
18
Summary Bibliography
18 20
Abstract
For the requirements definition of Enterprise Resource Planning (ERP) software, the management discipline and methodology of Business Process Management (BPM) has an important role in the Blueprint project phase of ERP implementation. Using BPM methodology, the business requirements are documented and analyzed. Process improvement is achieved via better support of business activities by ERP software. The Blueprint consists of functional, organizational and system process models developed by a Business Process Analysis (BPA) tool prior to the realization phase of customizing the ERP software. Because of the complexity of ERP software, the integration testing of ERP supported processes is a very essential task to lower business impacts of badly tested software. At the same time, testing is a resourceconsuming task in critical project phases. This whitepaper discusses an adaptive software solution for automatically transforming Blueprint business process models into test cases for a methodical and toolbased integration testing.
Motivation
ERP software is standard software from leading software vendors like SAP or Oracle. It comprises a comprehensive support of business processes, and is generically developed for the enterprise market. As implied by the definition of standard software, it is maintained by updates of the vendor and can be updated. It is in the nature of things that ERP standard software tends to be a complex construct consisting of many software components and configurations. ERP software needs to be customized to implement requirements and support business processes of an organization. Customization is a standard process in an ERP implementation, and is done by parameterization and change of the software components like the programs and configuration originally supplied by the software vendors. Beside functional acceptance, the software quality is influenced by errors in originally delivered software and customization. To achieve a qualified support of business processes, extensive software testing is mandatory not only on the software development level, e.g. using unit tests. The maturity of customized software is achieved by a continuous testing approach with functional use-case testing and integration testing. This is equally true for traditional transaction-based ERP systems and Business Process Management Systems (BPMS) with execution and integration of processes. Nowadays, SAP ERP is still a predominant transactional ERP system where SAP transactions like forms support business activities. Implementation of an ERP system is done through several project phases. SAP uses the Accelerated SAP (ASAP) project methodology where Blueprinting is the design phase of the ERP solution. In the realization phase, the ERP solution is customized based on the requirements specified in the Blueprint. For the methodical approach of testing, the definition of test plans based on test cases is a challenging task. To attain it, different members of the project team with various skills such as business key users, business analysts, quality assurance (QA) and software implementation team members must work together. It is even more complex because team members talk with different business or technical languages and are distributed across different locations including the employment of System Integrator
offshore development teams. Primarily, we can consider three challenging aspects in ensuring the quality of business processes ERP support: Costs The task of manually creating test cases is accomplished with high project efforts for the design of many end-to-end business scenarios. The traditional design of test cases takes qualified and costly resources out of the project core and testing teams. The business impact of insufficient integration testing reduces production efficiency, and increase software maintenance costs. Risks Even if skilled project team members design the test cases, there is a high risk of not capturing all relevant end-to-end scenarios to be executed as test cases, or having mostly incomplete test cases with business impacts in the production environment. In productive operation, a process may fail because the executed scenario was not tested or interfaces between different components were not identified for appropriate interface integration tests. If a business process has changed in the context of risk management like Sarbanes-Oxley Act (SOX), it is critical to miss the re-testing of process risk controls. Quality Bad test coverage leads to low software quality with high business impact. This negative impact is increased by an incomplete matching of the test data and insufficient reviews of work products and deliverables. All in all, the business impact of insufficiently tested end-to-end scenarios results in low quality business process support and higher software maintenance costs.
representation which can show the process logic and flow with different levels of abstraction. On a functional level, business processes are described by a model with a sequence of activities depicting the business logic.
Process models map business logic that describes end-to-end scenarios as sequences of business activities. The motivation for developing the solution described below is to use available Blueprint process models to automatically generate test cases that can be used by test management tools to lower project costs and reduce the risk of software quality impacts on IT supported business processes.
and improving automation by setting correct priorities for frequently-operated endto-end scenarios. It is important to identify gaps between business processes and related test cases to achieve completeness of test case coverage. Because of business and IT complexity, the emergence of Change Requests (CR) is not unusual in the project life cycle. Using BPM methodology and a BPA tool for Blueprinting an ERP solution reduces the absolute number and complexity of CRs because of a comprehensive understanding of the business logic and requirements. Each CR influences the scope of the software solution, and has an impact on the customizing. A change in a business process is managed in an ERP project by Change Management. This should include the impact of the changes with respect to related test cases. In order to achieve lower testing efforts, redundant and obsolete test cases have to be removed; and to ensure quality, existing test cases must be updated to reflect the change of the business process. Under software maintenance, test cases should be periodically reviewed to detect the impact of change requests.
A test case is an end-to-end business scenario as a sequence of activities to be executed sequentially. The process logic is explicitly formulated by using logical rules commonly expressed with exclusive execution (XOR), parallel execution (AND) and inclusive execution (OR) of a branch in the process flow. The OR rule is complex to understand. It can be comprehended as a combination of a XOR and an AND rule [OS08]. Because process activities can consists of human or system atomic tasks (atomic means here that a task cannot be divided further into subtasks, like a SAP transaction), a sub process is defined by several tasks or a process interface. We use statements as designation of activities in a test case. The statements are linked by transitions1 to the logical flow of an end-to-end scenario.
Elementary Sequence The sequence is the fundamental idea of the chain of discrete individual statements following each other in a certain order. It is the common semantic interpretation or implication causality.
Split and Join Patterns Split gateways have exactly one incoming and more than one outgoing connection, whereas join gateways have more than one incoming and exactly one outgoing edge. Although implicit splits and joins are not a good practice at all, they can be replaced by explicit gateways.
Linearization 1: A, B, C 2: A, C, B2
1: A, B 2: A, C
1: A,B 2: A, C 3: A, B, C 4: A,C, B2
1: A, B 2: A, C
Only needed for C3 condition coverage, not needed for C2 path, C1 branch or C0 statement coverage.
Linearization 1: A, B, C 2: B, A, C2
1: A, C 2: B, C
Synchronized OR join
1: A, C 2: B, C 3: A, B, C 4: B, A, C2
Implicit and explicit Iteration The iteration is the fundamental idea of the similar repetitive sequence. Iterations use only XOR connectors to create such a pattern.
Figure 4 Implicit iteration with F1, F2 and explicit iteration with F3, F4
Table 3 shows the results of the linearization of the iteration pattern of Figure 5. The iteration is expanded into all possible sequences of statements.
Table 1 Linearization of the iteration
10
Runs (N) 0 1 N = 2,
Linearization event (F1) so-called Top tested loop (F1, F2) (F1, F2, F1, F2, )
The linearization of the iteration is simple, but in practice it requires handling additional reasonable assumptions to reduce the number of test runs. An additional parameter in the generation algorithm of test cases describes how many loops have to be executed for an iteration pattern. Per default 2 loops are generated per iteration. Conclusion Based on a breadth first search (BFS) algorithm handling graph theory [MAI11] the logic of linearization of statements finds all potential end-to-end scenarios with the linear sequence of statements describing a test case. For complex business processes with deeply nested branches, the number of some thousands of potential possible end-to-end scenarios exceeds the practically approach of executing the generated test cases. The generation of test cases for practical use is improved by an adaptive algorithm for test case coverage.
Description
All statements and gateways as nodes of the control graph will be covered by the testing not possible in a logical order.
Pro
Less effort, all statements of the process flow are executed.
Con
Not all transitions (edges of the control graph) between statements will be passed through while testing the process model. All statements are weighted equally Empty branches are not covered Dependencies of decisions are not covered Loops are not tested sufficiently
C1
Branches
All edges of the control graph with all nodes and edges are tested at least 1 time. This includes the C0
11
Case Coverage of
Description
coverage and all functional needed test cases, but not all test cases for integration testing. Each rule with possible decision is included in a test case. High efforts by given number of possible test cases. All variations of rule decisions are covered in test cases. Very high effort for testing by potential number of test cases.
Pro
Con
Complex branches are weakly tested
C2
Paths
High effort for testing Based on conditions not all paths are covered
C3
Conditions
Important Considerations
Although the process is in the center of test case generation, it is important to understand the context of a statement in the end-to-end scenario. The context is described by the preceding and subsequent statements linked by transitions, the statement is supported by IT systems, the data of work products is linked with the help of interfaces between different technical components. Resources support the execution of the business process. Interfaces and IT Systems The activities (e.g. SAP process steps and transactions) are supported by IT systems. By the knowledge of this support it is clear if a transition links to statements executed in different IT systems. The statement associated work products have to be serialized and transformed using an interface between the two different IT systems. It is very important to review this interface link in the execution of test cases.
12
Work Products and Deliverables To properly describe test data and functional use case success criteria it is important to know which work products are processed in the end-to-end business scenario. Organizational and Resource Assignments Finally, the organization and resources supporting the execution of statements being responsible, accountable, controlled and informed (RACI) have extensive business knowledge which in many cases is not described by Blueprint documents. They know e.g. how to handle exceptions in workflows, or know about probabilities of executing branches in complex workflows. The algorithm uses this information for cost optimization of test case analysis.
13
variations for this process model. For this reason the algorithm uses an Exception parameter to avoid the time costly generation of too many C2 or C3 test cases.
14
structured IT view containing business scenarios, processes and process steps with linked SAP transactions. The SSM includes a test management that is integrated with the Blueprint. Modeling Method It is a characteristic of the BPM market that BPA tools use many different modeling methods (commonly nicknamed as a modeling language). The latter use a set of specific rules to describe a business process control flow and its relationships between processes, activities, resources, work products and the organization. The methods use various notations with symbols to visually express the organizational or process models by diagrams. In general, methods can be distinguished into legacy or vendor specific methods, and standard methods like BPMN developed and controlled by the Object Management Group (OMG). Today, what is mainly used are variants of flowcharting methods, Event-Process-Chain (EPC) variants and variations of BPMN-based methods. Even if a vendor claims that the promoted BPA tool uses a standard method, in most cases the tool implements a specific method version, or a method subset or extension as compared to the standard method. For model data exchange, BPA tools use mostly vendor specific XML formats. To interchange model data between tools of different vendors one can use standard formats like XMI (OMG) or XPDL (Workflow Management Coalition - WfMC). However because of variations in three dimensions 1. Implementation of modeling standard 2. Implementation of exchange format standard 3. BPM vendor commercial interests in many cases unexpected results occur when importing process models from a different BPA tool into the tool used. The US Department of Defense (DoD) made a review using standards BPMN and XPDL for model data interchange [MZM09 page 10]. No tool was capable of importing error-free models from another tool. Sometimes the tools failed to import its own (!) created export format. Model and Diagram Data Interchange To overcome the known problem of interoperability between tools and modeling standards, the BPMXchange (BPM-X) middleware for the interchange of model, master and meta data is available. This software solution establishes interoperability in the enterprise management systems landscape of an organization. This approach eliminates existing silos of enterprise meta-data information and avoids efforts of manually re-keying or re-mapping existing model data or visual diagrams. The automatic conversion model and diagram improves the quality of exchanged data and reduces needed human interactivity. The BPM-X middleware is an Enterprise Integration Application (EAI) framework for the execution of transformation sequences. The framework uses inbound and outbound adapters to read and write tool or standard-specific data formats. For standard formats like XPDL, BPM-X offers support
15
for different versions and so called flavors to support vendor specific implementations of exchange standards. To eliminate vendor specific (standard) methods, BPM-X uses pattern-based rules for transformations of model and diagram data described by tool specific methods. The automated process using BPM-X mainly consists of five steps, namely to 1. Export data from the supplier tool, database or repository 2. Import data into a BPM-X repository neutral model representation using an inbound adapter 3. Optionally improve model consistency 4. Execute a sequence of model transformations from supplier into consumer methods 5. Export into consumer tool format using an outbound adapter and import into the consumer tool In the case of model-based testing, the BPA tool is the supplier tool exporting the business process models. The consumer tools are testing tools like HP Quality Center (QC), SAP Solution Manager (SSM), or Microsoft Word in case of manual testing.
16
rules describe possible patterns that cannot be expressed through the meta model. Commonly the meta model plus notation is called a modeling language. From the point of view of generating test cases, this is an external method. The model data including the visual diagrams is loaded into the BPM-X middleware repository still using the external method.
Internally, BPM-X organizes its repository with model instances allowing several logical representations of the same model within one physical repository. To have a common initial point for the generation of test cases, BPM-X initially transforms the models stored in one model instance (1) into a new model instance (2). For test case generation, the internal method used is BPMN. This operation provides a normalization of a specific model method into the BPMN method. During this transformation of the business process, included work product objects are associated with activities by associations. Resources such as organizational units, groups, roles or IT applications are converted to so-called performers. Process activities, transactions or interfaces are converted to tasks, sub-processes or call activities. Test Case Generation Based on the BPMN Method BPM-X middleware offers a flexible framework to execute transformations in a sequence of so called atomic operations (AO). With an application programming interface (API), developers can develop own AOs. MBT is developed as a BPM-X AO to implement the test case generation.
17
Based on the model data stored in the instance (2) the software generates test case models and stores them in a new instance (3). The test case models are also described by the BPMN method. The algorithm described can be easily modified or replaced to deliver alternative generated test cases. The following figure shows the first four test cases generated on the sample SAP business process model. Based on this specific example including 4 XOR gateways with 2 branches each, a total of 2 4 equal 16 possible test cases can be generated.
18
These generated test case models can be converted into the model representation described by a consumer method and data format. E.g. export is possible into XPDL format for test case processing in HP QC, into SSM Blueprint projects or into Visio for manual test case documentation.
Table 3 Examples for Testing Tools and external formats
Tool HP QC SAP Solution Manager T-Code SOLAR02 SAP Solution Manager T-Code SOLAR01 Microsoft Office
Blueprint Project
Description Load test cases into HP QC based on XPDL Use tab test-case with type manual test case using Word documents can be used to assign the test case documents to a blueprint structure Implementation project with Scenarios and uploaded Processes for each test case model and Process-Steps for activities. Documentation, analyzing and manual execution of test cases
Summary
This whitepaper addresses the integrated solution of test management using business process models from the ERP Blueprint as a basis of automatically generated test cases for further use in a testing tool or manual testing. Using the BPM-X integration middleware for software interoperability builds a very flexible software solution for the interoperability of any BPA tools to be linked to a testing tool. The
19
BPM-X framework allows the development specific operations like the discussed test case generation. This new operation can be linked into the chain of transformations starting from the import of business process models, through transformations of model data by different methods into BPMN, to the export of test cases in a specific data format used by the testing tool. This concept of testing is much more business-focused instead of isolated functional testing not fully covering business logic and technical interfaces that supports data flow between business activities. The approach ensures the completeness of test case coverage to improve the quality of ERP supported business processes and lower the impact of weakly tested end-to-end scenarios. Costs High efforts are reduced by automatically generated test cases. By re-using business know-how from business process models defining probabilities of process flows, the test cases can be prioritized only retesting an appropriate number of test cases when software changes and not the full set of test cases for a business process that is very time consuming in execution of the test cases. Integration of test case tools like HP QC or SSM Blueprint test management or test projects speeds up the handling of test cases in complex ERP projects, as compared to manual analysis and description of test cases especially when the test cases change through change requests. By examining end-to-end business scenarios with high probabilities, the automation of tests with tools like HP Loadrunner or SAP eCATT scripts save big amounts of time by re-running theses test during change requests or ERP maintenance. The overall efforts are greatly reduced for costly resources of the QA team and business key-users. Risks Many business executives have concerns about the quality and resulting costs for implementation and maintenance of ERP software. By using the model based approach for integration testing, the business impact of insufficiently tested ERP software is eliminated. By the re-use of business logic given by process models and execution probabilities combined with systematic execution of prioritized end-toend scenarios, the relevant states of a business processes are reviewed. This includes identification and explicit testing of interfaces for exchange of data between activities supported by different involved ERP and 3rd party software components. A more methodical and tool-supported approach by using tools like QC or SSM lowers the risk of influence of inexperienced QA team members and insufficient project time available for testing. For revisions of business software, automatically generated test documentation can be used to audit and review test cases. Quality
20
Using business process models as a basis of their generation, the test cases re-use documented process knowledge. The use of BI technology to analyze test case data and the visual documentation of test case diagrams improves the analytical understanding of end-to-end business scenarios. Based on test cases it is much easier to define appropriate test data for integration and for functional use case testing. The integrated approach addresses the impact given by business requirement changes to test cases in the ERP project cycle or during the maintenance of ERP. The changed test cases can be re-tested again to eliminate side effects of software changes. At the bottom line, the described methodology of the model-based generation of test cases complements the concept of using BPM for ERP Blueprinting. This establishes an optimal alignment between the business and IT.
Bibliography
[OS08] [WP.1] [WP.2] Zur automatischen Ermittlung von Testszenarien aus EPK-Schemata, Oliver Skroch, 2008 http://en.wikipedia.org/wiki/Quartile http://de.wikipedia.org/wiki/Statement_Coverage
[MZM09] Enterprise Architecture based on Design Primitives and Patterns Guidelines for the Design of Business Process Models (DoDAF OV-6c) using BPMN, Michael zur Muehlen, 2009 [MAI11] Graph Theory and Algorithms, M. Ashraf Iqbal, 2011