Performance Considerations for the MDM Adapter

Reference Guide based on NetWeaver 7.31

Harrison Holland, Partner at Holland Systems @ http://hollandsystems.com 07.20.2013

Introduction
With the release Netweaver 7.1, SAP has released MDM specific adapters for PI which serves as an additional option for customers, whereas previously the file adapter was the primary option for interface development. The MDM PI Adapter introduces some functionality that is more appealing than the traditional file adapter, such as a more synchronous interface and end-to-end monitoring in the message monitor of PI. However, the architecture of this adapter is not very transparent which can lead to confusion when problems arise. This document serves to raise awareness to common performance issues and how to design an integration architecture that avoids these problems, in addition to demonstrating some configuration parameters which may help reduce performance problems.

IDoc Processing
Lets examine a common scenario where customer master data is being sent via IDoc to MDM using PI as the middleware. The ECC configuration is outside of the scope of this document; however there are various ways that the customer data could be generated: (1) MDM Generic Extractor, (2) ALE Output Control / Change Pointers, (3) Transaction Code BD12, among others.

Depicted in the picture above would be a scenario that generates a single IDoc for each customer record that is sent from ECC. For an ongoing interface with very low volumes this is no big deal. However as we will find out later in this document, when volumes are high it is more efficient from a performance perspective to batch your records into bundled IDocs. For each IDoc that is generated, a dialogue process is occupied on the server. There are a limited number of dialogue processes available to the server at any given time. If all the dialogue processes are occupied by IDocs then users will not be able to execute business transactions, or jobs may not be able to run. Performance Points 1. Bundle IDocs to contain multiple records. In a mass send scenario (ie: thousands or more) try grouping 1500-3000 master data records into IDocs. The number will depend on the complexity of your data structure, and how many segments are being populated. Schedule background jobs to process your IDocs (tip: you can create variants of the background jobs specifically for particular message types!)

2.

3.

In a cluster environment, consider dedicating an app server to IDoc processing in order to reduce the burden on the central instance

You may be wondering what these tips have to do with the MDM PI Adapter; however they will play an important role later as we will find out soon!

PI Message Flow Messages coming into PI are placed into queues. When a work thread becomes available the message is moved from the queue into the threads where it begins processing. Once the mapping is completed the message is passed to the adapter engine where it gets sent to the final destination. Continuing the example from before, here is an example of customer data being sent from ECC to MDM:

The first performance issue can arise when there are more messages to be processed than there are threads available. If the receiving system cannot process the messages as fast as PI performs the mapping, a bottleneck will arise and messages begin piling up in the queues. Tip: The number of threads available for a particular adapter type (ie: MDM Adapter) can be configured in the Netweaver Administrator. Try XPI Service: AF Core, with the property messaging.connectionDefinition.

The message size has a significant impact on the overall throughput in SAP PI. The picture below has been used in several documents to demonstrate the impact of message size of performance.

Our tests showed an improvement in raw PI processing from 3+ hours to 45 seconds for 9k customer records. Keep in mind the file size in question is the converted XML file that PI generates internally, not the source file. Therefore a flat file of 1MB can generate an internal XML file of 4+MB. Performance Points 1. Consider configuring queues specific to a particular interface to make sure that interfaces with large volumes that are not time sensitive do not block the queues that are needed for other messages that need to operate in near real-time. Consider adjusting your thread parameters to allow more threads for the MDM Adapter. Make sure to adjust your source files (if possible) to make sure that they meet the optimal size. Remember the discussion points in the IDoc processing section of this document.

2. 3.

MDM Adapter Lets begin by analyzing the differences between the MDM Adapter and the file adapter. Many customer s who have a mature SAP MDM environment may be very used to the file adapter, and therefore are not prepared for the design differences associated with moving to the MDM Adapter.

The most notable difference from a performance perspective is the fact that the MDM Adapter uses the Java API to establish a connection to the repository to run certain internal operations. This Java call allows the PI system to communicate synchronously to the repository but can also be the source of performance issues.

All threads on SAP PI are dependent upon the connections and operations available on the MDM repository. Exclusive lock operations such as executing a matching strategy can cause a major delay on the thread processing in PI.

Performance Points 1. 2. Take into account any operation that will require and exclusive lock on the repository Consider all activities which require a Java connection to be established, such as web service calls

MDM Server Even while using the MDM PI Adapter, the Import Server is used to process the inbound messages. Consider the following tips with regards to tuning the import server: The Chunk Size is a parameter that can be set in the mdis.ini file which determines the number of records processed at a time during an import. This setting has a large impact on the amount of RAM used, therefore large chunk sizes should be used in scenarios where RAM is not a concern. If memory consumption is high the parameter can be reduced to avoid running out of memory Setting the value low will reduce the time required to process each individual chunk, which can be used in cooperation with the Inter-Chunk Delay MS setting in order to break large imports up such that the repository is not locked for a considerable amount of time Setting the value higher will reduce the overall time required to complete the import job For 4GB of RAM, the recommended configuration is are Chunk Size = 50,000 and Number of Chunks Processed in Parallel=5. For servers with 8GB 64GB of RAM use Chunk Size = 50,000 and Number of Chunks Processed in Parallel should be set between 6 and 10 accordingly Unique fields should be defined on every MDM table possible. Any database table that does not have a unique key will take much longer to find a matching record. Use these unique keys as the matching key for an import map MDM spends considerable extra memory and processing power to maintain the sort index, which can dramatically slow down updates and imports on large repositories. Limit the number of fields that have a sort index Keyword searches are the most costly in terms of memory consumption. The import performance is significantly affected, so it is recommended to define text fields with keywords only when really necessary. Whenever possible, create separate maps for individual MDM tables. It is recommended to split a complex import map into individual maps. One map to load the main table and additional maps for each qualified lookup table When possible, load all reference data before loading the main table. Create separate source files for each MDM table when possible Complex XML files should be simplified to enable fast parsing by the MDIS. Multiple small XML files should be integrated into one larger XML file to avoid the parsing overhead of each file When possible, match according to only one primary key field (especially in maps built for the main table). For lookup tables, match only by display fields. For qualified lookup tables match by non-qualifiers

About the Author Harrison has over 10 years of experience working with SAP, focusing on master data strategies and systems integration. He is one of the founders of Holland Systems, an Atlanta-based boutique consulting firm that is revolutionizing the SAP consulting space. Harrison enjoys listening to loud music, playing video games, watching boxing, and motorsports. He competes in a local gokart league despite getting schooled by teenagers.