Recommendations for Performance Benchmarking

Shikhar Puri
Infosys Technologies Ltd 44 Electronics City, Hosur Road Bangalore 560100, India +91 9342842635 Shikhar_Puri@infosys.com

ABSTRACT
Performance Benchmarking for the application is becoming very essential before Production deployment. This paper covers few recommendations and best practices for Performance Benchmarking exercise.

Assess usability of Performance Testing Tool Performance expectations for transactions, system behavior

1. Introduction
The necessity of proof of performance for the application along with functional proof is becoming very popular in the industry. The performance testing and benchmarking for the application can be performed before deploying the product in the production environment or after the production deployment. If the business is expected to grow in future, the client is required to ascertain if the application can support the extended business volume on the existing infrastructure and what could be the potential areas of concern or bottleneck in terms of Performance. The areas need to be identified, where enhancement, tuning and upgrade is required. The performance testing and benchmarking also provides a proof of the performance for the product and sets baseline and benchmark for further enhancement in the application.

Performance metrics including server and network utilization Test approach Test validation strategy Collection of Workload Profile Peak and Off-peak time periods Total and concurrent number of users Key Transactions, frequency and complexity Transaction mix with respective user profile Usage pattern and think time between transactions Data Access intensity and usage of data

1.2

Test Suite Development

This phase would involve the following: Set up test environment Creation and validation of test scripts Creation of data population scripts Creation of transaction data pools to be used during test Setup for performance monitoring

2. Objective for Performance Benchmarking

The objective of Performance benchmarking is to eliminate performance bottlenecks in the Application and Infrastructure components by identifying and tuning them and to verify that the performance delivered is as expected from the system. This might involve conducting performance tests iteratively on the application against representative production workload and anticipatory data volumes as expected in production after a significant period of time.

1.3

Execution of Performance Tests

3. Performance Benchmarking Methodology

The Performance Testing life cycle goes in parallel to the Software Development Life Cycle and starts with the requirement gathering for the software application. Performance Testing Methodology will consist of following phases.

This phase will consist of number of benchmarking runs, each run with increased number of concurrent virtual users. This will continue till the performance testing objectives are met or breakpoint of the system is reached. If the system breaks down because the test environment infrastructure, the test results need to be extrapolated. Typical benchmarking cycle will consist of: Start performance test and monitoring scripts Collect Test and monitoring results Validate results

1.1

Planning Phase

The objective of planning phase is to evolve the test plan and strategy which would involve following activities and details: Study and Review of Application Architecture Understand the system functionality and Flow Review of the Hardware Components

1.4

Analysis and Review

Data collected from performance tests will be consolidated and analyzed. This will include: Analyze monitoring data

 Breakup of performance across servers Analysis of potential bottlenecks and limits to scalability Identify system and infrastructure related issues Sensitivity of performance to increase in load If the issues can be resolved, the execution cycle would be repeated after the issues have been resolved.

4. Recommendations
It is near to impossible to replicate the production profile in performance test environment. The profile can only be simulated in performance test environment. The effectiveness and correctness of the performance benchmarking exercise depends on the closeness of workload simulated in test environment to that in production environment. This section would cover some of the best practices which can be taken care while performing the performance benchmarking exercise. The general checklist points for carrying out the benchmarking exercises have not been included.

initial user ramp up phase and remain in the system till the test is over even if their assigned iterations have been completed. A large value think time can be included towards the end of the virtual user scripts. This helps in maintaining the ratio of active to inactive sessions in the system. This also depends on test requirement so scripts should be scenario specific. Data Pool selection: Data pool set for the transactions should be carefully selected. The transaction data usage for data pool should be in proportion to the real time usage of the data. If the real time system is supposed to perform an enquiry transaction, the distribution of data in terms of different query parameters, data creation dates, range of data selected (example wild card enquiries or date range), data access intensity (rows to be returned for query), usage of list of values, repetition intensity of the same data or similar data in further iteration of transactions should also be considered. These parameters produce the workload profile and test results near to real time scenario.

1.5

Transaction scripting

1.6

System configuration and setup

Performance benchmark test suite is developed by scripting the selected critical transactions. These scripts when executed generate a virtual user in the system which performs the action or transaction recorded in the script. Various data sets can be provided to the scripts to use during the test execution so that the scripts use different data sets in various iterations of the transaction. To simulate the real time production scenario, sleep time are also introduced in the scripts so that the transaction is executed similar to real time. These sleep times corresponds to think time, data entry time and time between the iterations. Few points which can be considered during the transaction scripting for virtual users are: Think Time generation: Most of the tools provide the facility to generate think times (data entry time and time between the iterations) randomly, but this method generally makes the test profile very much random and results can not be truly reproduced or repeated for verification. To avoid this scenario, think times can be scientifically generated. Either these think times can be calculated in advance to distribute the transaction profiles evenly for whole of the test duration or these times can be randomly generated in the first test and can be recorded. The recorded think times can be provided in the future tests to verify the repeatability of the test results. Transaction Distribution: The test results are considered for analysis and evaluation only for steady state of the system. The results for ramp up (initial) phase and ramp down (logout) phases are ignored. If the think times generation process is random, a major chunk of users might come into the system and start their transaction iterations at the same time. This will generate a resource usage spike at that moment of time and the results could be skewed. Also, the transactions having very low frequency of occurrence happen to execute in ramp up or down phase, these would be missed in system throughput considered for analysis and evaluation. Control over the scientifically generated think times help in devising the login behaviour and transaction start phase for the virtual users. The users start the iterations gradually, achieve a system steady state and come out of the system once throughput of all the critical transactions has been met. This portrays near to real time scenario. User Session Management: Scripts should be created in such a manner that all the virtual users are injected gradually during

The configuration and setup of the test environment should be similar to that in production environment. Few points which can be considered at this front are: Load balancing Scenario: This is true that the load balancing in the test environment should be similar to production environment but at the same time, if the load balancing policy is such that the users are distributed on round robin basis on multiple load balanced servers, this might create some issues in test environment. Various users in the real time execute multiple transactions in random fashion, the transaction usage pattern and behaviour does not remain constant over the time period. In the simulation exercise, transaction or group of transactions is recorded in a virtual script and that script is executed with multiple users. A virtual user repeats the same activity in multiple iterations with different data sets. If the virtual users or scripts with heavy transactions connect to any of the server, these users or sessions will keep on sending the requests on the same server. Though the number of users would be balanced on multiple load balanced servers but the load will not be balanced between the multiple requests. Distribution of Loaded Data: The test environment should be pre-populated to a certain level which is comparable to production environment data for around 6 months or one year. The time duration, for which data needs to be loaded, depends on the data archive and purging requirements for the system and the expected system longevity. This pre-populated data should have the distribution similar to that in production environment. Various business parameters and data distribution parameters need to be collected in advance or production profiling should be done to generate the data load scripts. If the production data is available, ideal scenario would be to replicate or clone the production data in the test environment. This helps in generating the response time and elapsed time of the transactions similar to production environment. Placement of data on discs: If the data population scripts are used for loading the data in test environment and the data is loaded sequentially in the tables, the data related to a specific table is populated on same discs. While performing the transactions on these tables or related tables, either the data is accessed quite quickly (due to operating system caching mechanism) or the disc become bottleneck (due to excessive I/Os on specific discs). This might not be the case for real time

scenario as multiple transactions are performed during the day and the data for a specific transaction is created gradually on daily basis. The data is distributed automatically on various discs and the probability of disc being bottleneck is reduced. To avoid this issue, the placement of data on discs should be made as close to production as possible. The ideal scenario would be to clone the production data in test environment keeping the disc structure similar to that in the production environment.

assigned and distributed accordingly avoiding congestion on client machine. It should be tried not to consume more than 7080% of the available memory on load injector.

1.8

Execution and Analysis

Few points which can be considered for test execution and analysis of test results: Validation of Transaction Response: Most of time the data used during the test is different from the data used during the recording or creation of the transaction script, so the response at various steps in the transaction script can be different from what is expected while recording or creation. To overcome this issue, the virtual user scripts should be devised in such a way that the response from the request is validated for success and further progress. The next steps in the transaction should be executed depending on the response received from servers. Profiling of Transaction steps: The transactions can be further divided into multiple logical steps and these steps should also be monitored for response or elapsed time. If the transaction response or elapsed time is above the expected response time, these logical steps provide a better picture to understand the bottleneck in the transactions. The problematic steps can be further looked into for removal of performance issues. Validating execution of Transactions: After the benchmark test, the simulation tool provides the summary of the test results including transaction response time, transaction throughput etc. The throughput mentioned in the test result summary is based on the execution of the steps mentioned in virtual script. There could be the scenario, when the transaction failed but the virtual user step was executed, this would be shown in the test result summary. The better way of verifying the transaction throughput is to verify the occurrence of the transaction from the database or the affected areas. This would be easier if the transaction is performing create, update or delete operations. For the read type of operations, the opening of screen or any logical thing can be logged somewhere so that the transaction occurrence or success can be verified. Transaction Response time: Generally the transaction response times are presented as 90th percentile transaction response time. This means that 90% of the total transaction response time occurrences are below the figure marked as 90th percentile figure for the candidate transaction. This is better than presenting the average response time figure, which averages out the peaks and variance in response times. Sometimes even 90th percentile figure can not portray the correct picture and further analysis should be done before presenting the result summary. Say the transaction is having very low number of iterations (e.g. 2 per test duration) and one of the iteration happened during the initial stage or when the system was not performing well, the 90th percentile response time would be the response time for the mentioned iteration and would be very high. In this scenario, we can ignore the mentioned iteration with proper justification and consider the response time of the 2nd iteration. Similarly if the frequency of transaction is high and 90th percentile response time is quite above the expected response time, the difference between 85th percentile and 90th percentile response time should also be checked. If the difference is not much, the transaction has some performance issues but if the difference is very high the iteration occurrences should be checked and the reasons should be thought for this difference. Most of time, it is observed that the transaction is performing well but there seems to be some issues

1.7

Load Injector Machine Setup

The virtual user scripts contain a single transaction or group of similar transactions which are generally executed sequentially. These scripts are configured to be executed on the load injector machine with the number of iterations as per the system workload profile and generate the transaction load on the system. Few points which can be considered while configuring the load injector machine and virtual scripts are: Data Caching: Mostly the virtual user script for a single transaction with assigned number of users is configured on single load injector machine. The data pool or pre-requisite data for the multiple iterations of the transaction could be different (as configured in the data pool) but the requests from all the assigned users for the candidate script are generated from the single load injector box. If the load injector box is caching some of the transaction specific static data, the response time of the further iteration of the candidate transaction would be reduced and would not portray the real time scenario. The machine setting should be properly done or the distribution of the virtual user script and transactions should be proper to avoid the data caching issue. Distribution of IPs on Load Injector: Multiple transaction scripts and virtual users configured on the load injector machine use the same IP address for execution. This in turn means that all the virtual users would be using the same IP subnet mask. In real time scenario, only few of the users would be using the same subnet mask. The test results in this scenario might differ from the real time results. Tools are available which can assign multiple IP addresses to different virtual users assigned on the same load injector box. This will generate a better picture of real time scenario. Bandwidth Simulation: The performance benchmarking exercise is normally performed on the server side. The load controller and injector machines are connected to the servers with High speed LANs and network congestion and delays in the transactions are not accounted. Tools are available to define the usage of the network bandwidth by virtual users on load injector machines. If the system is locally deployed or de-centralized, the benchmark tests should be performed with real time network bandwidth. If the system is globally rolled out having multiple sites injecting the load on the system at the same time, it should be tried that the transaction load from various sites is allocated on different load injector machines. These site specific machines should be simulated with the network bandwidth available at respective sites. The pre-requisite for this scenario would be the availability of network topology of various production sites. This would portray the near to real time network congestion and delays for the system. Memory Requirement of virtual users: All the users assigned to a load injector box require memory to execute the transaction. The maximum memory consumption by single virtual users should be derived in advance and virtual users should be

in the system when the iterations for the candidate transactions are performed.