JMeter

JMeter

Vijayaaanand E (vijayaaanand.e@tcs.com) Media & Information Services

Confidential

JMeter

Confidentiality Statement
2011 TATA CONSULTANCY SERVICES LIMITED All rights reserved. This document contains confidential information of TATA CONSULTANCY SERVICES LIMITED which is provided for the sole purpose of permitting the recipient to evaluate any proposal submitted herewith or separately. In consideration of receipt of this document, the recipient agrees to maintain such information in confidence and to not reproduce or otherwise disclose this information to any person outside the group directly responsible for evaluation of its contents, except that there is no obligation to maintain the confidentiality of any information which was known to the recipient prior to receipt of such information from TATA CONSULTANCY SERVICES LIMITED, or becomes publicly known through no fault of recipient, or is received without obligation of confidentiality from a third party owing no obligation of confidentiality to TATA CONSULTANCY SERVICES LIMITED. This document is the property of TATA CONSULTANCY SERVICES LIMITED and TATA CONSULTANCY SERVICES LIMITED may require the return of this document at any time at their sole discretion. Unauthorized access or copying is prohibited witho t the prior written permission of TATA CONSULTANCY SERVICES LIMITED. All product and company names mentioned herein may be trademarks of their respective owners.

Confidential

JMeter

Performance Testing What is Performance Testing?

Performance testing of an application / system is basically the process of understanding how the application and its operating environment behave at various user loads. In general, it is performed by simulating virtual users to determine / validate the scalability, availability, robustness, hardware & software resource utilization of the application thus ideally paving the way for effective capacity planning of the system for future usage. One of the main objectives of performance testing is to help maintain the system with low latency, high throughput, and low utilization. Various types of Performance Testing include: Load Testing, Stress Testing, Volume Testing, Spike Testing, Endurance Testing

Need for Performance Testing

Performance Testing sets the best possible performance expectation under a given configuration of infrastructure. It also highlights early in the testing process if changes need to be made before application goes into production. Some of the reasons for which Performance Testing has to be mandatorily carried out are: Poor Response Time A critical webpage may take one minute to show up. The expected response time is less than 10 seconds. Scalability When 100 users from new branch were introduced into the system, the application was unable to handle the additional load and response times degraded from 5 seconds to 50 seconds. Batch Window Issues The batch upload that is supposed to happen in a 1 AM to 7 AM window does not complete till 9:30 AM. This impacts the availability of data for online applications. Availability The application server needs to be restarted owing to resource leaks within the application. This causes a downtime of atleast 0.5 hours a day. Resource Capacity Planning An application consumes 90% CPU on a server that is supposed to host another application after 3 months. There is no extra capacity to take this additional load.

Costs associated with Performance testing

Performance testing usually has huge costs associated with it. The costs associated with Performance testing can be categorized as The Investment made on the Performance testing tool. Time consumed in Preparation of test data.

Confidential

JMeter

The Load Testing Infrastructure / environment setup should be as close as possible to the production environments. In case of administrative organizations, acquiring the required permissions is a tedious process, since performance tests typically require many administrative rights. Bottom-line is while there are many costs associated with Performance testing, the Investment on the tool is considered to be the most indispensable contributor to the total cost. Most of the commercial tools require separate licenses to be procured based on the user load which contributes to huge Performance testing effort cost itself.

Open Source Solution

Quest for better alternative The Investment on the tool has lead to search out for better alternative solutions. This quest has ended in the Open Source Performance Testing tools without sacrificing quality. What is Open Source? Open Source is defined as a software code that is available for users to examine and change freely without violating any patents, copyright laws, or licensing agreements. The idea with open source is that when everyone can work together and build upon existing tools, the ultimate result is much better software. It is a way for many companies and individuals to collaborate and improve software that each person could not do alone. Why should one go for open source? No cost Absolutely FREE!! Evolving Software (Can be customized to users needs). High portability. Scalable and offers an effective solution for growing business demands. Huge open source forum technical support which stands in par with any other commercial tool support. In fact its more advantageous as there is no cost or logistics involved like other commercial tool technical support.

Application Architecture An example

Portlet data is customized according to user queries and data is retrieved from the database. The data is then fed into the Common Content management system after which it is returned to the browser. So thus all the various Web & Web-service based applications which were involved in this Content management system were successfully tested using the Open Source Performance Testing Tool Apache JMeter.

Confidential

JMeter

Core Performance Testing Activities

The following are some of the key activities during the successful implementation of Performance testing for the leading financial organization using JMeter: Understand and Analyze the Business Requirements. Define Performance Test Goals and create a Performance Test Plan. Create JMeter Scripts for identified critical business flows and execute various load test scenarios. Prepare customized analysis summary reports for every test execution and an overall Engagement Summary Report at the end of the engagement. Based on the performance bottlenecks idenitified, provide suggestions to the application development team for tuning the application. Test executions are thus run in iterative manner until the desired acceptance criteria are met.

Implementation: Challenges faced

Being the first venture into the world of Open source Performance testing, the following were the challenges. Newer concepts and various ways of implementing different protocols had to be explored from scratch. For instance Bean Shell Technology, Handling FTP requests and dealing with various types of Java based requests. The reporting aspect of the tool was very limited. The data given by the tool was in a raw form and a lot of manual effort had to be involved in order to derive meaningful results and to generate elegant presentable reports. Lot of other open source JMeter utilities and plug-ins had to be be explored in order to effectively replicate the scenarios. The associates had to be highly flexible For instance JMeter cannot record the AJAX specific requests. However it will be able to play them back. Hence those missing requests, headers had to be captured from other HTTP profiling tools like HTTP Watch and Packet Sniffing tools such as Wireshark and had to be placed in JMeter. Hence one has to be highly flexible in terms of technical expertise and usage of various tools and technologies in order to yield the maximum benefits out of Open Source.

Open Source Benefits Huge Open Source Support - Since the tool has a huge open source support
community, learning new protocols were relatively easy and the queries posted almost had solutions. This is beneficial as most commercial tools include separate cost for providing technical support.

No need of High-tech infrastructure - Doesnt demand a state of the art

infrastructure for load testing.

Distributed Testing - Supports multiple load-injectors managed by a single

controller.

Confidential

JMeter

User-friendly GUI - Most of the popular Performance Testing open source tools
have a very good GUI thereby allowing faster operations and more precise timings. Hence learning and adapting to the tool becomes easier over a period of time.

No Licenses and Maintenance Issues - Most commercial tools require separate licenses to be procured based on the user load specifications which contribute to huge tool cost. In addition to that, there involves an extra burden of maintaining the licenses. With the use of Open Source tools no such cost / efforts are involved to explicitly maintain the tool. Apache JMeter Features & Overview
Apache JMeter is a powerful desktop performance tool from the Apache Jakarta project, written in Java, for load-testing web pages, web applications, and other static and dynamic resources including databases, files, Servlets, Perl scripts, Java Objects, FTP Servers, and more. The main component in JMeter is the Java Swing-based Graphical User Interface which can be used both for Scripting and Execution. Excellent cost saving solution for small projects as it is an open source tool. Robust in handling complex test scenarios that demand n number of virtual users. Complete portability and supports 100% all the Java based applications. Less scripting efforts as compared to other tools because of its user effective GUI. Is used to conduct Load, Stress, Volume & Endurance tests on the Web & Web-service based applications. HTTP & HTTPS load testing can be done without any adding additional pluggable samplers. Since it is Java based, the tool was highly compatible with most of the Java based requests i.e. it can be used to directly test the Java requests, JDBC requests and JMS publisher & Subscriber. Hence it is a very good cost effective solution for small development activities as well.

Apache JMeter Limitations

Reporting Framework Limitations 1. Does not provide enough statistics: Most of the open source tools (JMeter as well) does not provide the client side statistics summary reports in a presentable format as available with the other popular commercial tools such as HP Load Runner and Borland Silk Performer. 2. Huge Log Files: Added to that JMeter provides the log details for every user and every iteration thus in turn causing the log files generated to be very huge. 3. Manual Effort: Analyzing such voluminous log files involves a lot of manual effort in order to create highly sophisticated analysis reports.

Confidential

JMeter

4. Time Consuming: Above all, creating the final customer presentable summary report consumes huge amount of time.

Confidential