JMeter

White Paper
By Rajesh Aedma

About the Author

Rajesh Aedma is an associate of Tata Consultancy Services having around 3.9 years of total IT work e !erience" worked !rimarily in the domain of #usiness and Telecom. $is technological forte is %ava&%'(( and )e* services. $e has around + years of work e !erience on testing the %ava )e*services using %,eter.

JMeter

Contents

+. Introduction
'. #uilding a Test -lan

'

'

3. (lements of a Test -lan

/. #uilding a )e*Service Test -lan .. 1imitations of %,eter 2. Conclusion

0 ++ ++

Confidential

JMeter

1. Introduction A!ache %,eter is a +334 !ure %ava deskto! a!!lication designed to load test client&server software 5such as a we* a!!lication6. It may *e used to test !erformance *oth on static and dynamic resources such as static files" %ava Servlets" C7I scri!ts" %ava o*jects" data*ases" 8Tservers" and more. %,eter can *e used to simulate a heavy load on a server" network or o*ject to test its strength or to analy9e overall !erformance under different load ty!es. Additionally" %,eter can hel! you regression test your a!!lication *y letting you create test scri!ts with assertions to validate that your a!!lication is returning the results you e !ect. 8or ma imum fle i*ility" %,eter lets you create these assertions using regular e !ressions. The easiest way to *egin using %,eter is to first download the latest !roduction release and install it. The release contains all of the files you need to *uild and run )e*" 8T-" %:#C" and %;:I tests. Requirements %,eter re<uires your com!uting environment meets some minimum re<uirements. %ava =ersion %,eter re<uires a fully com!liant %=, +.3 or higher. )e are making a strong effort currently to *e com!ati*le with the +.3 %=,>s? though it is e !ected %,eter !erforms *est with +./ or *etter. #ecause %,eter uses only standard %ava A-Is" !lease do not file *ug re!orts if your %R( fails to run %,eter *ecause of %R( im!lementation issues. %ava +.3 does not include SS1 5$TT-S6 su!!ort @ youAll need to download %SS(. Also" it does not !erform as well as later %ava versions. Bse the latest version of %ava +./ or +.. for *est results C!erating Systems %,eter is a +334 %ava a!!lication and should run correctly on any system that has a com!liant %ava im!lementation. Installation Installing %,eter is a sna!. S!ecifics de!end on which release file you downloaded. To install a release *uild" sim!ly un9i! the 9i!&tar file into the directory where you want %,eter to *e installed. -rovided that you have a %R(&%:D correctly installed and the %A=AE$C,( environment varia*le set" there is nothing more for you to do. Running JMeter To run %,eter" run the jmeter.*at 5for )indows6 or jmeter 5for B;IF6 file. %,eter must *e started from the %,eter *in directory 5where those files are found6. The jmeter.*at file attem!ts to change to the a!!ro!riate directory if it can. 2. Building a Test Plan A test !lan descri*es a series of ste!s %,eter will e ecute when run. A com!lete test !lan will consists of one or more Thread 7rou!s" logic controllers" sam!le generating controllers" listeners" timers" assertions" and configuration elements. Confidential

JMeter

Adding and Removing (lements Adding elements to a test !lan can *e done *y right@clicking on an element in the tree" and choosing a new element from the GaddG list. Alternatively" elements can *e loaded from file and added *y choosing the Go!enG o!tion. To remove an element" make sure the element is selected" right@click on the element" and choose the GremoveG o!tion. 1oading and Saving (lements To load an element from file" right click on the e isting tree element to which you want to add the loaded element" and select the Go!enG o!tion. Choose the file where your elements are saved. %,eter will load the elements into the tree. To save tree elements" right click on an element and choose the GsaveG o!tion. %,eter will save the element selected" !lus all child elements *eneath it. In this way" you can save test tree fragments" individual elements" or the entire test !lan. Configuring Tree (lements Any element in the test tree will !resent controls in %,eterAs right@hand frame. These controls allow you to configure the *ehavior of that !articular test element. )hat can *e configured for an element de!ends on what ty!e of element it is. The Test Tree itself can *e mani!ulated *y dragging and dro!!ing com!onents around the test tree. Running a Test -lan To run your test !lan" choose GstartG from the GrunG menu item. To sto! your test !lan" choose Gsto!G from the same menu. %,eter does not automatically give any indication of whether it is currently running. Some listeners make it o*vious if %,eter is running" *ut the only way to *e sure is to check the GrunG menu. If GstartG is disa*led" and Gsto!G is ena*led" %,eter is running your test !lan 5or" at least" it thinks it is6. Sco!ing Rules The %,eter test tree contains elements that are *oth hierarchical and ordered. Some elements in the test trees are strictly hierarchical 51isteners" Config (lements" -ost@-rocessors" -re@ -rocessors" Assertions" Timers6" and some are !rimarily ordered 5controllers" sam!lers6. )hen you create your test !lan" you will create an ordered list of sam!le re<uest 5via Sam!lers6 that re!resent a set of ste!s to *e e ecuted. These re<uests are often organi9ed within controllers that are also ordered. 7iven the following test treeH

( am!le test tree The order of re<uests will *e" Cne" Two" Three" 8our. Some controllers affect the order of their su* elements" and you can read a*out these s!ecific controllers in the com!onent reference. Confidential

JMeter

Cther elements are hierarchical. An Assertion" for instance" is hierarchical in the test tree. If its !arent is a re<uest" then it is a!!lied to that re<uest. If its !arent is a Controller" then it affects all re<uests that are descendants of that Controller. In the following test treeH

$ierarchy e am!le Assertion I+ is a!!lied only to Re<uest Cne" while Assertion I' is a!!lied to Re<uests Two and Three. Another e am!le" this time using TimersH

com!le e am!le In this e am!le" the re<uests are named to reflect the order in which they will *e e ecuted. Timer I+ will a!!ly to Re<uests Two" Three" and 8our 5notice how order is irrelevant for hierarchical elements6. Assertion I+ will a!!ly only to Re<uest Three. Timer I' will affect all the re<uests. $o!efully these e am!les make it clear how configuration 5hierarchical6 elements are a!!lied. If you imagine each Re<uest *eing !assed u! the tree *ranches" to its !arent" then to its !arentAs !arent" etc" and each time collecting all the configuration elements of that !arent" then you will see how it works.

Confidential

JMeter

3. lements o! a Test Plan The Test -lan o*ject has a new check*o to select called G8unctional TestingG. If selected" it will cause %,eter to record the data returned from the server for each sam!le. If you have selected a file in your test listeners" this data will *e written to file. This can *e useful if you are doing a small run to ensure that %,eter is configured correctly" and that your server is returning the e !ected results. The conse<uence is that the file will grow huge <uickly" and %,eterAs !erformance will suffer. This o!tion should *e off if you are doing stress@testing 5it is off *y default6. If you are not recording the data to file" this o!tion makes no difference. Thread "rou# Thread grou! elements are the *eginning !oints of any test !lan. All elements of a test !lan must *e under a thread grou!. As the name im!lies" the thread grou! element controls the num*er of threads %,eter will use to e ecute your test. The controls for a thread grou! allow you toH J J J Set the num*er of threads Set the ram!@u! !eriod Set the num*er of times to e ecute the test

(ach thread will e ecute the test !lan in its entirety and com!letely inde!endently of other test threads. ,ulti!le threads are used to simulate concurrent connections to your server a!!lication. The ram!@u! !eriod tells %,eter how long to take to Gram!@u!G to the full num*er of threads chosen. If +3 threads are used" and the ram!@u! !eriod is +33 seconds" then %,eter will take +33 seconds to get all +3 threads u! and running. (ach thread will start +3 5+33&+36 seconds after the !revious thread was *egun. If there are 33 threads and a ram!@u! !eriod of +'3 seconds" then each successive thread will *e delayed *y / seconds. Ram!@u! needs to *e long enough to avoid too large a work@load at the start of a test" and short enough that the last threads start running *efore the first ones finish 5unless one wants that to ha!!en6. Start with Ram!@u! K num*er of threads and adjust as needed. #y default" the thread grou! is configured to loo! once through its elements. =ersion +.9 introduces a test run scheduler. Click the check*o at the *ottom of the Thread 7rou! !anel to reveal e tra fields in which you can enter the start and end times of the run. )hen the test is started" %,eter will wait if necessary until the start@time has *een reached. At the end of each cycle" %,eter checks if the end@time has *een reached" and if so" the run is sto!!ed" otherwise the test is allowed to continue until the iteration limit is reached. Alternatively" one can use the relative delay and duration fields. ;ote that delay overrides start@ time" and duration over@rides end@time. Controllers %,eter has two ty!es of ControllersH Sam!lers and 1ogical Controllers. Sam!lers tell %,eter to send re<uests to a server. 8or e am!le" add an $TT- Re<uest Sam!ler if you want %,eter to send an $TT- re<uest. Lou can also customi9e a re<uest *y adding one or more Configuration (lements to a Sam!ler. 8or more information" see Sam!lers. 1ogical Controllers let you customi9e the logic that %,eter uses to decide when to send re<uests. 8or e am!le" you can add an Interleave 1ogic Controller to alternate *etween two $TT- Re<uest Sam!lers. 8or more information" see 1ogical Controllers. Confidential

JMeter

Sam!lers Sam!lers tell %,eter to send re<uests to a server. %,eter sam!lers includeH J J J J J J J 8T- Re<uest $TT- Re<uest %:#C Re<uest %ava o*ject re<uest 1:A- Re<uest SCA-&F,1@R-C Re<uest )e*Service 5SCA-6 Re<uest

(ach sam!ler has several !ro!erties you can set. Lou can further customi9e a sam!ler *y adding one or more Configuration (lements to it. ;ote that %,eter sends re<uests in the order that the sam!lers a!!ear in the tree. If you are going to send multi!le re<uests of the same ty!e 5for e am!le" $TT- Re<uest6 to the same server" consider using a :efaults Configuration (lement. (ach controller has one or more :efaults elements 5see *elow6. Remem*er to add a 1istener to your Thread 7rou! to view and&or store the results of your re<uests to disk. If you are interested in having %,eter !erform *asic validation on the res!onse of your re<uest" add an Assertion to the Re<uest controller. 8or e am!le" in stress testing a we* a!!lication" the server may return a successful G$TT- Res!onseG code" *ut the !age may have errors on it or may *e missing sections. Lou could add assertions to check for certain $T,1 tags" common error strings" and so on. %,eter lets you create these assertions using regular e !ressions. %,eterAs *uilt@in sam!lers 1ogic Controllers 1ogic Controllers let you customi9e the logic that %,eter uses to decide when to send re<uests. 1ogic Controllers may have as child elements any of the followingH Sam!lers 5re<uests6" Configuration (lements" and other 1ogic Controllers like 1oo! Controller. 1ogic Controllers can change the order of re<uests coming from their child elements. They can modify the re<uests themselves" cause %,eter to re!eat re<uests" etc. $isteners 1isteners !rovide access to the information %,eter gathers a*out the test cases while %,eter runs. The 7ra!h Results listener !lots the res!onse times on a gra!h. The G=iew Results TreeG 1istener shows details of sam!ler re<uests and res!onses" and can dis!lay *asic $T,1 and F,1 re!resentations of the res!onse. Cther listeners !rovide summary or aggregation information. Additionally" listeners can direct the data they collect to a file for later use. (very listener in %,eter !rovides a field to indicate the file to store data to. 1isteners can *e added anywhere in the test. They will collect data only from elements at or *elow their level. Timers #y default" a %,eter thread sends re<uests without !ausing *etween each re<uest. )e recommend that you s!ecify a delay *y adding one of the availa*le timers to your Thread 7rou!. If you do not add a delay" %,eter could overwhelm your server *y making too many re<uests in a very short amount of time. Confidential

JMeter

The timer will cause %,eter to delay a certain amount of time *etween each re<uest that a thread makes. If you choose to add more than one timer to a Thread 7rou!" %,eter takes the sum of the timers and !auses for that amount of time *efore e ecuting the sam!lers to which they a!!ly. Assertions Assertions allow you to assert facts a*out res!onses received from the server *eing tested. Bsing an assertion" you can essentially GtestG that your a!!lication is returning the results you e !ect it to. 8or instance" you can assert that the res!onse to a <uery will contain some !articular te t. The te t you s!ecify can *e a -erl@style regular e !ression" and you can indicate that the res!onse is to contain the te t" or that it should match the whole res!onse. Lou can add an assertion to any Sam!ler. 8or e am!le" you can add an assertion to a $TTRe<uest that checks for the te t" GM&$T,1NG. %,eter will then check that the te t is !resent in the $TT- res!onse. If %,eter cannot find the te t" then it will mark this as a failed re<uest. To view the assertion results" add an Assertion 1istener to the Thread 7rou!. Con!iguration lements A configuration element works closely with a Sam!ler. Although it does not send re<uests 5e ce!t for $TT- -ro y Server6" it can add to or modify re<uests. A configuration element is accessi*le from only inside the tree *ranch where you !lace the element. 8or e am!le" if you want to add Bser :efined varia*les we can add the name&value !airs. And that name is used in the ml>s instead of modifying each time in the ml>s when multi!le times the value is needed. Pre%Processor lements A -re@-rocessor e ecutes some action !rior to a Sam!ler Re<uest *eing made. If a -re@ -rocessor is attached to a Sam!ler element" then it will e ecute just !rior to that sam!ler element running. A -re@-rocessor is most often used to modify the settings of a Sam!le Re<uest just *efore it runs" or to u!date varia*les that arenAt e tracted from res!onse te t. See the sco!ing rules for more details on when -re@-rocessors are e ecuted. Post%Processor lements A -ost@-rocessor e ecutes some action after a Sam!ler Re<uest has *een made. If a -ost@ -rocessor is attached to a Sam!ler element" then it will e ecute just after that sam!ler element runs. A -ost@-rocessor is most often used to !rocess the res!onse data" often to e tract values from it. See the sco!ing rules for more details on when -re@-rocessors are e ecuted. &ecution order +. Timers @ if any '. Sam!ler 3. -ost@-rocessors 5unless Sam!leResult is null6 /. Assertions 5unless Sam!leResult is null6 .. 1isteners 5unless Sam!leResult is null6 '. Building a (eb)er*ice Test Plan In this section" you will learn how to create a Test -lan to test a )e*Service. To construct the Test -lan" you will use the following elementsH Thread 7rou!" SCA-&F,1@R-C Re<uest" and Aggregate Results. Confidential

JMeter

If the sam!ler a!!ears to *e getting an error from the we*service" dou*le check the SCAmessage and make sure the format is correct. In !articular" make sure the mlns attri*utes are e actly the same as the )S:1. If the ml names!ace is different" the we*service will likely return an error. Fmethods contains a list of !u*lic we*service for those who want to test their test !lan. Adding Thread 7rou! The first ste! you want to do with every %,eter Test -lan is to add a Thread 7rou! element. The Thread 7rou! tells %,eter the num*er of users you want to simulate" how often the users should send re<uests" and the how many re<uests they should send. 7o ahead and add the Thread7rou! element *y first selecting the Test -lan" clicking your right mouse *utton to get the Add menu" and then select Add @@N Thread7rou!. Lou should now see the Thread 7rou! element under Test -lan. If you do not see the element" then Ge !andG the Test -lan tree *y clicking on the Test -lan element. ;e t" you need to modify the default !ro!erties. Select the Thread 7rou! element in the tree" if you have not already selected it. Lou should now see the Thread 7rou! Control -anel in the right section of the %,eter window 5see 8igure +.' *elow6

8igure +.'. Thread 7rou! with :efault =alues Start *y !roviding a more descri!tive name for our Thread 7rou!. In the name field" enter %akarta Bsers. )e can also increase the num*er of users 5called threads6. Adding )e*Service Re<uests In our Test -lan" we will use a %ava we*service. Since youAre using the we*service sam!ler" we wonAt go into the details of writing a we*service. If you donAt know how to write a we*service" google for we*service and familiari9e yourself with writing we*services for %ava and .;(T. It Confidential

JMeter

should *e noted there is a significant difference *etween how .;(T and %ava im!lement we*services. %,eter sends re<uests in the order that they a!!ear in the tree. Start *y adding the sam!ler SCA-&F,1@R-C Re<uest to the %akarta Bsers element 5Add @@N Sam!ler @@N SCA-&F,1@R-C Re<uest. Then" select the SCA-&F,1@R-C Re<uest element in the tree and edit the following !ro!erties 5see 8igure +./6H +. Change the ;ame field to OSCA-&F,1@R-C Re<uest G. '. (nter the A!!lication BR1 in the BR1 field.

8igure +./. SCA-&F,1@R-C Re<uest The last ste! is to !aste the SCA- message in the GSCA-&F,1@R-C :ataG te t area. Lou can o!tionally save the soa! message to a file and *rowse to the location. 8or convenience" there is a third o!tion of using a message folder. The sam!ler will randomly select files from a given folder and use the te t for the soa! message.

Confidential

JMeter

Currently" only %ava uses SCA-Action" so it is normal to have a *lank SCA-Action for all other we*services. The list includes %)S:-" )e*logic" A is" The ,ind (lectric 7lue" and gSoa!. Adding a 1istener to =iew Store the Test Results The final element you need to add to your Test -lan is a 1istener. This element is res!onsi*le for storing all of the results of your $TT- re<uests in a file. #y o*serving the results we can see the through!ut that !er hour how many re<uests are running" ma and min time" Sam!le re<uests and error 4. Select the %akarta Bsers element and add a Aggregate Results listener 5Add @@N 1istener @@N Aggregate Results6. ;e t" you need to s!ecify a directory and filename of the out!ut file. Lou can either ty!e it into the filename field" or select the #rowse *utton and *rowse to a directory and then enter a filename. 5See 8igure +..6

8igure +... Aggregate Results 1istener Saving the Test -lan

Confidential

10

JMeter

Although it is not re<uired" we recommend that you save the Test -lan to a file *efore running it. To save the Test -lan" select Save Test -lan from the 8ile menu 5with the latest release" it is no longer necessary to select the Test -lan element first6. %,eter allows you to save the entire Test -lan tree or only a !ortion of it. To save only the elements located in a !articular G*ranchG of the Test -lan tree" select the Test -lan element in the tree from which to start the G*ranchG" and then click your right mouse *utton to access the Save As menu item. Alternatively" select the a!!ro!riate Test -lan element and then select Save As from the (dit menu. Running the Test -lan 8rom the Run menu" select Run. %,eter lights u! a green s<uare in the u!!er@right@hand corner to indicate if a test is currently running. The s<uare is turned gray when all tests sto!. (ven after you select Gsto!G" the green light will stay on until all test threads have e ited. Cnce %,eter has finished running your Test -lan" select Sto! from the Run menu. If you selected a file to save the results to in your listener" then you will have a file that can *e o!ened in any visuali9er. (ach visuali9er will dis!lay the results in its own fashion. It is !ossi*le to have the same file o!en in more than one visuali9er. This is not a !ro*lem. %,eter will ensure during the test run that no sam!le is recorded to the same file more than once. +. $imitations o! JMeter I have o*served *y using %,eter for )e*service Testing" some limitations that %,eter does not su!!ort testing of 7BI a!!lications" %,eter only tests synchronous calls @ i.e. re<uest&res!onse as in $TT-. $owever" itAs all highly de!endent on the test details. A test that !arses res!onses for em*edded resources is going to kill %,eterAs !erformance a lot faster than one that doesnAt. Regular e !ression !arsers" assertions also add significant !rocessing time for %,eter. A com!le test might have +33 users as a ma on that machine. A sim!le test might allow u! to .33 users. It also de!ends on how fast the server is @ a faster server !uts more stress on %,eter *ecause each thread s!ends less time *locked on IC and more time !re!aring the ne t re<uest. And" of course" timers. A few timers increases dramatically the num*er of users that can effectively *e simulated. I think the *iggest limitation of %,eter is that the remote testing feature was not ade<uately made" and does not in fact get you anything in terms of scaling tests u!. ,. Conclusion These days, though the attitude towards Testing as a phase & Testing as a career has definitely undergone a sea change. This tool is useful for *oth functional testing as well as -erformance testing. It>s a freeware tool" we can download the latest version of %,eter. In this way" we can make use of Timers" Assertions" -re@-rocessors" -ost@-rocessors" etc.. for )e*service testing. Re!erences

1. References for more information in htt!H&&jakarta.a!ache.org&jmeter '. References for -erformance Testing )e* Services using %,eter in
htt!H&&dev'dev.*ea.com&!u*&a&'332&3P&jmeter@!erformance@testing.html

Confidential

11