Friday, August 29, 2014

Distrubuted JMeter testing using Docker

Distributed performance testing using JMeter has been around for a while.  The recipe goes something like this:


A JMeter client (the green box referred to here as JMeter master) drives the process.  It has the test script (the JMX file).  The actual testing is done by n JMeter Server instances (blue boxes above).  When the test is initiated, the JMX script and the necessary data files are made available to all the server instances.  The servers run the test instructions and communicate the results back to the JMeter client.

We had been implementing this recipe with the JMeter client running on a virtual machine (VM) and each of the JMeter server instances running on their own VM instances.  Although this was an improvement on the bare-metal days it was fraught with problems:

  • Provisioning of a VM with just the right pieces and parts -- a highly iterative process -- was tedious and time consuming
  • A change to the JMX script or the fixture data meant that the respective VM had to be reinstated.  More tedium.
  • Constraints in our work environment made addition of new JMeter server instances laborious
  • Finally there was the cost of VM startup & shutdown.  This was a nontrivial process and a time sink
I had been watching with interest the growing drumbeat of support for Docker.  The more I looked into it the more I found it appealing.
  • The relative ease of creating a Docker image
  • Ability to iterate, correct and improve on an existing image
  • Lastly the near non-existent startup time mean the feedback loop was really tight.
Noticing that the JMeter client (master) and the JMeter server were very similar from the point of view of provisioning I created a base JMeter image.  Off of the base I created two other images -- JMeter (for the master) and JMeter-server.  In summary:

jmeter-base
  |
  +-- jmeter
  |
  +-- jmeter-server

As you know the server process uses two ports; one to listen for instructions from the master and another to write responses back to the master.  The server image exposed two ports for this purpose.  In order to recreate the necessary interaction I did the following:
  • Started n-instances of jmeter-server.  Each of which was bound to two well known ports on the host
  • Using the container ID of the server instance I determined their IP addresses
  • Started the Jmeter client (master).  The client image was crafted to receive the location of the remote server instances during invocation and write its log & test results back to the host
When the JMeter client started up it connected with every server instance.  I monitored the master's log file on the host for all the action.  When the tests completed I simply removed all the Docker containers.  This left me with just the logs & test results! 

Along the way I had eliminated all of my earlier annoyances.

  • Creating and modifying the Docker image to just the right shape was fast and easy
  • Since the JMX and data files were maintained on the host a change in these files simply meant I restarted my container

  • Creating additional JMeter server instance was just a matter of another docker run invocation
  • Finally startup time was almost imperceptible!
Perfect.....well, almost.  I bothered me that the process of starting the containers and shutting them down was a manual process.  So I started with a pie in the sky idea of what I would be an acceptable invocation and proceeded to implement one.  I wanted to be able to do:

driver script datadir log-dir num-servers    

Driver must:
  • Create the specified number of JMeter server containers
  • Create the JMeter master container
  • Fire of the test
  • Wait for the test to complete
  • Remove all the containers
It took some scripting foo along with some Docker image revisions.  I now have a setup that allows me to:

driver.sh -s jmxfile.jmx -d data-dir -n 8

This does everything I need except the container cleanup.  I am in the process of implementing that.

Of course this only means I need to work on my next set of tweaks on my wish list.  But that's for another post.

You can find the work referenced in this blog at:

I gave a lightning talk on this work.  The slide deck I used for it is at http://www.slideshare.net/srivaths_sankaran/jmeter-docker-sitting-in-a-tree.


A huge hat tip to my colleague Vincent Batts for helping me along the way and letting me use his machine.  Vincent is a rare bird — brilliant technical chops, passionate, generous, patient and humble.



30 comments:

  1. wow. good work, thank you for the post.

    - Santosh

    ReplyDelete
  2. Good post, there is no official jmeter docker image yet so will grab yours :-)

    ReplyDelete
  3. There is no ssankara/jmeter* images, however i searched from docker hub and got it from Santosh Arakere Marigowda's docker hub.

    ReplyDelete
  4. Re comment from Anonymous on May 8, 2015 at 11:08 PM:

    Fair point. Yes, that image does not exist in any public registry. You have to build it from the jmeter & jmeter-server Dockerfiles and name it accordingly. Sorry for not stating that explicitly.

    ReplyDelete
  5. Hi Srivaths,

    Nice approach for load testing. We want to use something similar hosted on Azure infrastructure.
    Few questions:
    - when simulating concurrent user access how many users you simulate on a single JMeter server instance?
    - what would you recommend in terms of VM size that hosts docker containers that run JMeter? E.g what is relation between VM size and a number of containers it would make sense to run on it?

    Thanks
    Artem

    ReplyDelete
    Replies
    1. Artem:
      Glad you found my post useful.
      The size of the simulation is dictated by the expected use. You then want to apply that kind of load (load test). Then you want be able to *exceed* that to find the breaking point (stress test). That said there is a tipping point with regards to JMeter itself. Roughly you can handle about 100 users for a "typical" virtual machine (VM) with a single core & 8GBytes of RAM. Of course you would have to factor in the frequency of the requests made, the size of these requests etc.

      The answer to the second is to a large extent dictated by *what* the containers are doing (your first question). Start small (say 15 containers on a "typical" VM) and go from there. What I am trying to say here is that there is no easy formula you can apply to get an answer.

      Hope that helps.

      Sri

      Delete
  6. Hi Srivaths,

    This is brilliant! However do you have a idiot guide to complete setup?

    ReplyDelete
    Replies
    1. @Pavan:
      Glad you liked the article. I don't have much more by way of written instructions. I can try to help you with any particular portion on which you want elaboration.

      Sri

      Delete
    2. @srivaths,

      Thanks, iam having issue in running Elasticsearch,ELK,logstatsh and Grafana integration with Jmeter in Ubuntu and also issues in running Master and 3 servers.
      Will get in touch with you soon!

      Delete
  7. Very nice tutorial! I am new to docker. In JMeter we go for distributed testing when a machine can not generate enough load. So we need multiple hosts. If you create all the containers in the same host, how can you generate load? if you are going to install docker & set up jmeter with dockerfile in multiple hosts, how the driver script can be used here ?

    ReplyDelete
    Replies
    1. @RamKumar:
      You have made a good observation that all the containers are on the same host. The usage load that you can generate this way is predicated on

      a) the hardware of the host and

      b) what "usage" represents.

      Since you say that you are fairly new to Docker I would like to point out that you can typically create a several more JMeter-Server instances in the form of Docker containers compared to what you could do if the JMeter-Server instances were virtual machines.

      All that said, if you need to generate still more load than you can on one host simply repeat the above on another host. You could have a higher level master script that orchestrates all such server hosts.

      Delete
  8. Following statement is not correct -

    JMeter does not transfer test data filed to machines pumping load.

    ReplyDelete
  9. Looks really good, but I couldn't find your images (ssankara/jmeter and ssankara/jmeter-server) on docker hub, so building and running fails...

    ReplyDelete
    Replies
    1. Sorry, they aren't on Docker hub. You have to build them from the GitHub repositories.

      Delete
  10. Sri,

    You may be able to help me. How do you access a MariDB in a Docker container via a local instance of Jmeter, I have imported a JMX file to load test the MariaDB, but it can't access it. I can carry this task out locally when both applications are local, but I'd like to test the container remotely using Jmeter.

    ReplyDelete
  11. Treat the Docker container no differently than you would a remote server. If you are trying to connect to a MariaDB instance on a remote server you would provide the host & port in your connection string (property or whatever you are using). Do the same in this case. You should have access to the IP addresses of the Docker container(s) you have created.

    ReplyDelete
  12. This is great -- thanks! Is there a way to reference a csv file (using jmeter's CSV Data Set config)?

    ReplyDelete
  13. I am not familiar with that feature and I haven't tried it. That said, I would expect usage to be the same as in a non-Dockerized environment.

    ReplyDelete
  14. Hi Srivaths if i want to use jmx file with input some file how can i give the paths of that files like jmx file has input taken from /home/hosts.txt

    ReplyDelete
    Replies
    1. I haven't tried that. You can try by specifying a full qualified path.

      Delete
  15. Hi Srivaths,

    This is a very useful post and thanks for such a clear write up.
    I have a question regarding the way "WORK_DIR" needs to be setup for use in the driver script.
    Does this refer to a directory on the host machine or a volume mounted on the container?

    ReplyDelete
  16. Glad you found it helpful.

    WORK_DIR references a directory where the driver script runs i.e. the host machine.

    ReplyDelete
  17. Hi Srivaths,

    I'm trying to run multiple instances of the driver.sh script from the same host, but always run into the problem - "Error '125' while starting a jmeter server. Quitting". I figured this was due to the conflict in read/write port numbers being used, so, I tried using different port ranges for different runs. But, still getting the same error. Could you please help me find a solution for this issue?

    ReplyDelete
    Replies
    1. When I wrote this tool I did not have a need to run multiple instances of the driver script. So I did not consider or make accommodations for that use case. That is not to say that it will not work; just that I did not take that into account.

      I am little pressed for time at the moment so may not be able to immediately dedicate time to your request. I suggests the following tests to validate your hunch for the cause of error:

      1) Just run one instance of driver.sh and see if it work
      2) If (1) is successful try a minimal run (1 server each) with two instances of the driver

      If (2) is successful then crank it up a little (more servers on each driver) 'til it breaks.
      If (2) is not successful your theory could be correct. This will need more research.

      Hope that gets you going

      Sri

      Delete
  18. Hi Srivaths,

    Thankyou for this wonderful solution. As Sri Sankaran pointed out iam also getting a 125 Error while startin Jmeter server for 1 instance.

    ReplyDelete
  19. Hi,May i know how i can execute multiple .jmx scripts ??

    ReplyDelete
    Replies
    1. I had not considered that when I wrote this up. You could possibly have this process run in a loop with each JMX or update the driver script to consume a list of JMXs.

      Delete

Tweety thoughts

    follow me on Twitter