Distributed performance testing using JMeter has been around for a while. The recipe goes something like this:
A JMeter client (the green box referred to here as JMeter master) drives the process. It has the test script (the JMX file). The actual testing is done by n JMeter Server instances (blue boxes above). When the test is initiated, the JMX script and the necessary data files are made available to all the server instances. The servers run the test instructions and communicate the results back to the JMeter client.
We had been implementing this recipe with the JMeter client running on a virtual machine (VM) and each of the JMeter server instances running on their own VM instances. Although this was an improvement on the bare-metal days it was fraught with problems:
A JMeter client (the green box referred to here as JMeter master) drives the process. It has the test script (the JMX file). The actual testing is done by n JMeter Server instances (blue boxes above). When the test is initiated, the JMX script and the necessary data files are made available to all the server instances. The servers run the test instructions and communicate the results back to the JMeter client.
We had been implementing this recipe with the JMeter client running on a virtual machine (VM) and each of the JMeter server instances running on their own VM instances. Although this was an improvement on the bare-metal days it was fraught with problems:
- Provisioning of a VM with just the right pieces and parts -- a highly iterative process -- was tedious and time consuming
- A change to the JMX script or the fixture data meant that the respective VM had to be reinstated. More tedium.
- Constraints in our work environment made addition of new JMeter server instances laborious
- Finally there was the cost of VM startup & shutdown. This was a nontrivial process and a time sink
I had been watching with interest the growing drumbeat of support for Docker. The more I looked into it the more I found it appealing.
- The relative ease of creating a Docker image
- Ability to iterate, correct and improve on an existing image
- Lastly the near non-existent startup time mean the feedback loop was really tight.
Noticing that the JMeter client (master) and the JMeter server were very similar from the point of view of provisioning I created a base JMeter image. Off of the base I created two other images -- JMeter (for the master) and JMeter-server. In summary:
jmeter-base
|
+-- jmeter
|
+-- jmeter-server
As you know the server process uses two ports; one to listen for instructions from the master and another to write responses back to the master. The server image exposed two ports for this purpose. In order to recreate the necessary interaction I did the following:
- Started n-instances of jmeter-server. Each of which was bound to two well known ports on the host
- Using the container ID of the server instance I determined their IP addresses
- Started the Jmeter client (master). The client image was crafted to receive the location of the remote server instances during invocation and write its log & test results back to the host
When the JMeter client started up it connected with every server instance. I monitored the master's log file on the host for all the action. When the tests completed I simply removed all the Docker containers. This left me with just the logs & test results!
Along the way I had eliminated all of my earlier annoyances.
- Creating and modifying the Docker image to just the right shape was fast and easy
- Since the JMX and data files were maintained on the host a change in these files simply meant I restarted my container
- Creating additional JMeter server instance was just a matter of another docker run invocation
- Finally startup time was almost imperceptible!
Perfect.....well, almost. I bothered me that the process of starting the containers and shutting them down was a manual process. So I started with a pie in the sky idea of what I would be an acceptable invocation and proceeded to implement one. I wanted to be able to do:
driver script datadir log-dir num-servers
Driver must:
- Create the specified number of JMeter server containers
- Create the JMeter master container
- Fire of the test
- Wait for the test to complete
- Remove all the containers
It took some scripting foo along with some Docker image revisions. I now have a setup that allows me to:
driver.sh -s jmxfile.jmx -d data-dir -n 8
This does everything I need except the container cleanup. I am in the process of implementing that.
Of course this only means I need to work on my next set of tweaks on my wish list. But that's for another post.
You can find the work referenced in this blog at:
I gave a lightning talk on this work. The slide deck I used for it is at http://www.slideshare.net/srivaths_sankaran/jmeter-docker-sitting-in-a-tree.
A huge hat tip to my colleague Vincent Batts for helping me along the way and letting me use his machine. Vincent is a rare bird — brilliant technical chops, passionate, generous, patient and humble.
A huge hat tip to my colleague Vincent Batts for helping me along the way and letting me use his machine. Vincent is a rare bird — brilliant technical chops, passionate, generous, patient and humble.
