Distrubuted JMeter testing using Docker

Distributed performance testing using JMeter has been around for a while.  The recipe goes something like this:

A JMeter client (the green box referred to here as JMeter master) drives the process.  It has the test script (the JMX file).  The actual testing is done by n JMeter Server instances (blue boxes above).  When the test is initiated, the JMX script and the necessary data files are made available to all the server instances.  The servers run the test instructions and communicate the results back to the JMeter client.

We had been implementing this recipe with the JMeter client running on a virtual machine (VM) and each of the JMeter server instances running on their own VM instances.  Although this was an improvement on the bare-metal days it was fraught with problems:

• Provisioning of a VM with just the right pieces and parts -- a highly iterative process -- was tedious and time consuming
• A change to the JMX script or the fixture data meant that the respective VM had to be reinstated.  More tedium.
• Constraints in our work environment made addition of new JMeter server instances laborious
• Finally there was the cost of VM startup & shutdown.  This was a nontrivial process and a time sink

I had been watching with interest the growing drumbeat of support for Docker.  The more I looked into it the more I found it appealing.

• The relative ease of creating a Docker image
• Ability to iterate, correct and improve on an existing image
• Lastly the near non-existent startup time mean the feedback loop was really tight.

Noticing that the JMeter client (master) and the JMeter server were very similar from the point of view of provisioning I created a base JMeter image.  Off of the base I created two other images -- JMeter (for the master) and JMeter-server.  In summary:

jmeter-base

  |

  +-- jmeter

  |

  +-- jmeter-server

As you know the server process uses two ports; one to listen for instructions from the master and another to write responses back to the master.  The server image exposed two ports for this purpose.  In order to recreate the necessary interaction I did the following:

• Started n-instances of jmeter-server.  Each of which was bound to two well known ports on the host
• Using the container ID of the server instance I determined their IP addresses
• Started the Jmeter client (master).  The client image was crafted to receive the location of the remote server instances during invocation and write its log & test results back to the host

When the JMeter client started up it connected with every server instance.  I monitored the master's log file on the host for all the action.  When the tests completed I simply removed all the Docker containers.  This left me with just the logs & test results! 

Along the way I had eliminated all of my earlier annoyances.

• Creating and modifying the Docker image to just the right shape was fast and easy
• Since the JMX and data files were maintained on the host a change in these files simply meant I restarted my container

• Creating additional JMeter server instance was just a matter of another docker run invocation
• Finally startup time was almost imperceptible!

Perfect.....well, almost.  I bothered me that the process of starting the containers and shutting them down was a manual process.  So I started with a pie in the sky idea of what I would be an acceptable invocation and proceeded to implement one.  I wanted to be able to do:

driver script datadir log-dir num-servers    

Driver must:

• Create the specified number of JMeter server containers
• Create the JMeter master container
• Fire of the test
• Wait for the test to complete
• Remove all the containers

It took some scripting foo along with some Docker image revisions.  I now have a setup that allows me to:

driver.sh -s jmxfile.jmx -d data-dir -n 8

This does everything I need except the container cleanup.  I am in the process of implementing that.

Of course this only means I need to work on my next set of tweaks on my wish list.  But that's for another post.

You can find the work referenced in this blog at:

• https://github.com/srivaths/jmeter-driver
• https://github.com/srivaths/jmeter-base
• https://github.com/srivaths/jmeter
• https://github.com/srivaths/jmeter-server

I gave a lightning talk on this work.  The slide deck I used for it is at http://www.slideshare.net/srivaths_sankaran/jmeter-docker-sitting-in-a-tree.

---

A huge hat tip to my colleague Vincent Batts for helping me along the way and letting me use his machine.  Vincent is a rare bird — brilliant technical chops, passionate, generous, patient and humble.

Women in IT

I have come to appreciate the unique perspective each of us brings to the job.  By "perspective" I am going beyond technical skills and programming acumen.  It is that intangible quality that determines our design decisions, the thought process that directs our problem-solving strategy, that dictates how we communicate with our team. 

I feel that unique secret sauce that makes us who we are is involuntarily seeded by our gender.  To get a full breadth of capabilities I think it best if projects have both men and women.  The most successful projects in which I have been involved included a mix of men and women.  I don't think it is simply a coincidence.  I know that it is a key reason for the project's success.  It is therefore regrettable that there aren't more women in our line of work.

I came across a thought-provoking blog post by Ted Neward on this very subject.  In case you aren't familiar with Ted, he is an outstanding technologist.  I got to know Ted back when I used to speak on the NoFluffJustStuff tour.  The thing about Ted is that not only does he have impeccable technical credentials (Java, .net, Android, etc etc) he is not shy to share his opinion.  His presentations are quite an experience -- just be prepared for a side-bar or two on unrelated topics :-)

Here's hoping more women join the IT workforce.

Can't Git enough

Couple of weeks ago I attended the January offering of GitHub's online Git class taught by Matthew McCullough. I had been using Git for a little over a month and was starting to plateau off. I grokked the basics (at least so I thought) and stayed within the lines. The timing of the course was perfect for me. It helped solidify some concepts, explain the git idiom and presented several aha! moments. It was the kind of course that leaves you energized and exhausted at the same time. Energized because you learned something that you could apply immediately (like during the lunch break :-) and exhausted because you were facing a fire-hose of information for 7 hours!

The course material was well organized starting with just enough background information used to set the stage. Demonstrating Git's pedigree (been around for over 10 years) in the SCM space lent credence to those not impressed with its slavish following in the industry. The material was in resonance with what Matthew was saying -- amplifying his point where necessary, enumerating steps, screenshots where appropriate and links for additional information. Another sign of his thoroughness was demonstrating setup in all major platforms and not leaving the student high and dry having to connect the dots. Over the course of 7 hours (Six 50-min sessions with an hour for break) the class covered everything from setup to 3-stage thinking, branching, remote repos, with just the right amount of peek under-the-hood to explain how Git works.

The format of the course was as hands-on as you wanted it to be. The attendees were provided instructions in advance of the class on setting up their local environment. During the course Matthew helped us connect to the GitHub repository he was referencing. This allowed as to follow along with the steps he was doing as he was doing it! If that wasn't your style you could just watch and listen. It was very stimulating and engaging -- especially when several of us were simultaneously pushing our changes to the same remote repository!!

Matthew McCullough did a first rate job of the online class. It was conducted over WebEx and went over without any major hitch. I was impressed with how Matthew kept up with several stimuli vying for his attention. He addressed the questions being posed over WebEx, emails sent to him andeven tweets all the while staying on course! I wonder if he did a git clone matthew before the class! He answered questions and addressed issues seamlessly as if it were part of the course material.

Matthew belongs to a rare breed that is at the locus of knowledge of the subject, desire to teach and thoroughness of preparation. He set the bar very high at other fora at which I have watched him teach. This Git class did not disappoint. He did his homework and came prepared to teach. From attending several software symposia I can tell you that this is a rare trait. Presenters tend to lean heavily on the content. Yes, content is King. However one should know how to deliver it. Think: Otherwise I could just read the book. Matthew's passion not just for Git but for teaching shone through even at hour-7 of the class!

If you are new to Git you could not do any better than having Matthew as your teacher.

A rose by any other name...

Martin Odersky, Lex Spoon, and Bill Venners have collaborated to venture where everyone has gone before. Yes, I mean the age old story about right way to override the equals method in Java. I see you rolling your eyes with the "I know, I know I must override hashcode method as well" bored refrain.

Spare me a few more minutes and I will convince you that you must give the article a serious once over. I for one got a better appreciation of the complexity of the problem -- if you want it addressed fully.

The article warms up around the 3^rd of the four pitfalls they identify

1. Defining equals with the wrong signature
2. Changing equals without also changing hashCode
3. Defining equals in terms of mutable fields
4. Failing to define equals as an equivalence relation

Pitfall #3 discusses what happens when the state of an object that has been placed in a collection is modified. Typical equalsmethod implementation will result in it disappearing from the collection.

  public class Point {
    private int x, y;
    public Point(int i, int j) { x = i; y = j; }
    public int getX() { return x; }
    public void setX(int i) { x = i; }
    public int getY() { return y; }
    public void setY(int i) { y = i; }
    @Override public int hashCode() { return 41 * (41 + getX()) + getY(); }
    @Override public boolean equals(Object other) {
        boolean result = false;
        if (other instanceof Point) {
            Point that = (Point) other;
            result = (this.getX() == that.getX() && this.getY() == that.getY());
        }
        return result;
    }

}
..
HashSet hashSet = new HashSet();
Point p = new Point(1, 2);
hashSet.add(p);
p.setX(12);
System.out.println(hashSet.contains(p)); // <-- Prints false!!

Yep! A forehead smacking moment alright. The article suggests a way around this problem. Which leads us to the 4^th pitfall -- that of equivalence. This means that for non-null values, the following is true about the equals method:

• Reflexivity: x.equals(x)
• Symmetry: x.equals(y) => y.equals(x)
• Transitivity:x.equals(y) and y.equals(z) => x.equals(z)
• Consistence: x.equals(y) always returns the same result
• x.equals(null) is always false

With clear and concise examples, the article explains the complications of these contracts and how to implement them.

Run a single JUnit test method in Eclipse

Goes to show you, that you can teach an old dog new tricks. After years of using Eclipse I stumbled across a feature I am sure has existed since the glory days of Visual Age.

I had added yet another test method to a large JUnit TestCase class. I wanted to just run and re-run the new method 'til I had it just right without having to run all the other methods as well.

Yes of course, I could comment out all the other methods or just remove the @Test annotation. Back in the JUnit 3 days we have all -- at one time or another -- disabled tests by changing the "test" prefix of the test methods.

Somehow today, all these seemed abhorrent.

I wondered what would happen if I simply highlighted the method I wanted to test, and selected Run As > JUnit Test from the context menu.

Voila! That was the ticket!! It ran just the test I had wanted run.

As a bonus it even added TestCase.method_name as a "Run Configuration" for ease of future use.

FTW!

Mockito: Most helpful stacktrace

One of the things I have admired about the Spring framework is its helpful error messages. Yeah, of all the things I could point to, I picked that....However, work with me here. What I mean is that when things go wrong when using Spring, it spits out error messages that are full sentences that make sense and point the developer in the proper direction. None of the NoClassDefFoundError but not telling who which freaking class.

Now, I think Mockito -- the Java mocking framework -- has taken over the mantle of "King of error messages". Earlier today I had a new unit test I had just written fail. I didn't have to dig too much to find the error because Mockito pointed the way with the following stacktrace.

org.mockito.exceptions.misusing.UnfinishedStubbingException: 
Unifinished stubbing detected!
E.g. toReturn() may be missing.
Examples of correct stubbing:
    when(mock.isOk()).thenReturn(true);
    when(mock.isOk()).thenThrow(exception);
    doThrow(exception).when(mock).someVoidMethod();
Also make sure the method is not final - you cannot stub final methods.
 at
etc. etc. 

How cool is that! Hats off, Mockito-team. This developer appreciates the extra few minutes you took to put together that message. It helped me immensely.

Tab navigating the Mac UI

It was driving me nuts when I got my MacBook that I could not use the tab button to navigate across a form. Let alone forms, on a browser page, I could only tab between the address bar and the search field. To go anywhere else, I had to reach for the mouse. Aaargh!!

Surely the operating system that was God's gift to humanity could do better than that! It took a fair amount of Googling for me to find the obscure setting that controls it.

Here's what you need to do in order to be able to tab away across the entire screen.

Bring up the System Preferences window and select Keyboard & Mouse from the Hardware Category.



From the "Keyboard & Mouse" panel, select the "Keyboard Shortcuts" tab. The magical setting is at the bottom of that tab. Allow "Full keyboard access" by selecting the "All controls" radio button.



As this panel shows, you can impress your colleagues by toggling this setting using CTRL+Fn+F7.

The question remains, who in their right mind thought that the default setting was to be "Text boxes and lists only"!!