Using Cucumber for Load Testing

I sometimes get asked whether it’s possible to use Cucumber to test performance. The way to do it is to specify concrete examples of scenarios that the system will find itself in under stress. For example:

Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms

or

Given 1000 users are hitting the homepage simultaneously
Then each user should get a response within 2ms

Talking through these kinds of scenarios with your stakeholders will help you to understand where the boundary is for what they consider to be acceptable performance. You might find it hard to get them to be this specific at first, but help them understand that what you’re doing is drawing a line in the sand for the minimum acceptable performance – most of the time the application may be much faster than this. Now you have agreement about this, the next step is to work out how to automate these scenarios, by calling your load testing tool of choice from the Cucumber step definitions.

The key thing is to have Cucumber delegate to the stress testing tool, rather than the other way around. A common mistake people make is to simply point JMeter at existing Cucumber scenarios, but this doesn’t give you the benefit of having the parameters of the performance test documented in readable Cucumber scenarios.

These are not normal kinds of Cucumber tests. You would need to run these against an environment that’s representative of your production hardware, whereas normal behaviour scenarios could be run against any environment. It’s useful to create these scenarios early on during the project and run them against every build so they become constraints that every build must meet as the project progresses.

Published by Matt

I write software, and love learning how to do it even better.

Join the Conversation

8 Comments

  1. “The key thing is to have Cucumber delegate to the stress testing tool.”

    Hi Matt,

    Can you suggest some stress testing tools which Cucumber can delegate to?

    Thanks

  2. My team has started on bdd, and i am looking into a way of having cucumber rule all our tests.
    This brought me to a search for how to make cucumber drive performance tests.

    I read your blog entry, and my first thought was wow … i hurried to the pragmatic programmer website and bought the beta book (allready happily owning the cucumber book) … just to discover the current version doesnt mention anything about performance tests … now I feel slightly disappointed …

    So, could you give any hints as for how to make cucumber drive a performance test … like jmeter ?

    Agata

  3. Considering cucumber seems targeted at functional testing, would you let cucumber to drive the load test (entirely) or get a load generator to drive cucumber tests?

  4. @Derek, Cucumber is targeted at business-readable behaviour specification. So-called ‘non-functional’ requirements like performance can often be described using examples like this. Writing them down helps force the team to make decisions about what they consider acceptable behaviour.

    If you have a load testing tool delegating to Cucumber scenarios that just walk the app through basic behaviour, you’d lose that opportunity to discuss the performance constraints. You could still have the conversation of course, but your parameters would be buried away in the scripts for the load generator.

  5. My team is looking for a functional testing tool to overcome a backlog of testing needs and to use during Sprints. Cucumber would be my tool of choice if we can also use it to drive load tests. In essence, we would like the ability to run a Scenario N times using M threads to barrage the server to see how it holds up under load (e.g. “Scenario: Place order for new part” runs 100 times using a pool of 40 threads). Is something like this possible with Cucumber?

    Matt if I understand your suggestion above, I would use Cucumber to describe the test, setup fixtures, etc, and then have a step def that calls funkload. Funkload would do it’s thing, report results, and Cucumber would evaluate those results. The trouble I’m having with this is I seem to end up writing the same test twice—once as a Scenario and once as a funkload test as funkload doesn’t have access to my step defs, etc. Has anyone accomplished this and has an example they can share?

    Many thanks.

  6. @Lance: one option is to reuse your existing Cucumber scenarios, and parallelize their execution using a virtualization system. We use a Proxmox container to virtualize many small clients which execute predefined sets of scenarios.

Leave a comment

Your email address will not be published. Required fields are marked *