Is Cucumber just a scam?

David Heinemeier Hansson recently wrote on his blog:

Don’t use Cucumber unless you live in the magic kingdom of
non-programmers-writing-tests (and send me a bottle of fairy dust if you’re there!)

Well, good news readers! The magic kingdom is real! I’ve been there! Look, I even have a bottle of fairy dust. I keep it right next to the teabags:

Fairy Dust

Admittedly, that fairy dust is pretty good stuff, and I’ve been hitting it hard lately. Maybe it’s been making me hallucinate?

I decided I’d better check with some other people I know who’ve been to the magic kingdom too.

Lisa Clark told me a wonderful story about how her team use BDD:

I live in a magic kingdom and work in a castle (the Rackspace Castle). I’ve been using
Cucumber on a RESTful web service project since project inception over a year ago…
Currently our BDD sessions include the BA, QA, Dev Lead, and Developer. We will pull in
the PO or architects as needed.

Here’s how Lisa’s team have benefitted from using BDD:

We’ve found value in the BDD documentation process and obtaining a shared understanding of
what we’re building before it’s actually built. The added benefit of having executable
requirements and an automated functional test suite that’s ready when the code is ready is
icing.

Hear that readers? Lisa doesn’t see the primary benefit of Cucumber as being the testing, it’s the shared understanding that the team have built from writing the tests together. In Lisa’s magic kingdom, non-programmers don’t sit about writing tests on their own, they collaborate. And look the confidence that the whole team gets from that shared understanding:

The developers on my team have a strong level of confidence when delivering a
story that all scenarios have been coded and are working as we said they should be. The
QA knows up front exactly what we’re delivering with the story. I have confidence that
regardless of which developer owned the story, all expected scenarios are coded and
tested.

Of course there are other ways of getting that shared understanding and confidence. Working in small, cross-functional teams helps, and keeping the team together for a long time so that everyone becomes a domain expert is sadly under-appreciated by most big companies. Getting around a whiteboard or a set of design mock-ups to talk through a new feature is also invaluable, and some people find this is enough for them.

It really depends on the complexity of your domain: teams that work with complex, poorly understood business rules and requirements need all the tools they can get their hands on to manage that complexity. Many people I know have found this is the key benefit of using Cucumber: in a strange new domain, having a place to write down what you’re learning can really help you to stay sane.

I’ve never spoken to him about it, but my guess is that this is the reason DHH doesn’t see the need for Cucumber: he works on a team of good communicators who already have a wealth of domain expertise. In that context I might not use Cucumber either.

What I thought was most interesting about Lisa’s story was what happened when the team were under pressure and decided to throw off their BDD shackles:

We realized the value of our process when we bypassed it in order to quickly deliver a
number of features for a high profile effort. We absorbed a new BA, QA team, and Devs
that were unfamiliar with BDD, had tight timelines, and hoped to quickly knock out
features. These features have had a higher number of defects, did not meet their delivery
timelines, have a lack of automated testing (from both the developer and QA test fronts),
and general hesitation from developers in touching this code in fear of breaking
something. After our experience with these non-BDD implemented features the team (with
the support of management) has committed to full BDD for all new features.

Higher defects, missed deadlines, hesitation from developers to touch the code in case they break something… Does that sound familiar to you?

It certainly doesn’t sound like much fun. Pass the fairy dust.

Published by Matt

I write software, and love learning how to do it even better.

Join the Conversation

15 Comments

  1. You make a good point about the value of the team and the customer (who is part of the team, really) collaborating closely on the design. For almost as long as people have been programming electronic computers, using examples to pin down a precise understanding has been a valuable practice for thoese few teams in the know.

    Where I worry is that we confuse the value of examples, expressed in some precise way and some formal language, and the collaborative design process with a specific tool and a specific brand name of what’s essentially the same practice old lags were telling me about in the early 90s.

    We went through the same thing with UML a decade ago. Teams confused the benefit of getting together around a whiteboard and collaborating on a design using a shared, semi-formal visual language with the UML and UML tools. Various quite spurious ROI studies commissioned by some of the tool vendors, and other interested parties, talked of the benefits of UML or of CASE tool X. The real benefits are in using visual models (pictures) in a collaborative design process.

    So I say: using examples to explore and pin down a clear, shared understanding among the team/stakeholders in a highly collaborative – and, we hope, iterative – process can be the key to getting “bridging the communication gap”. But let’s not confuse the mode of transport with the destination 🙂

  2. Well, here follows a success story of a different kind:

    I recently embarked on a new project bringing the ease of use that RVM brings to Ruby developers to the Groovy community. So, GVM was born (http://gvmtool.net)

    From the outset, I chose to collaborate with my fellow developers by fleshing out our features in Cucumber’s Gherkin syntax. We used concise examples of how the tool should function on the command line, then implemented the behaviour as an outcome. All the scenarios for the project so far can be found here for all to see and collaborate on:

    https://github.com/gvmtool/gvm/tree/master/src/test/cucumber/gvm

    At this point the tool is alive, well and growing rapidly. It’s continually improving as new features are added or existing ones challenged. It might not be a big kingdom yet, but it’s magic nonetheless!

  3. For me, the primary value of TDD is that I write less code. I only code for actual expected specs and not layers of abstraction I won’t need, tightly coupled god methods, or tweaks and features nobody actually asked for. The time savings from doing TDD are immense.

    DHH’s points are valid if tests are just a way to check for bugs, but that’s an extremely immature view and use of TDD. That’s like novice-level TDD. TDD is a design tool.

    I use Cucumber and TDD on every project. It saves me a ton of development time, a ton of potential requirements misunderstandings, and a ton of tight coupling. I can’t imagine how slow I’d be without it.

  4. Totally agree with Jason Gorman.

    While “bridging the communication gap” is very important, cucumber is not an ideal mode of transport:

    • Gherkin language is too restrictive. How many times have I found myself correcting ‘incorrect’ BA’s steps? Many, many times. What if they were just writing them in English? One less translation level. Exactly what is needed for better communication;

    • Regex dispatching is a terrible idea. Classic showcase for ‘Have a problem? Use regex. Now you have two problems’

    • The whole ‘reuse step’ idea is flawed. If you want to reuse, the underlying language (ruby, java, etc.) provides much better tools to do that (packages, classes, modules, etc).

    Having said that, I am using cucumber both at work and for personal projects… But I am not happy with it! 🙂

  5. Well, BDD sounds like it could work for some teams, sure. But you lost your way around the end there.

    This example team went from the painful fiddly annoyance that is cucumber (which as you say has some potential benefits that may or may not outweigh that annoyance) to… nothing? No tests at all? No planning, no bug reports, no QA team working to find issues… nothing?

    Sorry to say, but these people are just a crappy team that were using BDD as a crutch to help them not suck. SOME teams are capable of not sucking no matter what process they use. DHH likes a regular, sane “test in a way that gives me confidence in my code base and ability to make future changes” testing methodology. Don’t act like cucumber/BDD is the only way to do this.

  6. And you can achieve that same communication using something like coulda, turnip, or just pending rspec/minitest-spec specs.

    If it can be easily marshaled to plain English, it can be easily discussed.

    Put another way: coding in pseudo-English is a red herring.

  7. Artem, you don’t have to use regexes. You can use strings. I hear that complaint from time to time about cucumber, so it seems that’s not a commonly known thing.

    From https://github.com/cucumber/cucumber/wiki/Step-Definitions “Some people are uncomfortable with Regular Expressions. It’s also possible to define Step Definitions using strings and $variables like this:”

    In my experience, it’s not very important for BAs to write gherkin. It’s not important for them to write the examples down at all. Communicating them to someone who can write them in a form that works is just fine.

    It is, however, important that they can read them. Having someone else have to tell them what the tests mean is a problem.

  8. Thanks for the post, Matt, and the pointer to DHH’s blog. I think he made some very good points in his post, and he declared his troll-like, intentions clearly: “So let me firebomb the debate with the following list of nuance-less opinions.”

    Where troll-like became troll was with point 6, which is the subject of your response: “Don’t use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you’re there!).”

    First, he called out Cucumber, not JBehave, FitNesse or any other tool, but he’s talking about “non-programmers-writing-tests” in general. Troll!

    Second, as George pointed out, it’s not who writes the examples that is important – it’s that there’s an increased level of communication between dev, test & business.

    And finally, I think we do ourselves a dis-service if we accept (even in jest) that there’s pixies, magic or any other flavour of faith involved in developing good software.

    Just say no (to the dust) & brew yourself another cup of tea.

  9. I’ve just come out of a specification workshop with two domain experts and one big data expert. In the workshop we wrote the first scenario for the first feature for a new project and in doing so discovered a whole load of stuff which was hitherto tacit knowledge.

    We now have this knowledge made explicit and stored in svn.

    This stuff is real, and it works!

  10. Great Post. Anything that counters DHH’s ignorance is a good post in my book :).

    I might add that yes BDD can be useless…if #1 is not done below. Devs shouldn’t be the one creating the BDD tests, it should be non-devs. But all too often, reality is that devs do end up doing them, or QA. And the business does not. Which defeats the entire purpose of BDD and its usefulness. The acceptance criteria should be coming down from PMs, other users outside of technology and that’s what the whole point of BDD is. If that’s not happening, it’s a waste of time usually. So a lot of times BDD fails because only devs are creating these tests and not the business. What looks to be “simple” such as assuming that non techies are gonna see these tests as easy to write ends up to them not being that “simple” is what the challenge ends up being most the time.

    Couple comments:

    1) BDD is useless if only the devs are the ones creating the tests. Non Developers have to have a part in it and be the ones actually creating the BDD tests, NOT the developers. Otherwise BDD is a waste of time and money

    2) Having all 3 kinds of tests: Acceptance, Integration, and TDD (non-integration) tests are all valuable. For any developer new to a project or company they are living documentation. I can and should be able to check out the original acceptance tests and get at least a decent idea of the initial intent of the system, and the current state of requirements, and if crap is working or not.

  11. When people compare automated acceptance testing using Cucumber with automated acceptance testing using Capybara or whatever else, it seems to me they are looking at the automated test cases as a way to ensure the product behaves as expected at the time of initial delivery. I rarely see anyone mention the benefit of having an absolutely up-to-date, comprehensive description of the product’s behavior, versioned and maintained alongside the production code.

    If one’s job is to “deliver” a chunk of code, pick up a check, and walk away, then those comparisons are probably right on the mark. If all you care about is correctness at a point in time, after which it will become someone else’s problem, why add another level of complexity to the automated testing stack? Keep it simple. Pick up your check and walk away. Don’t look back.

    If one’s job is to produce /and support/ a product during its production lifetime, then there is value in having a way for people to find out what the hell the product is supposed to be doing, at any time, at the touch of a button, and to specify changes in the product’s behavior in a way that mitigates the risk of regressions.

  12. Is there some organization large/small – that has some success after using ATDD ? not just test automation but ATDD ? Are there any metrics to prove that ? Any improvement in quality ?

Leave a comment

Your email address will not be published. Required fields are marked *