Book now

Cucumber 1.2.2 Released

This is a maintenance release, but marks a new period in Cucumber’s life as it was released by our new team member Oleg Sukhodolsky. Oleg has been doing a fantastic job since he joined the team a few weeks ago, closing tickets like a boss.

Here’s a summary of what’s in the release:

New Features

  • Ruby 2.0.0 support (#377 Matt Wynne & #357 @charliesome)
  • Capture duration value for json formatter (#329 Rick Beyer)
  • Added support for Hindi (hi), although some systems may need to install fonts which support the Devanagari script.
  • Obey program suffix when invoking bundler (#324 Eric Hodel)

Bugfixes

  • Fix class loading problems –format option had (#345, #346 @ksylvest)
  • Exit with failure status when interrupted (#299 @aaronjensen)
  • Cannot map table headers after table hashes is referenced (#275 @chrisbloom7 / Matt Wynne)
  • (before|after)_step aren’t called when scenario outline’s table is processed (#284 Oleg Sukhodolsky)
  • Raise exception when remote socket end disconnects using wire protocol (#348 @rdammkoehler)
  • Fix –dry-run option ignored when set via profile (#248 / #255 Igor Afonov)
  • More clear suggested ruby code for undefined steps (#328 / #331 @martco)
  • Fix exception in Html formatter with –expand mode and undefined steps (#336 Roberto Decurnex)
  • Fix Table.diff! problem with :surplus_row => false and interleaved surplus rows (#220)

BDD

Comments (0)

Permalink

TDD vs BDD

I regularly find myself explaining to people the difference between TDD (Test-Driven Development) and BDD (Behaviour-Driven Development). There still seems to be a lot of confusion over this, so I wanted to write this up for reference.

Late last year I was interviewed for a virtual panel on InfoQ along with Dan, Gojko, and Liz. Probably the most interesting part of that conversation covered the difference between TDD and BDD. Or rather the lack of any great difference.

We’ll start with some snippets from that discussion.

Both TDD and BDD include acceptance testing

One common misconception is that TDD is what you do when you’re unit-testing, and BDD is what you do when you’re writing customer-facing acceptance tests. Here’s Dan North on that point:

TDD – as originally described – is also about the behaviour of entire systems. Kent [Beck] specifically describes it as operating on multiple levels of abstraction, not just “down in the code”. BDD is equally important in this space, because describing the behaviour of systems is fractal: you can describe different granularities of behaviour from the entire application right down to individual small components, classes or functions.

Extreme Programming has always talked about writing acceptance tests, sometimes also called functional tests to describe what the customer expects to be done at the end of an iteration.

So this is nothing new. What’s new is how we explain it, and therefore how successful teams end up being in making it work for them.

BDD describes TDD done well

When Dan was working as a coach teaching TDD, he found that it was easier to get people to understand the principles of TDD if he stopped using the word ‘test’:

My experiences as a coach told me people were missing the point, with all this talk of unit tests, acceptance tests, functional tests, integration tests… Kent Beck’s style of TDD is a very smart way to develop software, so I tried removing the word “test” when I was coaching it, replacing it with things like behaviour, examples, scenarios etc. The result was very encouraging: People seemed to “get” TDD much quicker when I avoided referring to testing.

When Aslak and I wrote the Cucumber Book, I wrote this description of BDD:

BDD builds upon TDD by formalising the good habits of the best TDD practitioners.

That’s basically all there is to it. We want to re-explain TDD in a way that highlights the habits that successful TDD practitioners having been using for over a decade.

So what are those good habits?

Specifically, I think those good habits are:

  1. Working outside-in, starting from a business or organisational goal
  2. Using examples to clarify requirements
  3. Developing and using a ubiquitous language

Working outside-in seems obvious to habitual TDD practitioners, but many teams seem to limit themselves to doing this at the level of small units of code. Business-level black-box testing is still done manually, or automated as a check after the code has already been implemented.

This misses out of the major benefit of working outside-in, which is having the requirement challenged: if you need to explain to a computer how to check the requirement, you’ll need to be damn sure understand it yourself. If you don’t (and you often don’t) it’s much cheaper to find that out before you write the code.

Examples have always been a great way to make sure you really understand a requirement. What BDD does is formalise this by encouraging you to use scenarios to describe behaviour. These examples provide the perfect bridge between the business-facing and technology-facing sides of a team: they’re just formal enough that you can get a computer to check them, but anyone on the team can read them and make sure they’re describing behaviour that they actually want.

The GOOS Book, written by two of the best TDD practitioners in the business, frequently highlights the importance of domain language in our programs. In software teams, communication is probably the biggest overhead you have, and you make that communication a lot harder when you allow different dialects of terminology to be used by different parts of the team. Developing and then sticking to a consistent language takes deliberate effort, but it’s something that the best TDD practitioners have long learned will give them a significant advantage.

My experience is that BDD’s emphasis on collaboration, and the use of business-readable, executable specifications, means that this shared language develops much more quickly. When everyone is involved in writing documentation that describes what the system should do, they all get a chance to learn the language of the domain together.

So BDD really isn’t all that different to TDD. What BDD adds is a clear emphasis on what it takes to make TDD succeed.

BDD

Comments (5)

Permalink

Is Cucumber just a scam?

David Heinemeier Hansson recently wrote on his blog:

Don’t use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you’re there!)

Well, good news readers! The magic kingdom is real! I’ve been there! Look, I even have a bottle of fairy dust. I keep it right next to the teabags:

Fairy Dust

Admittedly, that fairy dust is pretty good stuff, and I’ve been hitting it hard lately. Maybe it’s been making me hallucinate?

I decided I’d better check with some other people I know who’ve been to the magic kingdom too.

Lisa Clark told me a wonderful story about how her team use BDD:

I live in a magic kingdom and work in a castle (the Rackspace Castle). I’ve been using Cucumber on a RESTful web service project since project inception over a year ago… Currently our BDD sessions include the BA, QA, Dev Lead, and Developer. We will pull in the PO or architects as needed.

Here’s how Lisa’s team have benefitted from using BDD:

We’ve found value in the BDD documentation process and obtaining a shared understanding of what we’re building before it’s actually built. The added benefit of having executable requirements and an automated functional test suite that’s ready when the code is ready is icing.

Hear that readers? Lisa doesn’t see the primary benefit of Cucumber as being the testing, it’s the shared understanding that the team have built from writing the tests together. In Lisa’s magic kingdom, non-programmers don’t sit about writing tests on their own, they collaborate. And look the confidence that the whole team gets from that shared understanding:

The developers on my team have a strong level of confidence when delivering a story that all scenarios have been coded and are working as we said they should be. The QA knows up front exactly what we’re delivering with the story. I have confidence that regardless of which developer owned the story, all expected scenarios are coded and tested.

Of course there are other ways of getting that shared understanding and confidence. Working in small, cross-functional teams helps, and keeping the team together for a long time so that everyone becomes a domain expert is sadly under-appreciated by most big companies. Getting around a whiteboard or a set of design mock-ups to talk through a new feature is also invaluable, and some people find this is enough for them.

It really depends on the complexity of your domain: teams that work with complex, poorly understood business rules and requirements need all the tools they can get their hands on to manage that complexity. Many people I know have found this is the key benefit of using Cucumber: in a strange new domain, having a place to write down what you’re learning can really help you to stay sane.

I’ve never spoken to him about it, but my guess is that this is the reason DHH doesn’t see the need for Cucumber: he works on a team of good communicators who already have a wealth of domain expertise. In that context I might not use Cucumber either.

What I thought was most interesting about Lisa’s story was what happened when the team were under pressure and decided to throw off their BDD shackles:

We realized the value of our process when we bypassed it in order to quickly deliver a number of features for a high profile effort. We absorbed a new BA, QA team, and Devs that were unfamiliar with BDD, had tight timelines, and hoped to quickly knock out features. These features have had a higher number of defects, did not meet their delivery timelines, have a lack of automated testing (from both the developer and QA test fronts), and general hesitation from developers in touching this code in fear of breaking something. After our experience with these non-BDD implemented features the team (with the support of management) has committed to full BDD for all new features.

Higher defects, missed deadlines, hesitation from developers to touch the code in case they break something… Does that sound familiar to you?

It certainly doesn’t sound like much fun. Pass the fairy dust.

BDD

Comments (11)

Permalink

Interview on QCon: BDD, Cucumber and Hexagonal Architectures

I was interviewed recently by QCon’s Peter Bell about various subjects dear to my heart: BDD, Cucumber, Relish and hexagonal architectures.

Click here to watch the video

Agile / Lean Software Development
BDD
Hexagonal Rails
Relish

Comments (0)

Permalink

Interviewed by RubySource

Last week I had a great chat with Pat Shaughnessy about The Cucumber Book which has been published as an interview on RubySource. Pat really managed to get to the heart of my opinions about how to use Cucumber effectively, so if you’re too busy to read the book, this will give you a good summary of my latest thinking on topics such as:

  • - why Cucumber is pointless unless you involve the whole team
  • - imperative vs declarative style, and why it matters
  • - testing domain objects directly from Cucumber
If you’re using Cucumber or SpecFlow day to day, or considering them and wondering what benefits you’ll get, I think this interview is worth a read.

Agile / Lean Software Development
BDD
Relish

Comments (1)

Permalink

Using Cucumber for Load Testing

I sometimes get asked whether it’s possible to use Cucumber to test performance. The way to do it is to specify concrete examples of scenarios that the system will find itself in under stress. For example:

Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms
or
Given 1000 users are hitting the homepage simultaneously
Then each user should get a response within 2ms
Talking through these kinds of scenarios with your stakeholders will help you to understand where the boundary is for what they consider to be acceptable performance. You might find it hard to get them to be this specific at first, but help them understand that what you’re doing is drawing a line in the sand for the minimum acceptable performance – most of the time the application may be much faster than this. Now you have agreement about this, the next step is to work out how to automate these scenarios, by calling your load testing tool of choice from the Cucumber step definitions.

The key thing is to have Cucumber delegate to the stress testing tool, rather than the other way around. A common mistake people make is to simply point JMeter at existing Cucumber scenarios, but this doesn’t give you the benefit of having the parameters of the performance test documented in readable Cucumber scenarios.

These are not normal kinds of Cucumber tests. You would need to run these against an environment that’s representative of your production hardware, whereas normal behaviour scenarios could be run against any environment. It’s useful to create these scenarios early on during the project and run them against every build so they become constraints that every build must meet as the project progresses.

Agile / Lean Software Development
BDD

Comments (8)

Permalink

Skillsmatter BDD Exchange

Last week I travelled down to London to the BDD Exchange conference. It was a one-day conference organised by Gojko Adzic and I had a great time. I missed Gojko’s talk as I travelled down from my cave in Scotland on the day, but I did arrive in time to see Chris Matt’s excellent lecture on what business analysis really should be about.

I particularly enjoyed the talk from Christian Hassa about teams failing to make BDD work. We can learn the most from failure, and Christian’s thoughtful analysis of what he’s observed in the field as a consultant with TechTalk is useful to any team trying to get the most from these techniques. The message of Christian’s talk very much echoed my own, that the tooling you use is entirely secondary to the collaborative relationship you need to build between the business and technical-facing members of the team. I was interested to learn about the tool, SpecLog, TechTalk are building to help teams with this problem, which seems to have many similar goals to my own Relish. It was nice of Christian to give Relish a name-check in his talk.

My session ran along the same theme as my talk from earlier in the year at Skillsmatter, describing the value of writing acceptance tests at the right level of abstraction, so that they describe business rules rather than implementation details. You can watch the session here.

Agile / Lean Software Development

Comments (2)

Permalink

Relish Roadmap

I want to give you some news about the future of Relish. A lot has happened since we first started the project just over a year ago. Justin quit the project to concentrate on his new role on the RSpec core team, The Cucumber Book went into beta, and my wife gave birth to our second child. All of those things have meant that Relish hasn’t been able to progress as quickly as I’d have ideally liked, so thanks for sticking with us!

I’m still passionate about the vision for Relish, and as I teach training courses on using Cucumber and BDD, it seems to resonate with a lot of people. With The Cucumber Book pretty much behind us now, Relish will come back into focus and the pace will pick up again. I hope to have it launched by the end of the first quarter of 2012.

Here’s a rough outline of what we plan to do:

Usability & Information Architecture Enhancements

I have an IA specialist on the team now, and you may have noticed the first tweaks we’ve been making this week. There’s a lot to catch up, so we’ll focus on this until the existing features are polished to his (and hopefully your) satisfaction. Please let us know how we’re doing as we progress, and let us know if any of the changes annoy or delight you.

If there’s anything specific you’d like to see done that isn’t already logged in UserVoice, please add it there or reply to this post.

Test Results

Relish can’t deliver on the promise of living documentation until the non-technical readers can tell, when they’re looking at a scenario, whether it’s passing or not. We’ll be building a plugin for Cucumber that allows developers and testers to send their test results to Relish, so each scenario can be rendered with a big green tick if it’s passing. This will also enable us to start building all kinds of exciting dashboards for project managers to get a high-level overview of what’s going on.

Commenting / Feedback

Just being able to read features is not enough. I want Relish to be a collaboration tool, and that means that stakeholders should be able to comment on and give feedback about the documentation they’re reading.

RSS / Activity Feeds

People who want to stay up to date with changes to a project will be able to get a nice high-level summary via RSS, and possibly other means too.

Enterprise Install

As the codebase stabilises, I’m becoming increasingly comfortable with the idea of a stand-alone installer for customers who are not happy about having their features stored outside of their firewall. In the new year, I’ll be looking for a couple of friendly enterprise customers to help me shape this. If you’re interested, please get in touch.

Plans & Pricing

Yes, it had to happen eventually :)

When we come out of private beta (which is still a few months away), it will be with paid accounts for private projects. This is a hard call to make, and I’ll be in working with our beta tester community to get an idea of what they feel is a reasonable price to pay for the service. Public accounts like RSpec and VCR will continue to be free.

As always, I’m keen to hear what you think of these plans. Let me know in the comments.

Agile / Lean Software Development

Comments (0)

Permalink

Fine-Slicing Beats Estimation for Predictability

As requested by JB in the comments to my previous post, here is some data about what happens when a team choose fine-slicing over estimation.

You’re about to see a CFD chart drawn by a team who used BDD to break down every requirement into scenarios before they started hacking on them. The items on the left aren’t actually scenarios in this case, they’re very small user stories which tended to be of a size of about 3-4 scenarios each. The point is, we broke everything down into the smallest pieces of behaviour we could, then re-assembled them into chunks that were meaningful enough to build together.

Rather than using story points to manage the variation in size of stories, we gave each story a value of one point, and used BDD analysis to try to ensure each story was a uniformly small size.

This data was collected over a period of about six months by a team of about eight developers. Their system (a high-volume website) was already live and they were adding features to it.

What strikes me the most is how straight the ‘done’ line is.

Who Needs Estimates

Agile / Lean Software Development

Comments (0)

Permalink

Using BDD Scenarios to Track Project Velocity

Before you write any code, start by brainstorming all the scenarios you’ll need to cover to make the story done. Do this collaboratively with everyone (devs, testers, UX, business people, product owner) who is interested in the story. Don’t try to make them valid Cucumber scenarios, just make a list of them on a whiteboard, index cards, or in a text file.

Now look at all the scenarios you have. Does the product owner really want you to build the product to satisfy all of them? Can you cut any out and defer them as another story to build later? Can you drop any of them altogether? Get rid of as many as you can until the story is as small as you can make it.

Now count how many scenarios you have left, and write that number on the story card. At the end of the iteration, count up how many scenarios you’re managed to deliver in total across all the stories you’ve done, and start using that as your velocity metric. It’s much, much more accurate in my experience than estimated story points. What’s more, the process of exploring the scenarios means you can agree a clear scope for the story before you get started.

Teams who are doing this well are getting things done much more quickly than they did before. Not only do they build a suite of automated regression tests, but they waste a lot less time writing the wrong code because of misunderstood requirements.

Agile / Lean Software Development

Comments (5)

Permalink