At last month’s CukeUp conference, I held a panel discussion between Aslak Hellesoy, Julien Biezemans, Oriol Gual and Jonas Nicklas. I chose these panelists because each of them has written a variation on the original Ruby Cucumber, and I wanted to try to pull these ideas together into a vision for Cucumber 2.0.
This post is a record of that conversation.
Why do we need a new Cucumber?
The current version of Cucumber has been around since 2008, and as Jonas has previously explained the code isn’t much fun to hack on. It’s quite bloated with features, some of which were added as experiments and maybe aren’t used by most users. Carrying this cruft around is holding us back from adding new features that we do want.
Also, the current mechanism for mapping from steps to Ruby code is pretty basic, and doesn’t really encourage the good practices we’ve learned as a community about organising your test code.
What is your vision?
I asked each of the participants what they’d learned from their fork of Cucumber, and what they’d like to see in a new version of Cucumber.
Aslak made three points:
- It should be possible to use Cucumber as a library, so that other tools can be built on top of it as well as the command-line interface.
- The way steps are defined in Cucumber-JVM, by simply annotating existing methods, allows you to focus on keeping your test code well organised. We should steal this idea for Cucumber-Ruby.
- There should be a collaborative web UI for writing Cucumber scenarios with non-technical stakeholders.
Julien added:
- Integrating the Cucumber-HTML project, which is essentially a web-based client for Cucumber, would be awesome.
- It would be nice to use the Cucumber-TCK, which is growing into a standard set of specs for any Cucumber implementation.
- He hasn’t implemented calling steps from step defs in Cucumber-JS, and he’s not sure he wants to.
- Parallel execution – can we make this possible?
Oriol echoed Aslak’s thoughts about keeping test code better organised: Spinach uses one class per feature, and is generally more object-oriented unlike Cucumber-Ruby’s flat anonymous blocks for step definitions.
Jonas said his goal for Turnip was mainly simplification:
- He thinks that the reason some people had started to abandon Cucumber was to do with it being perceived as complex.
- Jumping from plain english steps to Regexps is jarring, and too big a leap in abstraction. 98% of the time you don’t need regular expressions anyway. Turnip uses the concept of placeholders in strings instead to capture arguments.
- Using RSpec as the runner for Turnip hasn’t worked out so well.
What features should we remove?
It’s always fun making a ‘not’ list, so we brainstormed the current Cucumber features we’d like to kill in a 2.0 release:
In case you can’t read my scrawl, here’s that list again:
- Wire protocol
- Calling steps from step defs
- Tables (we’re talking about scenario outlines here)
- DRB / Spork
- World having everything mixed in, all the time
- Cucumber.yml
- Given / When / Then being regarded as the same by Cucumber
As you can see, we got the audience to come up and vote after the session. It looks like nested steps are living on borrowed time!
My view is that with a well-designed core, many of these features can still exist but as plug-ins, thereby allowing people who need them to continue to use them, but without the maintenance burden on the core team.
So what’s next?
Well, I think I have a pretty clear vision for what Cucumber 2.0 would look like. I’d like to start with a simple core library that has no GUI component at all, but can pass the TCK tests. Then we can start building tools around that core, like a console runner, and a websocket server for running cukes from a javascript browser app. I’m personally really excited about where that last idea could lead.
What I’m short on is time. Would it be crazy to ask if there are any companies who would consider sponsoring this project? I can’t imagine a better way to spend the summer than hacking out a brand new shiny Cucumber!
Update: Thanks for the comments, but I don’t find they make a great medium for discussion so I don’t tend to reply to them. If you’d like to discuss this post, please chip in on these threads on the cukes google group.
WRT “As you can see, we got the audience to come up and vote after the session. It looks like nested steps are living on borrowed time!”
Did you have any testers or business analysts in the audience?
I fear for cucumber becoming a programmer’s dream, but just another opaque tool being forced on others. Allowing (but not requiring) steps to be composed of smaller steps allows non-programmers to see and understand the process of decomposition.
Granted, there are difficulties right now with diagnosing errors in those nested steps. I’d rather see those difficulties addressed than to scrub Cucumber to a more “pure” implementation from a programmer’s point of view.
Hi Matt,
With regard to features to be removed:
I know several teams that use:
1) Calling Steps from Step Defs
2) Scenario Outlines
For 1) In the prose above there is mention of the big abstraction gap between steps and step defs. It will be interesting to see how this can be improved and at what level re-use is possible and encouraged
For 2) Scenario Outlines exist for the reason of not having the same scenarios repeated with different parameters. This can indeed be removed from cucumber if a gherkin compiler / pre-processor that expands scenario outlines is provided in its place?
Cheers
Pete
The team I work with wrote a cucumber implementation for node that includes a web front end.
Hey Matt,
we’re using Cucumber to develop and maintain a large (~1-2 MLOC) consumer application written in C++.
We depend on the wire protocol as an interface into our application. We like having ruby as an intermediate layer for step definitions. Our strategy is to implement ‘atomic’ user operation as ruby steps, and then combine these atoms into more complex compound steps to dry scenarios and make them more readable and robust. We also use quite a few scenario outlines.
I can understand your frustration of working on a legacy code base (after all, that’s way I do every single day). And I can totally feel the wish to start over, to do it right this time, to make a clean cut. It’s everything my colleagues and I moan over every day.
The number one reason we love Cucumber so much is that is has made testing our application so dead easy (after overcoming an initial integration cost). I fear that if you decide to take all these features away which we depend on so heavily, following you to Cucumber 2.0 would become too expensive for our team.
We simply don’t have the resources to write or maintain our own C++-port of Cucumber.
I guess my point is that I would like to ask you to not forget about an upgrade path for existing users of Cucumber.
Best wishes,
Tobias
+1 to killing “call steps from other steps”. This drove me crazy for years (even after I realized that the TextMate bundle had some navigation support). In combination with the general awfulness of step definitions, I eventually wrote https://github.com/jwilger/kookaburra in self-defense.
@George Dinwiddie: have you seen testers and BAs actually working on implementing step definitions? If so, have you seen it not turn into a steaming pile of poo? (I haven’t seen either, but as a consultant, I figure you’ve had much wider exposure.)
As for scenario outlines, I’ve used those multiple times. In retrospect, it’s mostly been in cases where I could’ve gotten away with one “golden path” and one alternate scenario, and the rest of the table exercised at a lower level… but that’s a choice I’d prefer to make for performance reasons, rather than having the framework force it on me. There’s value in having all important examples at the same level of abstraction.
Sam, I’ve seen testers write tests in imperative form, and I’ve helped them refactor that into declarative form using “Extract Step Definition.” It’s a smaller, easier step out of pooville.
BTW, I’ve seen programmers make a steaming pile of poo writing steps in Ruby, so that won’t magically save you. It takes constant vigilance, careful thought, and communication.
As a programmer, I’ve definitely been responsible for some steaming piles of step definitions myself, and certainly didn’t mean to imply otherwise. (=
I was more curious about non-developer participation in test automation. At my last job, I got our POs writing Cucumber and even managing their changes in Git — but definitely wouldn’t have expected them to actually implement.
I like the “Extract Step Definition” refactoring, and can see how it would work well with testers. Thanks!
I no longer use the wire protocol since retiring Cuke4Nuke, but I remember there being several wire protocol-based bridges to other languages. The wire protocol provides a quick way to spin up a new language implementation, so I’d hate to see it go away (unless no one is really using it). If getting Cucumber to a new language requires a full native port, that seems to me to put up an unnecessary barrier.
Scenario outlines can certainly be abused. But I’ve seen many cases where the data varied within the same structure across a set of scenarios. Refactoring to a scenario outline improved readability by putting the focus on the data. To compete with FitNesse, which handles this situation particularly well, I think scenario outlines need to stay.
Why do you want to kill spork/drb support? Is spork falling out of favor in the community in lieu of something else?
You piqued my curiosity with the getting rid of regex. I don’t have much problem with regex because I am try not to write crazy steps. I say “create a user”, not “create a user with email: xx@xx.com and firstname: dd”. People try to stuff too much information into the steps I think. Interesting what the alternative is.
I love using Regex in my steps, mostly so I can have one step cover a bunch of related conditions e.g.
Then /^(?:they|I) (should|should not) see “([^”]*)”$/ do |shouldornot, content|
page.send shouldornot, have_content(content)
end
I guess I could get away with using placeholders and transforms, but I find regular expressions to be useful and not that hard to work with.
I’m also curious about the spork decision. I like using guard and spork to keep things going in the background and not break my flow.
I agree with the nested steps, though. Those get ugly quick. There should still be a nice way to package more granular (I click x, I fill in Y) steps for higher level reuse (I fill out the login form).
I’m having a hard enough time getting my company to use it, that I’ll probably never get them close to sponsoring it, but I am excited about 2.0 and hope to be able to contribute some code.
I think not allowing step defs to call other steps is showing limited vision for how Cucumber can effectively be used.
Even if some peole personally don’t like the practice, there are many of us that use this technique to what we consider good effect.
-1 to killing calling steps from step defs – as I think someone else has mentioned, it allows the tests to be high level and the detail defined elsewhere, whilst keeping the detail visible to anyone that needs to know just what “When I login as a new user” really does.
I would say getting rid of calling steps from other steps is a good direction to go. It creates an inner dependency on step definitions that imo shouldn’t exist. Instead creating easily readable method calls should be the more used practice.