Acceptance Tests Trump Unit Tests

At work, we have been practising something approximating Acceptance Test Driven Development now for several months. This means that pretty much every feature of the system that a user would expect to be there, has an automated test to ensure that it really is.

It has given me a whole new perspective on the value of tests as artefacts produced by a project.

I made a pledge to myself when I started this new job in August that I would not (knowingly) check in a single line of code that wasn’t driven out by a failing test. At the time, I thought this would always mean a failing unit test, but I’m starting to see that this isn’t always necessary, or in fact even wise.

Don’t get me wrong. Unit testing is extremely important, and there’s no doubt that practising TDD helps you to write well-structured, low-defect code in an really satisfying manner. But I do feel like the extent to which TDD, at the level of unit testing alone, allows for subsequent changes to the behaviour of the code, has been oversold.

If you think you’re doing TDD, and you’re only writing unit tests, I think you’re doing it wrong.

As new requirements come in, the tensions influencing the design of the code shift. Refactoring eases these tensions, but by definition means that the design has to change. This almost certainly means that some, often significant, portion of the unit tests around that area of the code will have to change too.

I struggled with this for a long time. I had worked hard on those tests, for one thing, and was intuitively resistant to letting go of them. More than that, I knew that somewhere in there, they were testing behaviour that I wanted to preserve: if I threw them out, how would I know it still worked?

Yet those old unit tests were so coupled to the old design that I wanted to change…

Gulliver

In my mind, I have started to picture the tests we write to drive out a system like little strings, each one pulling at the code in a slightly different direction. The sum total of these tensions is, hopefully, the system we want right now.

While these strings are useful to make sure the code doesn’t fall loose and do something unexpected, they can sometimes mean that the code, like Gulliver in the picture above, is to restrained and inflexible to change.

The promise of writing automated tests up front is regression confidence: if every change to the system is covered by a test, then it’s impossible to accidentally reverse that change without being alerted by a failing test. Yet how often do unit tests really give us regression alerts, compared to the number of times they whinge an whine when we simply refactor the design without altering the behaviour at all? Worse still, how often do they fail to let us know when the mocks or stubs for one unit fail to accurately simulate the actual behaviour of that unit?

Enter acceptance tests.

By working at a higher level, acceptance tests give you a number of advantages over unit tests:

  • You get a much larger level of coverage per test
  • You get more space within which to refactor
  • You will test through layers to ensure they integrate correctly
  • They remain valuable even as underlying implementation technology changes

Admittedly, the larger level of coverage per test has a downside: When you get a regression failure, the signpost to the point of failure isn’t as clear. This is where unit tests come in: if you haven’t written any at all yet, you can use something like the saff squeeze to isolate the fault and cover it with a new test.

They’re also much slower to run, which can be important when you’re iterating quickly over changes to a specific part of the system.

To be clear, I’m not advocating that you stop unit testing altogether. I do feel there’s a better balance to strike, though, than forcing yourself to get 100% coverage from unit tests alone. They’re not always the most appropriate tool for the job.

To go back to the metaphor of the pulling strings, I think of acceptance tests as sturdy ropes, anchoring the system to the real world. While sometimes the little strings will need to be cut in order to facilitate a refactoring, the acceptance tests live on.

The main thing is to have the assurance that if you accidentally regress the behaviour of the system, something will let you know. As long as every change you make is driven out by some kind of automated test, be it at the system level or the unit level, I think you’re on the right track.

Agile / Lean Software Development

Comments (2)

Permalink

DRY up your Cucumber Steps

Update (13th Jume 2012): This is an old, old post that still gets a lot of hits. I don’t recommend this practice anymore. Instead, I recommend composing Ruby methods that carry out these actions. For more details, please see The Cucumber Book.

A while back, I asked the Cucumber team for the ability to call in the steps of one scenario from another.

The canonical example of this is the ‘log in’ scenario:

Scenario: User logs in
  Given there is a User whose username is "matt"
  And I follow "log in"
  And I enter "matt" in "username"
  And I enter the User's password in "password"
  And I press "Log In"
  Then I should be logged in
  And I should see the text "Hello matt"

Phew. Now obviously I don’t want all this noise in the scenario every time I specify behaviour that requires a logged in user. I want to write something like this:

Scenario: User views their dashboard
  Given I am logged in
  And I follow the "dashboard" link
  Then I should see "This is your dashboard"

Thanks to the fabulous creativity of the Cucumber community, this is now possible. It’s also highly recommended, as it’s a great way to help you keep your step files tidy and DRY of excess duplication.

Given /I am logged in/ do
  steps %{
    Given there is a User
    When I follow "log in"
    And I enter "#{User.first.username}" in "username"
    And I enter "#{User.first.password}" in "password"
    And I press "Log In"
  }
end

I’m doing this more and more now – writing simple ‘building block’ steps and assembling them to make steps that read nicely and make sense to the stakeholders.

Agile / Lean Software Development

Comments (13)

Permalink

“Total Programming” and the XP Team

Pair programming brings a great many benefits to a team that’s truly mastered it.

Those of us who are lucky enough to have experienced working on a really effective XP team know about that almost magical thing that starts to happen when the barriers between different members of the team break down, egos and code ownership are cast aside, and the team starts to evolve the codebase as one, mighty, coding machine. It’s truly a joy to be a part of, or even just to watch.

dutch-team-78

Continue Reading »

Agile / Lean Software Development

Comments (3)

Permalink

Your Private Methods are None of my Business

A common sloppy mistake I see programmers making is to forget to declare a method as private when it’s only being used inside a class. This seems to happen more in Ruby, or maybe it just bothers me more because we don’t have interfaces. Rails ActiveRecord validation callbacks are a classic example where people often forget to do this.

Continue Reading »

Agile / Lean Software Development

Comments (8)

Permalink

Scrabbling up the Learning Curve

A few months ago I was at the peak of my powers.

I was leading a team of ten C# developers building a huge project on Microsoft’s .NET platform. I had been working on the Windows platform for years, and there was rarely a question from the team I couldn’t answer quickly and confidently, drawing on a deep well of past experience.

Continue Reading »

Agile / Lean Software Development

Comments (1)

Permalink

Come to CITCON

Some people think there is no conference for those of us who care about CI and testing, but oh yes there is.

As an avid reader of this blog, I know that you, like me, realise that continuous integration and testing are to software development what the spirit level and the plumb-line are to the construction industry: powerful tools that will one day be regarded as essential for any professional practitioner.
plumb-line
If you fancy meeting other like minds, come and join me at CITCON, the Continuous Integration and Testing Conference. What could be finer?

Agile / Lean Software Development

Comments (0)

Permalink

Story Driven Development – Just Another *DD?

Bryan Helmkamp, who maintains the handy little library webrat, did a talk recently at GoRuCo 2008 which explains his experiences using RSpec plain-text stories to build ruby-on-rails applications in a manner he calls ‘Story Driven Development’:

Before code is written, the team produces executable scenarios for a user story.

Continue Reading »

Agile / Lean Software Development

Comments (3)

Permalink

9 Techniques to Save you from the Bug-Swamp

How many unresolved bugs are there in your system right now?

Yes, yes, smarty-pants: the ones you already know about.

Ten? Fifty? Two hundred? More?!

I find it frighteningly common to see teams let a huge backlog of bugs build up. They set up a trusted system like Jira or Bugzilla and then use it as a dumping bucket for a whole variety of irritations, glitches, missing features, and genuine defects.

Pretty soon the list becomes un-managably large. Nobody dares go in there, because it’s full of bad news. This drains the team’s energy and morale.

Getting on top of, and staying on top of your bug list is hugely important, and I outline my thoughts and experiences on achieving that below.

Continue Reading »

Agile / Lean Software Development

Comments (1)

Permalink

Software as an Art Form

I just came across an article written five years ago by Richard Gabriel proposing a university course run along the lines of his own Master’s in Fine Arts in poetry. The idea evidently gathered some momentum at the time, but now seems to have come to a halt.

What a shame. I’ve had an idea to get back to university on a ‘some day’ list for a while, and this would have been pretty much exactly my thing, I think. Seems to me like it was just an idea ahead of its time… I wonder how popular something like that would be now, with the increased focus on Quality that is sneaking in the back door behind the noisy razzmatazz of agile adoption.

Uncategorized

Comments (0)

Permalink

Casting with ‘as’ in C#

Anyone else hate this?

Doofer doofer = GetWidget(widgetId) as Doofer;
Then, somewhere miles away from this innocuous little line, I find that my doofer is unexpectedly null. This is a great way to introduce hard-to-trace bugs, unless of course it’s complimented with guard clauses everywhere to test for nulls, which most of the time means unneccesary clutter.

What’s wrong with good old…

Doofer doofer = (Doofer)GetWidget(widgetId);
?

In this second case, if the result of GetWidget() can’t be turned into a Doofer, I’ll get an exception thrown right there and then. Assuming that my code here works specifically on doofers, I may as well spit out an exception right up here – how the heck am I supposed to work with this other type of widget you’ve sent me?

I think the only time to use a soft ‘as’ cast is in the tiny minority of cases where you really don’t care if the result of GetWidget can be turned into a Doofer. It seems to me that 90% of the time, you really do care, and you’d like to know as soon as it’s not. Even in that 10%, you’d probably be better off using a null object.

It’s a shame, because the ‘as’ operator is a bit more readable, and we all hate having to slam all those brackets in everywhere. Fortunately, C# provides a little-known solution to this, known as implicit casting.

Uncategorized

Comments (4)

Permalink