It has given me a whole new perspective on the value of tests as artefacts produced by a project.

I made a pledge to myself when I started this new job in August that I would not (knowingly) check in a single line of code that wasn’t driven out by a failing test. At the time, I thought this would always mean a failing unit test, but I’m starting to see that this isn’t always necessary, or in fact even wise.

Don’t get me wrong. Unit testing is extremely important, and there’s no doubt that practising TDD helps you to write well-structured, low-defect code in an really satisfying manner. But I do feel like the extent to which TDD, at the level of unit testing alone, allows for subsequent changes to the behaviour of the code, has been oversold.

If you think you’re doing TDD, and you’re only writing unit tests, I think you’re doing it wrong.

As new requirements come in, the tensions influencing the design of the code shift. Refactoring eases these tensions, but by definition means that the design has to change. This almost certainly means that some, often significant, portion of the unit tests around that area of the code will have to change too.

I struggled with this for a long time. I had worked hard on those tests, for one thing, and was intuitively resistant to letting go of them. More than that, I knew that somewhere in there, they were testing behaviour that I wanted to preserve: if I threw them out, how would I know it still worked?

Yet those old unit tests were so coupled to the old design that I wanted to change…

In my mind, I have started to picture the tests we write to drive out a system like little strings, each one pulling at the code in a slightly different direction. The sum total of these tensions is, hopefully, the system we want right now.

While these strings are useful to make sure the code doesn’t fall loose and do something unexpected, they can sometimes mean that the code, like Gulliver in the picture above, is to restrained and inflexible to change.

The promise of writing automated tests up front is regression confidence: if every change to the system is covered by a test, then it’s impossible to accidentally reverse that change without being alerted by a failing test. Yet how often do unit tests really give us regression alerts, compared to the number of times they whinge an whine when we simply refactor the design without altering the behaviour at all? Worse still, how often do they fail to let us know when the mocks or stubs for one unit fail to accurately simulate the actual behaviour of that unit?

Enter acceptance tests.

By working at a higher level, acceptance tests give you a number of advantages over unit tests:

• You get a much larger level of coverage per test
• You get more space within which to refactor
• You will test through layers to ensure they integrate correctly
• They remain valuable even as underlying implementation technology changes

Admittedly, the larger level of coverage per test has a downside: When you get a regression failure, the signpost to the point of failure isn’t as clear. This is where unit tests come in: if you haven’t written any at all yet, you can use something like the saff squeeze to isolate the fault and cover it with a new test.

They’re also much slower to run, which can be important when you’re iterating quickly over changes to a specific part of the system.

To be clear, I’m not advocating that you stop unit testing altogether. I do feel there’s a better balance to strike, though, than forcing yourself to get 100% coverage from unit tests alone. They’re not always the most appropriate tool for the job.

To go back to the metaphor of the pulling strings, I think of acceptance tests as sturdy ropes, anchoring the system to the real world. While sometimes the little strings will need to be cut in order to facilitate a refactoring, the acceptance tests live on.

The main thing is to have the assurance that if you accidentally regress the behaviour of the system, something will let you know. As long as every change you make is driven out by some kind of automated test, be it at the system level or the unit level, I think you’re on the right track.

A while back, I asked the Cucumber team for the ability to call in the steps of one scenario from another.

The canonical example of this is the ‘log in’ scenario:

Scenario: User logs in
  Given there is a User whose username is "matt"
  And I follow "log in"
  And I enter "matt" in "username"
  And I enter the User's password in "password"
  And I press "Log In"
  Then I should be logged in
  And I should see the text "Hello matt"

Phew. Now obviously I don’t want all this noise in the scenario every time I specify behaviour that requires a logged in user. I want to write something like this:

Scenario: User views their dashboard
  Given I am logged in
  And I follow the "dashboard" link
  Then I should see "This is your dashboard"

Thanks to the fabulous creativity of the Cucumber community, this is now possible. It’s also highly recommended, as it’s a great way to help you keep your step files tidy and DRY of excess duplication.

Given /I am logged in/ do
  steps %{
    Given there is a User
    When I follow "log in"
    And I enter "#{User.first.username}" in "username"
    And I enter "#{User.first.password}" in "password"
    And I press "Log In"
  }
end

I’m doing this more and more now – writing simple ‘building block’ steps and assembling them to make steps that read nicely and make sense to the stakeholders.

Those of us who are lucky enough to have experienced working on a really effective XP team know about that almost magical thing that starts to happen when the barriers between different members of the team break down, egos and code ownership are cast aside, and the team starts to evolve the codebase as one, mighty, coding machine. It’s truly a joy to be a part of, or even just to watch.

When I worked at the BBC, we tried various experiments with different styles of pairing and retrospected on them regularly to figure out what was working. We went from the extreme of the same pair working together right through a whole story, sometimes staying together for a whole iteration, to switching up pairs several times a day. In the end, of course, we found a sweet spot somewhere in the middle, and seemed to know intuitively when a pair was going stale and needed a fresh injection of energy by changing up the members.

Compare this ideal with this description of “Total Football” (that’s ‘Soccer’ for those of you speak US English!)

“Total Football” is the label for an influential theory of tactical association football in which any player can take over the role of any other player in the team. It was pioneered by Dutch football club Ajax Amsterdam.
In Total Football, a player who moves out of his position is replaced by another from his team, thus retaining the team’s intended organizational structure. In this fluid system, no player > is fixed in his nominal role; anyone can be successively an attacker, a midfielder and a defender.

Regularly rotated pairs of programmers are essential in developing this kind of fluid self-organisation within a development team. While specialisation may become important for larger teams, there’s no doubt that keeping the whole team familiar with the whole codebase for as long as possible brings huge benefits.

For me, the biggest, and often the least mentioned of these benefits is that the team develops a kind of shared consciousness about the code and how they conventionally like to work with it. Not only does this mean the team can move extremely quickly, knowing exactly where to find something and how best to hack on it, but it is a stake through duplication’s dark and twisted heart: “Didn’t we work on something a bit like this last week, Brett?”. Once a team starts to think and communicate together this effectively, code re-use goes through the roof as common problems are solved in consistent ways, patterns emerge and are factored out for re-use.

You can probably tell, I’m a massive fan of pair programming, mainly because I absolutely love being this effective in my day-to-day work. How about you, loyal reader? What are your experiences with pair programming? Have you tried it? How was it for you?

It’s trivially easy in Ruby to call the private methods of another object if you really want to. There’s no intimidating ugly barbed-wire fence around them like there is in a language like C#, but there’s a fence nonetheless. And that fence is there not, as in other languages to aggressively prevent people from doing the wrong thing, but to politely suggest that they should not.

So polite. So very Ruby. Why would the author of a class want to make such a suggestion? Because that private method is part of the internal workings of the class, and they want to retain the right to change those internal workings without upsetting anyone. It’s all about encapsulation, and that encapsulation becomes extremely important as soon as you want to change that class.

Every time you code an object, you’re designing an API. The smaller that API, the safer it will be for someone to change that class. When I’m working in a class with a lot of public methods, I feel exposed, naked. The class is wandering around with its belly hanging out: “Look at my hairy gut! Give us a tickle!”.

When I want to change a class, the first thing I worry about is ‘how many classes will be affected if I make this change’? By marking a method as private, you are clearly telling me that I’m safe – that I’m inside the encapsulation boundary of the class and can refactor with impunity.

So in Ruby our classes may not wear iron-clad chastity belts like the statically-typed languages, but that doesn’t mean they shouldn’t preserve some modesty and keep themselves covered up as much as possible. Or am I just an old-fashioned prude?

I was leading a team of ten C# developers building a huge project on Microsoft’s .NET platform. I had been working on the Windows platform for years, and there was rarely a question from the team I couldn’t answer quickly and confidently, drawing on a deep well of past experience.

When doing TDD it goes without saying that you write the spec or test before you write the implementation. When that implementation will involve an interaction with some dependency, you’ll need, as part of your spec, to precisely describe the way that interaction will happen in terms of mock object expectations, or some other sensing on a fake of the dependency.

In order to do this quickly and easily, you’ll need to

• Understand the API for the dependency (or at least the subset you’ll use)
• Know how to express the interaction with your dependency in a mocking framework.

As I start to learn ruby and the rails framework, I’m quickly realising how much harder it is to do TDD when you don’t have a clue what you’re doing! Or more precisely, you don’t know the system APIs well enough to confidently say how you’ll want to call them to get the behaviour you want, without a little prior experimentation. And even when you do, you may not know your local neighbourhood mocking framework well enough yet to ask him to do some more involved stuff, like validating arguments sent to mocked out methods.

When you hit that ‘how do I…’ barrier, it’s tempting, often even necessary, to break out of the TDD rhythm and do some ‘scratch coding’ as Michael Feathers would call it.

Just make sure you roll-back your working files before you write that first test, cowboy!

As an avid reader of this blog, I know that you, like me, realise that continuous integration and testing are to software development what the spirit level and the plumb-line are to the construction industry: powerful tools that will one day be regarded as essential for any professional practitioner.


If you fancy meeting other like minds, come and join me at CITCON, the Continuous Integration and Testing Conference. What could be finer?

Before code is written, the team produces executable scenarios for a user story.

The talk is very much grounded on the assumption that you’re working within the rails framework, but it clearly illustrates that driving your application through automated acceptance tests that can easily be understood by your customer / product owner / business analyst / stakeholder (delete as applicable in your world) and easily maintained by you is very possible with this technology. Even if you’re working in another framework, it should give you plenty of ideas about how you’re approaching your delivery of user stories.

Personally I think we may end up calling this ‘Scenario Driven Development’ – a phrase I first heard Jason Gorman use – as it seems to me that it’s the scenarios that you build within a story that really help you to shape the software.

Find the slides here.

And watch the talk itself here.

Yes, yes, smarty-pants: the ones you already know about.

Ten? Fifty? Two hundred? More?!

I find it frighteningly common to see teams let a huge backlog of bugs build up. They set up a trusted system like Jira or Bugzilla and then use it as a dumping bucket for a whole variety of irritations, glitches, missing features, and genuine defects.

Pretty soon the list becomes un-managably large. Nobody dares go in there, because it’s full of bad news. This drains the team’s energy and morale.

Getting on top of, and staying on top of your bug list is hugely important, and I outline my thoughts and experiences on achieving that below.

Draw Some Axes

There’s nothing like getting your bugs out of the tracker system and up onto the wall on a big, colourful, chart to really ram it into people’s faces that something needs to be done.

It’s also a terrific motivator as you start to get on top of the bug list and are rewarded with real visible progress on the wall.

A great simple tool for this is the bug-down ‘finger chart’. Grap your flip-chart or whiteboard pens, and let’s get stuck in.

• Draw an x-axis scale plotting days from today.
• Draw a y-axis scale going up to, say, 1.5 x the total number of bugs your tracker system knows about – open or closed.
• Mark a point for today at the total number of bugs, open or closed.
• Now beneath that, mark a point at the total number of closed bugs.

That gap between the two points is what you need to work on.

The beauty of this simple chart is that it shows you:

• How quickly you’re finding new bugs
• Whether you’re winning the war (is the top line going up steeper than the bottom line?)

Plot these points every day, and rotate the task so that everyone is in touch with the situation.

Turn Bugs into Features To Do

As James Shore says, many items in your bug tracker are probably features in disguise. If an application is in production, it’s hopefully usable in it’s current state, even if it’s doesn’t do everything you wanted it to do in every single scenario.

See how many of your bugs you can re-write like this:

BUG: Users without javascript can’t sign-up. The sign-up form uses AJAX and won’t work for users with javascript disabled.

STORY: As a user with javascript disabled, I want to be able to sign up for the site, so that I can use it like everyone else.

or, perhaps more cheaply:

STORY: As a user with javascript disabled, I expect to see a helpful message telling me that I need to enable javascript to use the site, so that I don’t feel excluded and confused.

Bingo! one less bug to fix. This feature or user story can now be prioritised and planned alongside all the other changes you’d like to make to the software.

Alternatively, take a buggy part of the product and write a story that says something like: As a user in the upload photo page, I expect the site to be bug-free. Now carry all the bugs relating to the photo upload with that story card, prioritise it, and when it comes out of development it should be shiny, gleaming and bug-free.

Bug Monday

Fixing bugs is demoralising. It’s no coincidence that they can usually be found in the grottiest bits of the code, where misunderstandings and mistaken assumptions abound. It’s no fun hanging out there. It’s hard work and requires a lot of energy. Some people find it easier than others.

We found that fixing bugs as a team effort really made a difference for us. The old adage ‘many hands make light work’ really comes into its own here, and by all mucking in together, the team can generate a bug-quashing momentum that no one or two individuals could compete with.

Much better to put the whole team of five onto bugs for a single, horrible day, than to force one poor bastard to do it for the whole week.

Talking of which…

Take, Don’t Give

Assigning bugs to particular people is very close to assigning blame. Anyone and everyone in the team should feel a collective responsibility to defeating and controlling the bug count, and hopefully most members of the team will feel confident enough to dive in to any area of the code to make a fix.

Having said that…

Specialize

Although I’m big on the XP principle of ‘no code ownership’, I’m also a pragmatist. If you wrote the WidgetFragulator last week while I was on holiday, and there’s a bug in the WidgetFragulator, it’s probably going to be quicker for you to fix it. It might, on the other hand, offer a great opportunity for me to get my head around the new code you wrote. Make a quick judgement within the team about what will give the most value, balancing long-term knowledge-sharing within the team against the short-term business benefit of getting the fix done ASAP.

Write a Test

If you’ve got a defect in your code, it’s usually for one of two reasons:

1. The guy who wrote it didn’t know what the software was supposed to do. What he implemented was correct to his understanding of what it was supposed to do, but his understanding was wrong.
2. The guy who wrote it knew what the software was supposed to do, but he made a mistake.

In the first instance, you may well be lucky enough to have an existing unit test that you can modify to specify the correct behaviour. In the second instance you certainly won’t.

By writing a unit test to reproduce the bug, you get to do something mildly creative, you save time tesing out different options for the fix, and you make sure that this particular bug will never go undetected again.

Aim for Zero Defects

David Allen’s excellent GTD methodology promotes a two minute rule: if it will only take two minutes, do it now!

Encouraging your team to take a similar attitude to defects saves everyone a huge amount of time. Rather than wasting effort finding defects, logging them in a tracker, reviewing the lists, prioritising them, re-reading them weeks later to figure out what the hell they mean… just fix it! Now!

This also promotes a culture within the team where everybody cares. They care about the quality of the product, and if they see a typo, they’ll take the time to stop and fix it before moving on.

Keep Shorter Lists

While David Allen’s excellent GTD methodology teaches us that having a trusted system where you can dump everything out of your head will elevate your mind and allow you to concentrate on the tasks at hand, lean thinking teaches us that maintaining long lists of anything is inventory, and inventory is waste.

These two guiding principles apparently contradict.

It’s easy to see how wasteful it is to keep a large system full of bugs: many of the entires may be out of date, for issues that have already been fixed, and each time you open one up you’ll have to waste 30 seconds figuring out if the bug is still relevant. Whatever you do, don’t let your bug-tracker become one great big ‘hmph’ pile for all the things your team can’t make a decision about.

For every bug in there, ask yourselves these quesions:

• Is this really a defect, or can we call it a new feature and prioritise it?
• If it’s a defect, why don’t we fix it this week?
• If we don’t need to fix it this week, when should we fix it by? Does it even matter if we never fix it?

Create Less Bugs

Finally, a really obvious point but one worth ramming home nonetheless. Using quality-focussed software engineering techniques like Test-Driven Deveopment, pair programming and refactoring will help you to quash bugs at source: when the code is written.

What a shame. I’ve had an idea to get back to university on a ‘some day’ list for a while, and this would have been pretty much exactly my thing, I think. Seems to me like it was just an idea ahead of its time… I wonder how popular something like that would be now, with the increased focus on Quality that is sneaking in the back door behind the noisy razzmatazz of agile adoption.

Doofer doofer = GetWidget(widgetId) as Doofer;

Then, somewhere miles away from this innocuous little line, I find that my doofer is unexpectedly null. This is a great way to introduce hard-to-trace bugs, unless of course it’s complimented with guard clauses everywhere to test for nulls, which most of the time means unneccesary clutter.

What’s wrong with good old…

Doofer doofer = (Doofer)GetWidget(widgetId);

?

In this second case, if the result of GetWidget() canâ€t be turned into a Doofer, Iâ€ll get an exception thrown right there and then. Assuming that my code here works specifically on doofers, I may as well spit out an exception right up here – how the heck am I supposed to work with this other type of widget you’ve sent me?

I think the only time to use a soft ‘as’ cast is in the tiny minority of cases where you really donâ€t care if the result of GetWidget can be turned into a Doofer. It seems to me that 90% of the time, you really do care, and youâ€d like to know as soon as itâ€s not. Even in that 10%, you’d probably be better off using a null object.

It’s a shame, because the ‘as’ operator is a bit more readable, and we all hate having to slam all those brackets in everywhere. Fortunately, C# provides a little-known solution to this, known as implicit casting.