Awesome Acceptance Testing

My notes on DanNorth and JoeWalnes‘ session at Spa 2008.

Five artefacts:

• Automation – the glue that binds the tests to the code
• Vocabulary – the language that the tests are expressed in
• Syntax – the technology that the tests are expressed in (C#, Java)
• Intent – the actual scenario being tested
• Harness – the thing that runs the tests and tells you if they passed

Four roles. People might fill more than one, or more than one person might be in a role:

• Stakeholder
• Analyst
• Tester
• Developer

Taking a requirement, the Stakeholder and the Analyst have a conversation:

• what does that requirement mean?
• how can we create a shared understanding?

Then the Analyst and the Tester have a conversation:

• what is the scope of (‘bigness’) of this requirement?
• how will we know when we’re done?
• => Scenarios (examples)

Tester then ‘monkeyfies’ the scenarios, using the following template:

Given …
– assumptions, context in which the scenario occurs.

When …
– user action, interaction with the system

Then …
– expected outcome

e.g.
Given we have an account holder Joe
and their current account contains $100
and the interest rate is 10%
When Joe withdraws $120
Then Joe’s balance should be $-22

The tester and the developer sit down and write an automated test to implement each scenario.

You might chain these up, but you can always categorise test code into these three partitions. This really helps how you look at test code.

Consistency Validation Between ‘Units’

See the Consumer Driven Contracts paper on Martin Fowler‘s website.

Tooling for Automation

Consider extending / creating the domain model to cover the application itself – the UI, the back end.

Loads of tools are availlable. Use whatever works and build on it.

Building a Vocabulary

Ubiquitous Language – Start with a shared language. It becomes ubiquitous when it appears everywhere – documents, code, databases, conversations.

You will use different vocabularies in different bounded contexts. A context might be your problem domain, testing domain, software domain, or the user interface domain.

Beware which roles understand you when you’re talking in a particular domain. Often terms will span domains.

e.g. NHibernateCustomerRepository
<– 1 –><– 2–><– 3 –>

1 = 3rd Party Provider Domain
2 = Problem Domain
3 – Software Domain

Make your tests tell a story – make it flow. Don’t hide away things in Setup methods that will make the test hard to read. If that means a little bit of duplication, so be it. ‘Damp not DRY’.

Syntax – Implementing Your Tests

• write your own
• keep it simple. don’t fart around writing too fancy a DSL. you’ll be surprised what testers / analysts / stakeholders will be prepared to read.
• great way to learn
• Jbehave2
• training wheels?
• rspec
• very nice.
• create templates for each given / when / then which you can plug together with parameter values into scenarios
• fit
• concordion
• nbehave – joe ocampo

Basically what you need is a way to assemble different permutations and combinations of Given / When / Then with different parameters to make different scenarios.

Expressing Intent

Think in terms of narrative, flow. Think in terms of bounded contexts, and who the audience (role) is for that context. Who will understand that vocabulary?

Make sure the intent is clear – that’s the main thing.

Harness

Do you want to hook into continuous integration build?

Which version of the code is it going to run against?

Keep the tests in two buckets:
* in progress
* done

Those which are in the ‘done’ bucket, should always work, those which are in progress are allowed to be failing, until you make them pass.

Getting Started

Things you can do today.

• Try it for your next requirement
• Given When Then helps guide the tests
• It’s a collaborative process – get people involved
• Works for bug fixes
• a bug is a scenario that you missed in the first place
  use the tools you’re most comfortable with
• doesn’t have to be perfect

Down The Line

What to aim for.

• ALL requirements have acceptance criteria specified up front
• helps with estimation
• acceptance tests are automated where appropriate
• just having thought about it helps – you may come back to automating it later.
• Push button, availlable to all.
• helps build trust with stakeholders

Advice

• Automate pragmatically
• Don’t try to automate what you can’t do manually
• Testing is validating an outcome against intention
• Non functional requirements
• Plan for false positives
• Quality is a variable
• doesn’t mean you don’t go test first
• doesn’t mean low quality code
• does mean how complete is the solution? – how many scenarios / edge cases are you going to try and meet?

Summary

• Have a shared understanding of done
• There is no Golden Hammer
• Be aware of the five aspects of test automation
• Automation, Vocabulary, Syntax, Intent, Harness
• Start simple, then you can benefit now