Story Testing

Description

From the ExtremeProgramming? mailing list:

''Years ago (pre-XP), when I was at my best discipline, I'd write all the tests for a part of a system before writing its code. One of the eye-openers for me in learning XP was the way test-driven development is incremental test-first programming. (One lesson of the incremental part, and some hard knocks, is that it gives me the encouragement to keep the discipline.)''

''From the "acceptance" testing point of view, I still find it hard to write in advance a full set of acceptance tests for a feature. I can almost always write some, but when I try to write the full set in advance I find I've left some tests out, and I've gotten some wrong. I'm not a tester by training so you can take it with salt, but I've > reviewed a number of test plans, and I don't think it's just me. ''--Bill Wake

The client I'm currently working with uses IXP and does extensive StoryTesting (IXP's name for customer testing). We use http://fitnesse.org

In the past, I've had trouble getting customers to implement storytests, so in this engagement, I made a point of having storytests written before code. IXP calls this "storytest-driven development" and I like it a lot. It works well.

All of our storytests are written by one of the OnsiteCustomer?''''''s. In the past, it's usually been our domain expert. At the beginning of each iteration, she sits down, sometimes with a programmer, and sketches out a simple Fit table that shows the kinds of tests she's going to write. Programmers take that initial table, automate it, and start working on the story while our domain expert continues fleshing out more of the document.

The tests are written from her viewpoint as a domain expert, not from the programmers'. She focuses on the essence of the story she's documenting: that is, what are the important parts of the story. Sometimes that's UI. Sometimes it's end-to-end behavior. Sometimes it's obscure domain rules. When it's the latter, her tests don't mention the UI. Even when they do, the fixture operates on the "application model," just barely below the actual UI.

Because she's an expert, I don't think she's ever had any serious problems figuring out which tests to write. She doesn't worry about coverage; she worries about describing how the story works. The Fit documents are our executable requirements.

We have another customer on the team who's a tester. She works with our domain expert, and after her, to add more storytests so that she's comfortable with the coverage of the tests.

This approach has worked well and it's provided me with a fair share of learning opportunities. In the beginning, we didn't enforce the "write a storytest before writing code" and we had problems with customers not writing storytests.

We also had trouble figuring out when to transition from writing storytest code to writing unit tests. That resulted in too few unit tests being written at first. We learned that the best way to work is to write the fixture right away, but only enough so that it fails without throwing exceptions. As soon as we do, we switch over to test-driven development, avoiding the temptation to make the fixture work right away. In the beginning, we would start writing code to make the fixture pass right in the fixture itself, then factor it out. That's a neat approach to storytest-driven development--very natural feeling--but it made the decision about switching to unittest-driven development too hard. There was too much temptation to just rely on storytests rather than writing all the unit tests you need.

I really like using Fit for storytest-driven development. It's completely cured one of my biggest gripes with XP: that acceptance tests were too hard for the customer to do. With Fit, customers really can write storytests, and the path for storytest-driven development is nice and clear. --Jim Shore