Exploratory v Scripted testing – my understanding evolves further, I think...

On Friday afternoon our team sat down in a meeting room for our regular Friday Forum which is a recurring meeting request with the sole purpose of allowing anyone within the team to use the session to hold a brainstorm, discuss a new idea / technique or give a demo of a new product. The first Friday Forum of the new year saw my manager relay James Whittaker’s Exploratory Tours presentation which he’d given at Eurostar in Nov ’08 which was followed by a discussion on whether this was the kind of thing which our test team might use and whether it’d be of benefit to us.

As we got to the discussion one of our testers raised an interesting question, “How can we ensure we get good coverage when doing exploratory testing?” at which point two or three of our team jumped in with, “Well how can you ensure that with scripted testing?” I think this question highlights the view of ET as ad-hoc, unplanned and which leads to uncertainty over its value.

Until now, and despite reading about this elsewhere, I’ve struggled to explain how I think the two differ, or are similar, in a way which removes the doubt from the doubters. However, I think I’m starting to get a bit of clarity around it…

I think using the term “scripted testing” is misleading; ET can be scripted. The difference is that “scripted” tests are simply written upfront, usually before any testing has happened but exploratory tests are thought-up or created as we test or as our understanding of the program evolves. Surely we’re simply talking about tests which are written prior to the start of the test cycle and tests which are written after the start of the test cycle? The cause of concern by the tester mentioned above was that usually, pre-designed tests are written against requirements & design documents where as ET (in his mind) is performed by exploring the program without the use of such material. I feel this is a misunderstanding and occurs because appropriate time isn’t allowed for ET during test cycles. Where has anyone written that you can’t use these documents during an ET session?

So I’m now in a position where I’ve seen why this tester has this view that it’s difficult to understand what coverage you’re getting when performing ET because you’re not using requirements or design documents as a basis for your tests.

I tried to think of another way in which we could define ET but it’s difficult. I’d prefer to say pre-design tests versus concurrent testing, if “testing” encapsulates the learning, designing & executing of tests. I’ve read about this before, I understand it (I think), but it wasn’t until that tester asked that question that I seemed to get a bit more clarity and perspective on what I think ET is all about.

From an internal perspective I think we need a shift in how we think about these. The decision our test leads need to make is; “are we writing our tests upfront?” or “are we going to write our tests as we test?”. Both will use exploration, both will use documentation, both will result in some form of script.

Testing: It’s largely about the mindset, isn’t it?

I was sat on the train to the recent SIGIST and I got thinking about what a colleague had told me about James Whittaker’s talk at Eurostar on the topic of Exploratory Testing and how Microsoft are trialling this Exploratory Tour idea. The idea, I think, is fantastic and I understand that James is writing a book which will cover the tours in detail and so, rather than talking about these I want to use this blog to look at how I took the concept, and applied it to the nearest thing I had that very morning when travelling to SIGIST, the train…

To recap, James Whittaker of Microsoft looked at using tour analogies to focus their Exploratory Testing and having reflected on this idea it seems to be very much about the mindset with which the tester adopts when designing or executing tests and ultimately, it doesn’t matter whether he or she is doing ET, some form of prescriptive test or an automated test – the mindset is still important and can be consistent across all forms of testing.

So, how can the mindset, or approach to some testing be linked directly with trains? Well, here’s what I came up with (and this was at 7.15am so I make no apologies if it’s complete tosh):

• The fast train tour: Take the quickest / shortest route through the program
• The slow train tour: Stop and observe (test?) each point in the program
• The Bullet Train tour: Review each function for performance; how quickly does this program perform?
• The Flying Scotsman tour: See the heuristic oracle, “consistent with previous product”
• The Rush hour tour: Apply lots of data / input at every opportunity (stress / load testing?)
• The Viaduct tour: High level view of the system, what might a customer’s first impressions of this product be?
• The Underground tour: Look at the program from a lower level; focus on data, the code…
• The Rotate train tour (I don't know what these are called!): Ensure a consistent change of direction, maybe through testing navigation, whilst testing the program
• The Signal Point tour: Test for correct instruction / messaging from the application
• The Impatient Commuter tour: test for any unexpected delays in operation, possibly whilst performing the fast train tour
• The Packed Station Tour: Get lots of users using the program, how does it react? How good is the information provided to different users? How do they perceive the usefulness of the information provided?

Irrespective of the use of the tours I believe this would aid consideration for different types of tests, again, different mindsets required to test an application. They’re almost heuristics, I guess…
Can you think of any more? How can you use this type of thinking to inspire thought around tests?

Final note; this is an expansion of James Whittaker / Microsoft’s idea and I merely wish to expand, not take credit, for this type of thinking…