Exploratory v Scripted testing – my understanding evolves further, I think...

On Friday afternoon our team sat down in a meeting room for our regular Friday Forum which is a recurring meeting request with the sole purpose of allowing anyone within the team to use the session to hold a brainstorm, discuss a new idea / technique or give a demo of a new product. The first Friday Forum of the new year saw my manager relay James Whittaker’s Exploratory Tours presentation which he’d given at Eurostar in Nov ’08 which was followed by a discussion on whether this was the kind of thing which our test team might use and whether it’d be of benefit to us.

As we got to the discussion one of our testers raised an interesting question, “How can we ensure we get good coverage when doing exploratory testing?” at which point two or three of our team jumped in with, “Well how can you ensure that with scripted testing?” I think this question highlights the view of ET as ad-hoc, unplanned and which leads to uncertainty over its value.

Until now, and despite reading about this elsewhere, I’ve struggled to explain how I think the two differ, or are similar, in a way which removes the doubt from the doubters. However, I think I’m starting to get a bit of clarity around it…

I think using the term “scripted testing” is misleading; ET can be scripted. The difference is that “scripted” tests are simply written upfront, usually before any testing has happened but exploratory tests are thought-up or created as we test or as our understanding of the program evolves. Surely we’re simply talking about tests which are written prior to the start of the test cycle and tests which are written after the start of the test cycle? The cause of concern by the tester mentioned above was that usually, pre-designed tests are written against requirements & design documents where as ET (in his mind) is performed by exploring the program without the use of such material. I feel this is a misunderstanding and occurs because appropriate time isn’t allowed for ET during test cycles. Where has anyone written that you can’t use these documents during an ET session?

So I’m now in a position where I’ve seen why this tester has this view that it’s difficult to understand what coverage you’re getting when performing ET because you’re not using requirements or design documents as a basis for your tests.

I tried to think of another way in which we could define ET but it’s difficult. I’d prefer to say pre-design tests versus concurrent testing, if “testing” encapsulates the learning, designing & executing of tests. I’ve read about this before, I understand it (I think), but it wasn’t until that tester asked that question that I seemed to get a bit more clarity and perspective on what I think ET is all about.

From an internal perspective I think we need a shift in how we think about these. The decision our test leads need to make is; “are we writing our tests upfront?” or “are we going to write our tests as we test?”. Both will use exploration, both will use documentation, both will result in some form of script.