Tuesday, 13 January 2009

Exploratory v Scripted testing – my understanding evolves further, I think...

On Friday afternoon our team sat down in a meeting room for our regular Friday Forum which is a recurring meeting request with the sole purpose of allowing anyone within the team to use the session to hold a brainstorm, discuss a new idea / technique or give a demo of a new product. The first Friday Forum of the new year saw my manager relay James Whittaker’s Exploratory Tours presentation which he’d given at Eurostar in Nov ’08 which was followed by a discussion on whether this was the kind of thing which our test team might use and whether it’d be of benefit to us.

As we got to the discussion one of our testers raised an interesting question, “How can we ensure we get good coverage when doing exploratory testing?” at which point two or three of our team jumped in with, “Well how can you ensure that with scripted testing?” I think this question highlights the view of ET as ad-hoc, unplanned and which leads to uncertainty over its value.

Until now, and despite reading about this elsewhere, I’ve struggled to explain how I think the two differ, or are similar, in a way which removes the doubt from the doubters. However, I think I’m starting to get a bit of clarity around it…

I think using the term “scripted testing” is misleading; ET can be scripted. The difference is that “scripted” tests are simply written upfront, usually before any testing has happened but exploratory tests are thought-up or created as we test or as our understanding of the program evolves. Surely we’re simply talking about tests which are written prior to the start of the test cycle and tests which are written after the start of the test cycle? The cause of concern by the tester mentioned above was that usually, pre-designed tests are written against requirements & design documents where as ET (in his mind) is performed by exploring the program without the use of such material. I feel this is a misunderstanding and occurs because appropriate time isn’t allowed for ET during test cycles. Where has anyone written that you can’t use these documents during an ET session?

So I’m now in a position where I’ve seen why this tester has this view that it’s difficult to understand what coverage you’re getting when performing ET because you’re not using requirements or design documents as a basis for your tests.

I tried to think of another way in which we could define ET but it’s difficult. I’d prefer to say pre-design tests versus concurrent testing, if “testing” encapsulates the learning, designing & executing of tests. I’ve read about this before, I understand it (I think), but it wasn’t until that tester asked that question that I seemed to get a bit more clarity and perspective on what I think ET is all about.

From an internal perspective I think we need a shift in how we think about these. The decision our test leads need to make is; “are we writing our tests upfront?” or “are we going to write our tests as we test?”. Both will use exploration, both will use documentation, both will result in some form of script.


Philk said...

" The decision our test leads need to make is; “are we writing our tests upfront?” or “are we going to write our tests as we test?”. "

Why not both approaches ? Why limit yourself to one ?

In your reading did you come across this old blog post from Bj Rollings - Exploratory testing vs. Scripted testing; Is it really only either or?

Simon Godfrey said...

Hi Phil,

Maybe I didn't phrase that correctly. My meaning was that they shouldn't think in terms of "Am I doing scripted or exploratory?" more, "Am I designing my tests now, next month, during testing?". There are a whole bunch of decisions which follow, i.e. how am I testing but I wanted to avoid differentiation between the two.

Rob Lambert said...

Hi Phil,

Good post. I struggled with exploratory testing at first but realised all along that I was performing it, but with scripts. The reason being was that I would write the scripts against the spec at a reasonably high level to avoid having to re-write the steps when screens and designs changed (which they always do). I would then use the script as a 'guide' and just perform exploratory techniques.

The 'guide' is essentially what I set as a charter which guides my path. I then use a tool like James Bachs exploratory tool which allows me to record how long I spent on charter versus other paths I chose because they looked interesting.

This gave me the metrics my bosses needed but gave me the freedom to test.

The main points I looked at were:
The design (even if cleverly done up front) still changed as new information became available and people actually started using the interface.
Changes always happen to functions and features - so why have the overhead of costly scripts to write.
Exploratory does not mean (no documentation), it simply means you write the test cases as you have the information. This means the information is up to date and context driven.
There are some really cool posts from James Bach on the subject. I also stumbled across this excellent posting about up front UI design: http://parlezuml.com/blog/?postid=760

Gives some real clarity to designs and why they change. Perfect for exploratory.