The diary of a software tester

Breaking the back of automation

2009-05-21T11:12:00.000+01:00

I’ve worked in this test team for almost 8 years now and in that time we’ve had numerous cycles where we’ve tried to make automation work for us but each time we’ve failed for a number of reasons…and now we’re trying again.

Originally we tried with WinRunner, then we tried TestPartner, then we tried QTP and now we’re trying with Selenium. To clarify “automation”, I’m talking about record & playback here as we’ve successfully used code to make other aspects of testing easier but for the purposes of this blog I’m talking about record & playback.

There are numerous reasons why I don’t think we’ve managed to make automation work for us in the past, and these are a few of them:

• Test leads not being aware of the benefits of automation and therefore not thinking about automation early in the product development and ensuring the product’s designed with automation/testability in mind
• Skillset – i.e. testers having the appropriate knowledge, or interest, to do the coding around any R&P
• Interest
• Thinking too big, too soon
• Products not working with automation tools

We’re now coming back round to another automation cycle and this time the weapon of choice is Selenium. Selenium’s a hugely popular tool in the world of automation at the moment and our development team are actually using it to automation their dev tests and they’re building a large range of tests which they run as part of nightly builds.

I’m really hoping we can finally get to a point where our team understand what opportunities a tool like Selenium can offer them and how they can use it to their benefit. I want to see tools being used to automate the mundane, checking/verification type tests that we use people to perform so our testers can concentrate on thinking up complex tests which requires them to spend time gaining a deeper level of understanding about the product – something a tool will never accomplish.

Hopefully I won’t be writing the same blog in 12 months time.

We ain’t got no SMART requirements

2009-05-13T16:56:00.000+01:00

As a manager I write objectives for people in my team and I’m told these objectives have to be SMART; that’s Specific, Measurable, Achievable, Realistic and Timebound. And, as a manager, I have people within my team complaining about a lack of quality requirements across projects – they lack clarity, they don’t detail expectations, they’re ambiguous etc. So why don’t people write SMART requirements?

Requirements should be specific because an ambiguous requirement isn’t going to be a testable requirement and the more open to interpretation a requirement is the poorer it is. More importantly, your interpretation of a requirement may be an incorrect interpretation and that means you could end up testing and providing information which is of no value to your project team.

Requirements should be measurable, and by measurable I mean testable. If you can’t measure whether the requirement’s been met, or whether it performs as expected, it’s not testable.

Requirements should be achievable. Ok, there is scope for highly desirable requirements which may make it in to the product if time allows (ever seen that happen?!) but this should serve as a warning. Does it sound achievable to you? If it doesn’t you probably want to question it before you spend time designing tests for it…

Requirements should be realistic. Well, yes, but sometimes we don’t know until we try to implement a requirement that we can’t actually implement it in the given timescale, or because of timescales, and therefore it’s not realistic. But again, if it doesn’t sound realistic it should probably be questioned before you start spending time designing those tests.

Requirements should be timebound. Think performance. “Requirement 1: Function A must do Y in N seconds” Is that realistic? What if it doesn’t? Would we still release if it took an extra 10 seconds, 30 seconds or 10 minutes? How good are your performance requirements?

The SMART method of reviewing requirements has it’s flaws but it does highlight the need for good requirements. You might use SMART as a guide to review requirements as it’ll raise some questions for you to ask or another method might be to try and write early high level test cases against requirements as a way of teasing out ambiguity or a lack of testability. Whatever your method, it’s key that we understand product requirements and more importantly, it’s key we ensure our understanding is aligned with that of the project team.

Get out of the office and be inspired

2009-05-01T12:03:00.002+01:00

This isn’t particularly pertaining to software testing, but I thought I’d blog about it anyway. Yesterday I went down to London and spent the day at two schools, one primary, on secondary, in one of the more deprived parts of London. Working at an Education company it’s important that the employees who work here take the time to visit schools, our customers, and build an empathy for our end users, something which can be quite difficult when you develop and test market products.

The day in itself was absolutely fascinating. Schools have moved on dramatically since I were there (not that long ago) and it’s great to see students using web-based products to collaborate on assignments with students of other schools around the region, country, or world.

It’s been far to long since I’ve been to a school, probably a couple of years, long enough, but it was an inspiring day and I’ve come back to the office enthused about what we do as a company and how I can play my part in ensuring we meet the demands of our customers.

Similarly, I’ve felt the same when returning from something like SIGIST, or Eurostar. That time away from the office allows you to regain perspective, to think new thoughts, and to be inspired.

So, to my point! Getting out of the office to meet customers, go on a training course or attend work-related conferences is a great way to inspire and energise your self. We can all become tired, or bogged down in our day-to-day work and responsibilities and it’s important to know you don’t always need two weeks in the Maldives, however nice that’d be, to re-charge your batteries!

An Observation on the challenge of being a tester

2009-04-28T09:44:00.000+01:00

Last night, Monday night, was shopping night, so I met my girlfriend in Tesco’s after work and we set about our weekly shop. I noticed recently that Tesco’s have altered the setup of their tills slightly and they’ve changed the arrangement so that customers are now able to scan their clubcards without the cashier having to do it and the card payment device was also moved toward the end of the till, in the area where you place your shopping in to bags – whether this is to make the whole process quicker, or just remove tasks from the cashier I’m not sure.

Anyway, as we went through the tills and completed bagging up our goods I took my Tesco clubcard from my wallet and went to scan it through the clubcard-scanner, which is situated at adult waist height, pointing slightly downward. “You can’t use that”, declared the cashier, “We’re not using them anymore.” “You’ve only just had them installed haven’t you?”, said I. “Yeah, but they realised that the laser beams which scan the barcode can damage children’s eyes so we can’t use them”, “you’d have thought they’d have tested that”, she then added. I wasn’t sure how to reply to that, as a hundred reasons why it might not have been tested flew through my head and I almost told her I was a tester, but then I’d imagine that information would have fallen on deaf ears – what did she care what I did for a living? Instead, I just laughed, thanked her, and wandered off thinking that this was a good topic for a blog post.

So what makes it a good topic for a blog post? Well, for me it highlights the difficulties that we have as testers. On the one hand, it sounds like whoever manufactured and tested this product didn’t consider the types of users that would be using or impacted by this device and for that there is no excuse. But there’s also the consideration that the barcode reader may have been through tens of test cycles, with masses of defects found & fixed, or maybe the test team only had a week to test the damn thing before someone told them it had to be released and so they spent the whole week ensuring it could actually read barcodes and transmit that data to the client, rather than test or understand how dangerous the rays were from the device.

As testers we have an almost impossible job because there will always be untested situations, scenarios or environments at the point of release and we never truly know if those untested areas will be the very areas which throw up a customer critical defect upon release. With risk assessments, and good judgement, we can have a very good go and ensuring we’ve removed most of the risk ahead of release, but there will always be that percentage of risk we release with, and that’s what we’ll always worry about.

Are you doing your bit to save money?

2009-04-24T12:02:00.001+01:00

*I just wrote this as a blog at work, so thought I'd share it*

As we all know, it’s key that in the current climate (not the Sauna!) we ensure we’re doing our bit to reduce spend and monitor cost across projects. So, off the top of my head here are a few things we can be doing to make efficient use of our money.

1. More Exploratory Testing – the savings aren’t clear, but it is evident that the big-upfront-test-scripting approach could be replaced by exploratory testing. There may be a need to invest more time in training for your testers and you may take people out of their comfort zones but choosing to adopt a more explorative approach to your testing could reduce the cost of your test cycles.

2. Write less detailed test scripts – how much detail are you putting in to your scripts? And, more importantly, is that level of detail really required? We know that people who’re new to a product require detailed steps to guide them through the product but as with the exploratory point, could appropriate training replace this? Less detailed scripts could also reduce maintenance effort later on in the project when functions and features change which require script updates and re-writes.

3. Complete clarity on coverage PT1 – are we spending time and money testing on configurations which aren’t high priority? Will testing on those configurations provide useful information? Or is that testing which can wait for another time? Are the project team in agreement with your planned testing scope? Let’s ensure we’re not burning time and money running tests which aren’t producing useful information, and information which is required here and now.

4. Complete clarity on coverage PT2 – how are you ensuring that you’re not going to find showstopper or major defects via new tests in later test cycles? How can you make sure the really important bugs are found as early as possible? Are you talking to the right people to understand what the high risk areas of the product are? Ultimately, are you test the right stuff?

5. Risk Assessments – the primary aim of a tester should be to find important bugs fast and using a risk assessment is the main tool which facilitates this. So, have you performed a risk assessment? Are you testing the most important or complex areas first? How are you going to make sure you don’t come across a showstopper on the final day of your testing?

There are undoubtedly numerous other ways in which we can actively control and reduce cost and it’s also worth considering that whenever a decision is made to cut cost there can be an impact on time and quality. I don’t want to teach my grandma to suck eggs but hopefully this will make you think a little about how you’re doing things, and whether you can be working smarter, or more efficiently, or just making decisions about your approach with one eye on the cost aspect.

Usability: A second class citizen?

2009-04-23T10:58:00.000+01:00

I’m sure we’ve all been in the situation where you get an opportunity to test a product so you sit down and you start testing for functionality, you run some system tests, you run some security tests and maybe a performance test or two before you then run some usability tests. Your defects are raised, most of the functional defects get fixed, the big security and performance issues are fixed but then you look at your usability defects are they’re set as “Fix in the Future” or “Not a problem” or “Proposing not to fix” – Argh! Why is this?

Usually, as we discussed in our team meeting this morning, it’s for two reasons:
1. You’re testing for usability too late in the cycle
2. Time pressures mean the focus is on making the product functional, not usable

Quite often it seems that, what appears to be a small usability issue, actually requires a significant (> 1hour) amount of time to fix because the fix requires some re-working of how the UI works, or it would have a significant impact on other functionality – which tells us we’re testing for usability too late in the software development lifecycle.

Our other battle is members of the project who lack empathy with the end user. “Why on earth would we want to hold up a project to fix these paltry 10 usability defects?” (I recognise that sometimes it’s absolutely the right decision to release with those 10 defects unfixed). So we sit in defect reviews and watch as our usability defects get set to the statuses shown above, usually because we’ve failed to articulate the real end-user impact of those issues.

So, what are we doing about it?

Well, we’re doing a few things about this. We’re trying to use the power of the checklist to create a common set of usability heuristics which can be run against a design doc, a prototype, an early version of the product or a final version of the product. And as that implies, we also recognise that we need to test for usability earlier in the development of the product. It’s clear that we’re testing for usability across most of our products but generally we’re leaving it until too late in the day to do this so it’s key we test for usability as early as we can in the development of a product and continue to focus on usability throughout its creation. Finally, we recognise that we need to get better at influencing our project team on the impact of these defects. We need to be better at understanding our customers and also understanding those people within the business who also understand those customers and who can therefore help us articulate why these issues are important.

Approximately 50-60% of calls our support teams take are “How to” calls which suggests usability of our products is an area in need of improvement so hopefully some of the measures we’re putting in place will start to reduce this in future product releases.

Very interested in hearing other’s experiences with usability testing – have you seen the same problems? If so, what are you doing about it?

Accessibility requirements? Pfft

2009-03-18T16:18:00.003Z

Q. What do you do when you find your product fails to meet the Accessibility standards which the requirements state it must adhere to?

A. Remove the requirement.

Quicker than fixing the bugs ennit?

Yes, this has recently happened.

Testing, what's that??

2009-03-06T15:52:00.004Z

It's been a while since I blogged so to dust off the cobwebs I thought I'd mention what I've been doing for the last 5 weeks. The answer? Not reading much on testing.

It occured to me recently that I spent a lot of time reading and thinking about testing and worried that perhaps I had my balance wrong because that outweighed the time I spent thinking about leadership and management, so I've tilted the balance.

Instead, I've recently been reading blogs on leadership and innovation, and I've been reading books such as "The one minute manager" and "Who moved my cheese?" which are both thought-provoking and inspiring reads. Why? Because I felt the balance wasn't reflective of my daily priorities, and also, whilst testing is a passion of mine, it's perhaps not where my future aspirations lie and so I wanted to sharpen the saw in a different way.

I've found a change in focus quite energising and it's been rewarding to try new ideas out and apply them to day-to-date life as a team leader.

I'll drop some links of what i've been reading if anyone's interested, and I might just have to transform this blog in to a testing / leadership thing, the latter mentioning lightbulb moments, rather than anything ground-breaking!

Back to school

2009-01-27T08:51:00.004Z

Recently I blogged about sharpening the saw where I mentioned that I wanted to start learning to code in VB as it was an area I'd identified as a weakness in my skillset (not the only weakness, I might add...).

Then, on Friday I read this blog (Don't Forget to Test) which reminded me of another concern of mine; I haven't tested in ages. I've been in a management role for 18 months and although I spent 12 of those months still doing some test leading I haven't been doing any testing at all. This worries me; I'm meant to be in a position where I can coach testers about most things testing, but I've not done any testing myself in ages...

I've had this dilemma before and figured it wasn't feasible due to time / other constraints but I've decided to challenge that. I've offered my team the chance to have a free tester, i.e. me (I won't put my time against the project), for a little bit of time each week so help them do some testing. I said I'd be more than happy to run some ET tests which they may be struggling to get the time to run, or I'd happily give a fresh perspective on some aspect of a program - whatever they want really.

We'll see how this goes and I'll come back and blog about it. I've found, so far, the single biggest challenge to team leading is trying to stay up-to-date with the challenges the team face on a day-2-day basis and you can read all the blogs you like but unless you allow yourself a little time to test I'd worry you could end up removed from the realities of life as a tester!

Understanding risks

2009-01-21T13:26:00.001Z

I’ve just been reading a blog from BJ Rollison of Microsoft (http://blogs.msdn.com/imtesty/archive/2009/01/19/the-minefield-myth-part-1.aspx) and it struck a chord with me and some of the concerns I have with regression testing.

When I initially came across the minefield analogy I really liked it, I thought, “That’s what we’re doing. We’re running tons of regression tests and they’re not finding many bugs so why don’t we take more of an exploratory approach to it?” Then I read Bj’s blog and I think, “You’re right, regression testing does have it’s place!” but then I find myself wondering about the age old problem of a test team having enough information to fully understand risks.

So you run a bunch of tests (you wander through the minefield), you find some mines, you miss some mines, the mines you find are removed and the mines you don’t find are still there. Some mines aren’t properly removed and so still exist, probably pretty close to the original mine you found. Someone makes some changes to the mine field, maybe they change the size of it, maybe they change where some of the mines are located or maybe they change what the mines look like? What if you’re asked to scour the minefield and check for mines again but nobody tells you what’s changed; how do you do that?

If you follow the same paths you may find some mines that are missed, if you look closely you may find some of the mines which weren’t removed properly and if you’re lucky you may find some mines which have been introduced when someone made some changes.

Here’s my problem. I don’t believe that as testers we get enough information about what’s changed. We know what defects should have been removed, so we can re-test them. We know where the defects clustered before, so we can run regression tests in those areas but what we don’t get, in my opinion, is enough information about what’s changed between test cycles. You exit a test cycle, you prepare your test environments and you want to re-circulate your strategy, probably based on an updated risk assessment, but you can’t. You can’t because nobody seems to be able to tell you what’s changed – you don’t know how big that minefield now is, you don’t know what the new mines look like and you don’t know how many new mines have been added to the field.

This uncertainty costs testers time, effort and of course money. A lack of information leads to uncertainty which leads to over-testing, often through masses of regression tests, and a wild-goose chase which may not throw up any useful information. Testers strive on information - information about risks, information about requirements and information about what’s changed and until we get better at asking for that information or the holders of that information get more empathetic to our needs we’re always going to be less efficient and less useful than we could be.

Exploratory v Scripted testing – my understanding evolves further, I think...

2009-01-13T09:56:00.000Z

On Friday afternoon our team sat down in a meeting room for our regular Friday Forum which is a recurring meeting request with the sole purpose of allowing anyone within the team to use the session to hold a brainstorm, discuss a new idea / technique or give a demo of a new product. The first Friday Forum of the new year saw my manager relay James Whittaker’s Exploratory Tours presentation which he’d given at Eurostar in Nov ’08 which was followed by a discussion on whether this was the kind of thing which our test team might use and whether it’d be of benefit to us.

As we got to the discussion one of our testers raised an interesting question, “How can we ensure we get good coverage when doing exploratory testing?” at which point two or three of our team jumped in with, “Well how can you ensure that with scripted testing?” I think this question highlights the view of ET as ad-hoc, unplanned and which leads to uncertainty over its value.

Until now, and despite reading about this elsewhere, I’ve struggled to explain how I think the two differ, or are similar, in a way which removes the doubt from the doubters. However, I think I’m starting to get a bit of clarity around it…

I think using the term “scripted testing” is misleading; ET can be scripted. The difference is that “scripted” tests are simply written upfront, usually before any testing has happened but exploratory tests are thought-up or created as we test or as our understanding of the program evolves. Surely we’re simply talking about tests which are written prior to the start of the test cycle and tests which are written after the start of the test cycle? The cause of concern by the tester mentioned above was that usually, pre-designed tests are written against requirements & design documents where as ET (in his mind) is performed by exploring the program without the use of such material. I feel this is a misunderstanding and occurs because appropriate time isn’t allowed for ET during test cycles. Where has anyone written that you can’t use these documents during an ET session?

So I’m now in a position where I’ve seen why this tester has this view that it’s difficult to understand what coverage you’re getting when performing ET because you’re not using requirements or design documents as a basis for your tests.

I tried to think of another way in which we could define ET but it’s difficult. I’d prefer to say pre-design tests versus concurrent testing, if “testing” encapsulates the learning, designing & executing of tests. I’ve read about this before, I understand it (I think), but it wasn’t until that tester asked that question that I seemed to get a bit more clarity and perspective on what I think ET is all about.

From an internal perspective I think we need a shift in how we think about these. The decision our test leads need to make is; “are we writing our tests upfront?” or “are we going to write our tests as we test?”. Both will use exploration, both will use documentation, both will result in some form of script.

Sharpening the saw

2009-01-08T08:52:00.000Z

As part of my “sharpening the saw” (Heusser) activities in 2009 I’ve decided to start learning VB as a way of familiarising myself with how code can be used, written and read. The reason for this is probably two-fold; 1. I have a subconscious worry that a large majority of the testing experts / bloggers are able to code and 2. I feel that due to now being in a managerial role I’m moving away from some of the technical aspects of the job which I used to enjoy, a lot.

Why VB? I looked at other languages, but VB and the MS IDE seems to be a great way for a complete newb such as myself to get in to computer programming. It’s interesting that the IDE provides links explicitly for new developers and the way the lessons, via an online help, are structured are already making it easy to start understanding some of the terminology and concepts.

I’ll continue to blog about this, as it’ll hopefully show my progress and may be inspire others to follow suit…

If you’re doing anything similar, I’d love to hear about it.

Finally, my “sharpening the saw” quote comes from an article in the April 2008 edition of Better Software magazine. The article is written by Matthew Heusser.

1st day of work in 2009 and straight into a coverage reporting conundrum

2009-01-05T16:38:00.001Z

I’m working on something called Early Life Monitoring (as mentioned in a blog below) whereby I review any defects found after project release and review them to understand if they were found prior to release and if not, whether they should have been found by our test effort.

Today I’ve been looking through the list of issues to try and I’m attempting to give each missed defect a catchy category which highlights why those defects weren’t found prior to release which lead me to this problem; summarising defects which sit outside of your testing visibility? I.e. you didn’t test for it, and you didn’t communicate that you weren’t testing it.

To further detail, when I write a test strategy (or, when I did before a move into a management role) I’d define what I was testing, I’d define what I wasn’t testing but there’s still this huge area of the product which isn’t covered by those two; what is that area called? It’s like a testing abyss, those tests you don’t consider, or communicate; how do you communicate that to your departmental manager? Or your technical support representative who’s wondering why we didn’t test for that?

Currently, I’m at a loss. Maybe there isn’t a word to summarise it, maybe I shouldn’t need to look for that word, maybe my brain’s not sharp enough to find an answer to this question today…

Happy new year to all.

Getting from the strategy to the tests

2008-12-18T12:56:00.004Z

Our test team have had a bit of feedback, which was nice, but it's left me struggling to think of solutions.

The feedback: "I'd like to see how the test team go from the test strategy, to the test cases, and I'd like that process to be more visible to a wider audience".

This sounds like a request for a test design walk-through, which we don't do, and never have. It's usually just in tester's head this part isn't it? We write a strategy which says what we're going to test, how long it'll take, how many testers we need, what environments we'll test on and how much it'll cost and then we get some testers, or scripters, and we write the tests before we get the product and start testing. If we're lucky, a developer might review some of our tests / planned coverage and highlight some inaccuracies but that's usually as good as it get's. And now someone wants us to make that whole thought process more visible; a reasonable request I guess.

Reality is, I haven't a clue how we'd do this and I haven't heard of people doing this elsewhere. Are people familiar with this kind of process? Do people need to do this? If so, how have you done this?

There's no reason why we couldn't write another document which details how we get from the strategy to the tests, maybe discussing risks and types of testing along the way but when we feel we don't get much input in to the strategy it's going to be difficult to think of reasons to justify yet another document...

Wordle of my blog...

2008-12-17T12:25:00.003Z

Inspired by Corey Goldberg's blog I did a Wordle for my blog (below) - quite interesting I think!

I wonder if team objectives show up anything...

Mindsets continued: Training testers how to think around problems

2008-12-16T13:28:00.003Z

I’ve been doing a lot of reading around testing for a good couple of years and I’m only just starting to get some clarity around the issues which face us as testers. When I read about testing I come across material which details ways to go about resolving problems, usually in isolation, and very rarely do they appear to tie up with other problems which is where it can become difficult to understand how all these models or techniques can come together.

For example this post highlights two solutions to two problems and is one of the few examples of articles I’ve seen which link the two and show how they overlap.

Problem 1: You don’t know what to test, or how the program works, or what to test first
Solution 1: Use a mnemonic, or an application tour
Problem 2: How do you transform those tours in to tests?
Solution 2: You apply oracle
Result: Provide the tester with the tours, or mnemonics and the possible oracles, and they can start to test a system using their own brain, rather than a prescriptive test.
Problem: It’s dangerous to assume that a tester understands all the problems detail above:

I can’t test everything
Hold on, what is everything?
Ok, how do I now choose what to test?
Sometimes I won’t know where to start
Sometimes I won’t know whether I’ve got a problem or not

In my experience I worry that tester training is somewhat disjointed and we don’t do a very good job of educating new testers on the key problems which face testers, and the techniques which they can use to solve those problems. Is it that we teach new testers how to run the tests, rather than how they think about what tests they need to run, and how they decide if those tests are problems which need escalating. Is it that we teach the solution, which is often inflexible, rather than how to think of a solution, which is infinitely more flexible?

The AST Black Box Software Testing Foundations course , along with some of the reading I’ve recently done has highlighted to me some inadequacies in how I’ve seen testers trained, and how I’ve been trained previously, which makes me want to further understand what kind of knowledge / understanding any tester should have before they pick up a mouse and start testing on a project.

Therefore, I’d be interested on hearing how other teams train new testers or what wisdom / reading experienced testers look back on as material which provided them with clarity on what it means to be a tester, and the techniques we can use to overcome the problems we’re exposed too.

Finally, apologies for a fairly unstructured blog, free time + energy on topic = blog!

Testing: It’s largely about the mindset, isn’t it?

2008-12-12T10:49:00.000Z

I was sat on the train to the recent SIGIST and I got thinking about what a colleague had told me about James Whittaker’s talk at Eurostar on the topic of Exploratory Testing and how Microsoft are trialling this Exploratory Tour idea. The idea, I think, is fantastic and I understand that James is writing a book which will cover the tours in detail and so, rather than talking about these I want to use this blog to look at how I took the concept, and applied it to the nearest thing I had that very morning when travelling to SIGIST, the train…

To recap, James Whittaker of Microsoft looked at using tour analogies to focus their Exploratory Testing and having reflected on this idea it seems to be very much about the mindset with which the tester adopts when designing or executing tests and ultimately, it doesn’t matter whether he or she is doing ET, some form of prescriptive test or an automated test – the mindset is still important and can be consistent across all forms of testing.

So, how can the mindset, or approach to some testing be linked directly with trains? Well, here’s what I came up with (and this was at 7.15am so I make no apologies if it’s complete tosh):

The fast train tour: Take the quickest / shortest route through the program
The slow train tour: Stop and observe (test?) each point in the program
The Bullet Train tour: Review each function for performance; how quickly does this program perform?
The Flying Scotsman tour: See the heuristic oracle, “consistent with previous product”
The Rush hour tour: Apply lots of data / input at every opportunity (stress / load testing?)
The Viaduct tour: High level view of the system, what might a customer’s first impressions of this product be?
The Underground tour: Look at the program from a lower level; focus on data, the code…
The Rotate train tour (I don't know what these are called!): Ensure a consistent change of direction, maybe through testing navigation, whilst testing the program
The Signal Point tour: Test for correct instruction / messaging from the application
The Impatient Commuter tour: test for any unexpected delays in operation, possibly whilst performing the fast train tour
The Packed Station Tour: Get lots of users using the program, how does it react? How good is the information provided to different users? How do they perceive the usefulness of the information provided?

Irrespective of the use of the tours I believe this would aid consideration for different types of tests, again, different mindsets required to test an application. They’re almost heuristics, I guess…
Can you think of any more? How can you use this type of thinking to inspire thought around tests?

Final note; this is an expansion of James Whittaker / Microsoft’s idea and I merely wish to expand, not take credit, for this type of thinking…

An informal research project: Understanding code coverage achieved through tests

2008-12-10T13:58:00.000Z

Yesterday I attended the December SIGIST and of particular interest was a talk by Microsoft’s BJ Rollison which looked to dispel a number of myths about Exploratory Testing. However, one thing in particular which interested me about this talk was where BJ referred to an experiment they’d been conducting which looked at the kind of code coverage testers could gain, on a program, when doing either exploratory, or scripted testing.

Now, I’ll not go in to the detail in this post but this got me thinking about code coverage and how, within our team, we never look at code coverage; we don’t try to understand how much of the code our (the test team’s) tests cover. Therefore, I’m planning to run an informal research project which investigates just this.

It’s important to understand that I’m not going to use this to turn around and say, “Hey, we’ve achieved 80% code coverage so let’s release!”. No, what I want to understand is the rough numbers and use them to understand more about how we test. If on component X we’re achieving 20% code coverage then I want to understand what this means? Should we be able to cover more? Are the developers covering the other 70-80%? Are we releasing products which have masses of untested code in them? Either way, I expect this will tell us something.

Whilst I hope the figures are positive, that is, our tests are covering a considerable amount of the code, there’s also a part of me which thinks a low figure may un-earth some questions about how we design tests, and what information we should be using to ensure that our tests are gaining maximum code coverage.

I’m unsure how feasible this will be, but I’ll keep the blog updated with any progress we make along the way…

Post Release Defects: What I’ve learnt from them?

2008-12-08T16:14:00.000Z

A while ago I blogged about post release defects, and how, as test teams, we can learn from the defects which we didn’t spot prior to the release of the product. I think it’s important, as a tester, to continue to review your testing after the product’s been released because there are questions we can ask of ourselves, such as – should we have tested for that? Why didn’t we spot that? What might we have done differently to test that? And I’ve performed such an assessment on a project I’ve recently worked on.

The first problem I came across when trying to do this was trying to capture the issues which are found after the release of a product. I decided, due to the large number of support calls being made, that any issue which resulted in a software update, or support document, would be counted in my measures. Because it was such a large project, which was released with a number of known issues it wasn’t feasible to count every support call logged (although this might be feasible for smaller products).

This gave me a list of around 100 issues to review, which I split in to the following categories:
· We found it before release (we knew the defect existed prior to release)
· We didn’t find it, we could have found it, but we wouldn’t expect to find it
· We should have seen it (we missed it in our coverage)
· We didn’t see it, we couldn’t have seen it and we shouldn’t have seen it

By doing this, we were able to understand that 60% of the defects which were either being fixed in updates, or which required support articles to inform the customers of issues, or provide a work-around, were known prior to release and we decided to release with those defects in the system. As a test lead this was useful to know, it gave me confidence in our coverage, and our decisions, and backed up the messages we’d been delivering prior to release of the product, i.e. these are bugs which will impact customers!

The other categories allowed us to understand some gaps in our testing; there were defects which were simply due to gaps in coverage, a scenario which hadn’t been tested, or a type of hardware which hadn’t been included in our coverage, and these we can learn from. On top of this there were some defects which we wouldn’t expect to find; such as defects which were found due to inconsistencies with 3rd party products, or defects with areas of the product which had been de-scoped from the testing effort earlier in the project (due to time pressures).

So, to conclude – this post-release defect review gave us confidence in the testing performed, something which we’d started to question because of the volume of defects which were being found by customers. It also highlighted some gaps in our coverage, which we can learn from in future projects and finally, it told us that generally – we’d done a pretty good job. For those who use DDP, ours was around 95%.

To keep this readable I’ve excluded a lot of detail, and decision making, from the process above but if anyone’s interested in knowing more I’d be happy to discuss on a 1-2-1 basis, or via another blog if the interest is there.

Evolution of Exploratory Testing

2008-12-06T10:42:00.000Z

I wanted to talk a little bit about what we’re doing with Exploratory Testing where I work, where it’s come from, where I think its heading and what we’ve done so far.

I’ve been at my current company for around 7 years and as far as I can remember “exploratory testing” has been around for the whole time but for some reason, until recently, its always been used in an “ad-hoc” way and was usually bolted on to the end of a largely scripted test cycle as something of a last minute bug-hunt exercise.

Around 3-4 years ago a colleague of mine attended a couple of Exploratory Testing workshops held by James Lyndsay in London and from his attendance at those we developed a template which allowed test leads to structure an exploratory test; it highlighted areas of a program in which a tester should focus when “exploratory testing”.

The next change in how we used Exploratory came around 12 months ago when we were doing a large project (2-3 years development + testing). We’d been asked to de-scope our testing effort because our test cycles were too large so one of the things we looked at was reducing the cycle by doing more exploratory testing. We looked at one area of the program in particular, realised our scripted tests weren’t telling us much new about the product, and decided that we’d stop scripted testing and start exploratory testing this area in each test cycle. We had the benefit that the testers assigned to that area of the product had a lot of product knowledge, and therefore our only dilemma was how we plan each cycle, and report progress. This is where we came upon James Bach’s’ Session-Based-Test-Management and we used the basics of this as a way of planning. We split the particular area of the product into chunks, turned the chunks in to time-boxed sessions, set the testers some guidelines for items within the chunk which they should ensure they look at, and followed each session up with a de-brief where the tester discussed what he, or she, had tested with the test lead. This allow the test lead to understand what had been tested, ask questions, understand defects and it also allowed the tester to request more time, if they felt they’d run out of time.

So where do we head next? Well, I still don’t think we’re doing Exploratory Testing as it’d be described in the industry, I don’t think we’re in a place where we’re doing “simultaneous learning, test design & test execution” and that’s where I’d like us to be. I’d like us to have testers who can sit at a program, use their array of test techniques and tools and test a product by using the results of their last test to guide their focus for the next test they perform – but we’re not there yet.

We’re looking at expanding how we use Exploratory Testing by trialling Microsoft’s Exploratory Tour idea which looks like it does an excellent job of setting a testers’ mindset ahead of testing and this is key, I think mindset has a huge part to play in the quality of testing a tester performs. I’m hoping we can further understand how Microsoft use their tours and then either use it as-is, or adapt it, and then trial it on a few projects – tying it in with the SBTM; another advancement in how we do Exploratory.

My goal is to see Exploratory Testing seen as an equal to scripted testing within our test team, and the wider division, which isn’t something we’ll achieve over night. However, our use of Exploratory Testing is progressing and it’s exciting to see where it’s come so far, and where it may end up.