Yesterday I attended the December SIGIST and of particular interest was a talk by Microsoft’s BJ Rollison which looked to dispel a number of myths about Exploratory Testing. However, one thing in particular which interested me about this talk was where BJ referred to an experiment they’d been conducting which looked at the kind of code coverage testers could gain, on a program, when doing either exploratory, or scripted testing.
Now, I’ll not go in to the detail in this post but this got me thinking about code coverage and how, within our team, we never look at code coverage; we don’t try to understand how much of the code our (the test team’s) tests cover. Therefore, I’m planning to run an informal research project which investigates just this.
It’s important to understand that I’m not going to use this to turn around and say, “Hey, we’ve achieved 80% code coverage so let’s release!”. No, what I want to understand is the rough numbers and use them to understand more about how we test. If on component X we’re achieving 20% code coverage then I want to understand what this means? Should we be able to cover more? Are the developers covering the other 70-80%? Are we releasing products which have masses of untested code in them? Either way, I expect this will tell us something.
Whilst I hope the figures are positive, that is, our tests are covering a considerable amount of the code, there’s also a part of me which thinks a low figure may un-earth some questions about how we design tests, and what information we should be using to ensure that our tests are gaining maximum code coverage.
I’m unsure how feasible this will be, but I’ll keep the blog updated with any progress we make along the way…
2 comments:
"Code coverage" has a broad range of meanings. The most common meaning involves structural coverage (for example, which statements have been tested). Achieving high structural coverage is relatively easy during programmer-testing (where the tester/author has direct access to the code) and very hard during system testing (it is hard even to reliably measure structural coverage during system testing, especially exploratory system testing by many testers, and even harder for testers to interpret the structural coverage data they get).
I continue to wonder how useful structural coverage is as a measure of system testing.
Some relevant materials of mine are at http://www.kaner.com/pdfs/measurement_segue.pdf, http://www.kaner.com/pdfs/metrics2004.pdf, http://www.kaner.com/pdfs/pnsqc00.pdf, and http://www.kaner.com/pdfs/negligence_and_testing_coverage.pdf
I think it is more important for me, when I am doing system testing, to assess coverage of the specification, of the failure mode list (the list of ways I imagine the program might fail), of the key user scenarios, and of the environments in which the product will run. These are not addressed by structural coverage and they are hard to reach with unit-level or lower-level-integration tests of the kind that are easily tracked by code coverage tools.
I don't think it is interesting to criticize system testing because it does not attempt to redo what programmer-testing should already have achieved.
I teach separate semester-length courses in black box system testing and in programmer testing. It is fascinating looking at the mindsets that evolve during the semester, and how different they are in the two courses. Reaching up to the realm of system-level risks is hard for the programmer-testing students and reaching down to the realm of implementation detail is hard for the system testing students (even for the students who take both courses). I've seen the same difficulties in practice.
I see your point Cem, and it ties up with what BJ said because Microsoft's testers are I think more fortunate that most in that they have the ability to perform "programmer testing" and "system teting". That is, they're not system testers trying to understand code coverage.
I'd like to understand how our development team approach this, maybe for no other reason to increase confidence in the testing which both our teams, combined, are performing.
I'll check out your material as it'll give me food for thought on my prosed research.
Simon
Post a Comment