Another Regression Test Trap

I mentioned in a previous post, ref [1], a trap to be aware of when thinking about regression testing. Time for another...

There is a lot of "rubbish" written about regression testing. As with other aspects of testing it is sometimes commoditized, packaged and "controlled".

Trap

Regression Testing is the act of re-running tests - that have previously been developed, designed or executed. Period.

The trap is that it stops there...

Sometimes, there is a contextual aspect included - which is addressed by stating that analysis of changes in the product, including the processes that were used for that change should be made. But in many cases - and indeed, in several pieces of literature discussing regression testing - the extent of this analysis is about setting the regression test scope from the previously developed or designed test ideas.

Hmmm, seems reasonable? Or does it? Let's look at a couple of commonly available sources of information.

Other Sources on Regression Testing:

Wikipedia, ref [2]: "Regression testing can be used to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change."

So, selecting a minimum set of tests - systematically - is an efficient test of a system, if that set provides adequate "coverage" of a change in the system. Mmm, good luck with that. This is bordering on pseudoscience, ref [4], - it's an untestable and vague claim. Having said that, the statement is qualified by "can" - hmm, ok, not so helpful though.

ISTQB glossary, ref [3]: "regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is
changed."

I can almost agree with this. Almost. Well, ok, not really. What's a big sticking point for me? "Ensure". My dictionary (New Oxford American to hand just now) states that ensure is either a guarantee or the action of making sure... Ok, so how can my testing guarantee or make sure that a defect has not been introduced. Why would it want to make sure that a defect hasn't been uncovered - wouldn't you really want to know about this????

Note, it's possible to find similar problematic definitions in testing text books.


And?

Ok, am I splitting hairs? Am I dissecting these definitions more than I should? Well, what level of clarity should I hold them too? Should I be satisfied that they give "surface" explanations and that I should not examine or look too deeply? The thing is, if a definition falls apart under scrutiny is it worth using? I can't use it and I don't advocate it they can't define something then the definition falls apart (or becomes ambiguous) under scrutiny is it worth using????


What's the problem?

My issues with typical regression scopes are connected to two often unstated assumptions:

1. An assumption that the regression scope that "worked" once is sufficient now - where the system has changed. (This means that the number of test ideas covered previously - whether expressed as instances (test cases) or not - are good enough for a legacy scope now.)
2. An assumption that a previous regression scope is somehow a conformance scope. (This means that an instance of a test idea - a test case - is correct now, in the new/modified system.)

Note, if these assumptions are made they should be stated. It could be adequate in some cases to assess the previous scope and determine that it is sufficient for the current needs. It could be useful to state that the previous regression scope is being treated as a conformance scope (or reference scope, or oracle) - if that's what the assessment of the current problem/system produces.

I think a post on how I'd approach a regression test problem is in order...

Tester Challenge
On a train journey returning from SWET4 to Stockholm I sat with James Bach and he tested me on "regression testing". He suggested there was one case where a previously executed test campaign could be repeated as a regression testing campaign. I gave him two.

I'll leave these as an exercise for the reader...

References
[1] Regression Test Trap #492 http://testers-headache.blogspot.com/2013/02/regression-test-trap-492.html
[2] Wikipedia: Regression testing http://en.wikipedia.org/wiki/Regression_test (phrase checked 29 April 2013)
[3] ISTQB: Standard glossary of terms used in Software Testing, version 2.2 http://www.istqb.org/downloads/finish/20/101.html
[4] Wikipedia: Pseudoscience http://en.wikipedia.org/wiki/Pseudoscience

Bonus Reference
[5] Checklist for identifying dubious technologies http://www.randi.org/images/commentary/200703/Checklist%20for%20identifying%20dubious%20technical%20processes%20and%20products.pdf