To test or not to test, just checking...

#softwaretesting

To test or not to test, that is the question....

There was a very good and incisive post from Michael Bolton, here, the other day, distinguishing testing from checking. A lot of good points and items to trigger reflection.

The evidence of a good post is that it gives you something to think about. I started thinking about my own experience, specifically in areas of performance, load & robustness testing and automated regression suites.

I asked myself a couple of questions - with Michael's post in mind:

When is a test not a test?

Does it matter to distinguish a test from a check?

Note, when I talk about a test check or "check" below I'll use it in the meaning that Michael used.

A Test Case

In my particular working area (telecomms) a test case is a generic term. It's a label that can be attached to a group of test steps - to be used in a test, a "check" or a measurement.

In performance testing we have scenarios executing from which measurements about the system under test are gathered. These scenarios are labelled as test cases even if they don't explicitly test or check.

The output is used for comparison and tracking against established benchmarks, or even creating benchmarks. The results are analysed with stakeholders to determine whether further investigation is needed.

There is no problem in the organisation in calling this measurement a test case - it's purely a way of grouping some steps that contribute to an activity.

So test cases, checks or performance measurements can be designed and executed but they are collectively referred to as test cases - or even just tests. The test case, check or measurement relates to the activity and not the unit container.

Test (Case) or Not Test (Case)

The need to define, distinguish and be clear about terminology is a natural part of a critical thinking approach.

So, is there a conflict in using a general label such as test or test case?

A test or test case is a container. It doesn't say very much unless we know something about how they are applied - e.g. whether it's a unit, functional, robustness or other type of test case.

With most testing the point at which a test check emerges is when it has gone through one successful iteration - even a unit check is a unit test until it "works" - meaning the correct behaviour is verified (or agreed). The first times it is executed may reveal information about the unit design or the design of the unit test that may need modification to get an accepted result.

So from first inception it is a test case - once the correct behaviour has been verified (and agreed, if need be) then it can be used for regression purposes - at which point it might be thought of as a check. However, I'll probably still think of it as a test, see below.

To Test

When I think about software testing I like Gerry Weinberg's definition of testing about information gathering. In lay terms I like Michelle Smith's description as an investigative reporter, here.

Taking a dictionary definition of test (not specific to software - from the Oxford Concise dictionary): "A critical examination or evaluation of the qualities or abilities of a person or thing." I can agree with this also.

So is a test case written with lots of expected results a test or a check?

An expected result from the test is defined. However, the determination of the regression test/suite as being suitable to give input to the stakeholders is the critical evaluation step, ie the decision that the outcome will give valid input to the stakeholders.

A regression suite should be kept under review for relevance and value of the information that it provides. Note, not all of these tests will be automated and this will always throw up an interesting dimension.

So, in this context, it's a test and not a check.

Checking vs Testing?

In Elisabeth's example, here, the difference between preforming tests for the GUI and exploring the system for usability is really a problem of communication - it wasn't clear (maybe to the developer, tester or both) that the type of testing being talked about was different.

Labelling it as checking and testing is one way to distinguish. Another way, using an agile example, is to say that one type of testing belongs to quadrant 1 and the other is quadrant 3 (referring to agile testing quadrants.)

So, what's in a name? Or what's in a label?

Q&A

Q: When is a test not a test?

A: When it's a performance measurement, but even then I group it as a test. Test is just a label for grouping.

Q: Does it matter to distinguish a test from a check?

A: In general, no. Items are grouped as tests. In my experience regression tests are selected for the information about the system they will provide.

If we need to distinguish between checks and tests then there may be other issues that are at the root cause of the problem.

Are you putting the evaluation into your regression selection criteria?

Do you or your co-workers know how to distinguish between different types of testing?

Are you "checking" that you're "testing"?