Checking and Information Value

It's now over three years ago that I wrote my last post on "testing and checking" - and I wrote three in total. I consider them a little 'rough' but I knew they contained some gems so revisited them recently - refs [1], [2] & [3].

James and Michael wrote a blog recently, clarifying and redefining what they meant by testing and checking, ref [4].

After reading it - and thinking about it - I felt that some emphasis was missing - for me. This doesn't talk directly to the definitions but more the application of them.

Note, James & Michael seem to apply this refined terminology within the frame of the Rapid Software Testing methodology, ref [11], but I think others will use it outside of that also.

Some History re-cap
After reading the original post, ref [5], I started wondering, "why?", what was the motivation behind making the distinction. I didn't experience the problem being described and couldn't see how the distinction might help me. As part of that I started wondering about how checks and checking was being framed and the extent to which it was being wrapped in testing.

My original questions:

• Why make a check when no testing is involved?
• How or why would a check be made without testing?

- ie why make a check and not analyse the result; why make a check without any thinking/analysis (testing) beforehand?

Note, the definition didn't say that a check wouldn't be evaluated, but it could be misinterpreted that a check without testing was useful (although I didn't think that was the intention).

So, at the time, I couldn't detach checking from testing - I couldn't see a situation where a check would be made without testing. That was my perspective and how I would advocate checking.

But I'm also very aware that there will be people/organisations that do some form of checking without any testing... How might that happen?

• "We always run this batch of scripts. When they pass then the SW is ready."

Note, I believe the definitions are trying to enable more accurate observation and discussion about elements that were testing or not.


Potential Trap

A check (or checking) is used in isolation without any test "thinking" (whether as analysis beforehand or afterwards.) This case is a form of the trap mentioned in ref [6].

On to Information Value
So, what was I missing? My early posts were looking for reasons / needs for the distinction - and I accepted that the distinction was useful (although, I admit, not immediately as I didn't see the utility in them in the beginning - that came later.) My remaining question was around emphasizing the importance of testing and the selection of a check as being rooted in testing.

During the last year I started thinking more and more about emphasizing information value of testing. This is partly trying to establish congruence between customer needs and testing goals, ref [7] & [8], partly about introducing test framing, ref [9], and framing checking from a testing perspective, ref [10].

So, where does this bring me?

Checking in isolation
Talking about checking in isolation has value when identifying the activity - whether as something that is planned or an observation in a process or activity.

• "On the upcoming release we'll reuse some checks from the last release - they will give feedback on basic data configuration - whether the data configured pre-upgrade is still visible after upgrade." (plan)
• "You're reporting a fault due to an automated script flagging an issue?" (activity) "Ok, has the result and activity been double-checked - that the flagging is correct?" (testing)
• "There is a bunch of automated scripts that run on every code change - and they never issue any problem indication. Is that ok, or is it time to review their content and purpose?" (process) 

Information Value of Checks and Checking
When considering whether to use a check or checking there is an implicit activity of comparing what you'd like to know about a product, what you can achieve (and how) and what each instance of a check might give. At least, I claim (with only anecdotal proof) that this is the case in many good testers. For example:

Wanted Information about product

Note, this example excludes people not making an evaluation about a check or whether or not to do any checking or what the check outcome says...

Part of this evaluation (sometimes estimation) is to understand the value of making a check - something cheap and easy might be preferable, but something expensive that returns a large information value might also be worth the effort. The key thing is the decision to make the check - because it will provide some information.

Example: In the past I've advocated 169hr tests (checks) - the result might be good/no good (pass/fail), but the value of it might be part regulatory PLUS a lot of additional data that can be mined/explored.

Why it might be useful to check or perform checking is then attempting to work out the value of the information it might give. This is information about the product "under test".

Information value can be estimated beforehand and/or assessed afterwards.

Checking as part of Testing
When information value (or at least its intention) connected to a check or checking is captured then the check or checking becomes part of testing.

A model of testing, checking and information value could be expressed as:

Testing ≥ Checking + Information value

In mathematical terms it could be written:

Testing ⊇ Checking + Information value

This means every instance of "Checking + Information value" is an element of Testing but also that "Testing" does not equal "Checking + Information value".

Checks (with no associated information value) ∉ Testing

Meaning checks without associated information value (either decision to make a check or analysis of a check outcome) are not part of testing.

Conclusion
Whether I think of checks, checking and testing inside the Rapid Software Testing methodology or not I think the terminology is useful. It is useful to identify artifacts of checking in the wild - whether in observed processes, activities or intended actions. It is useful to identify and categorize as any field scientist would categorize their observations.

In talking to people about processes and their actions it is very useful to describe and illustrate when a process was being followed, maybe even identifying assumptions that were not being stated and how they might confuse or muddy communication about what was really happening. It might be useful to highlight the extra work needed to make the activity into good testing.

In my worldview of testing there are no checks or checking in isolation, there is either an intention or idea beforehand about the usefulness of such a check or an analysis afterwards about the usefulness of the check or other assessment about the information it yields. Often, in the cases of good checking (implying good testing) both the beforehand intention/idea/hypothesis and the resulting assessment will both happen. That is my ambition where checks are used.

In the cases where checks are used without pre-determined intention or resultant analysis - and this does happen - my ambition is to raise the bar. In my worldview this is not testing. I want to raise the bar partly by awareness - noticing (and getting others to notice) this happens, partly by contrasting this to how it might look if we connect some information value to those checks. In so doing, understanding the real value (and maybe cost) of good testing. Note, there is no automatic raised cost by adding information value to checks to get testing, but rather a reduction in the overheads of "fake testing" (an activity which was maybe thought to be testing but wasn't).

I wish to raise the bar of checking by emphasizing the associated thinking attached to checking to identify information value (positive or negative), and so raise the bar of testing.

Thoughts, comments?

References
[1] The Tester's Headache: To test or not to test, just checking... http://testers-headache.blogspot.com/2009/08/to-test-or-not-to-test-just-checking.html
[2] The Tester's Headache: Sapient checking? http://testers-headache.blogspot.com/2009/09/sapient-checking.html
[3] The Tester's Headache: More notes on testing & checking http://testers-headache.blogspot.com/2009/09/more-notes-on-testing-checking.html
[4] Satisfice: Testing and Checking Refined http://www.satisfice.com/blog/archives/856
[5] Developsense: Testing vs. Checking http://www.developsense.com/2009/08/testing-vs-checking.html
[6] The Tester's Headache: Another Regression Test Trap http://testers-headache.blogspot.com/2013/03/another-regression-test-trap.html
[7] YorkyAbroad Slideshare: Testing Lessons From The Rolling Stones http://www.slideshare.net/YorkyAbroad/testing-lessons-from-the-rolling-stones
[8] The Tester's Headache: Mapping Information Value in Testing http://testers-headache.blogspot.com/2012/12/mapping-information-value-in-testing.html
[9] Developsense: Test Framing http://www.developsense.com/resources/TestFraming.pdf
[10] The Tester's Headache: Framing: Some Decision and Analysis Frames in Testing http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html
[11] Satisfice: Comment on Testing and Checking Refined http://www.satisfice.com/blog/archives/856/comment-page-1#comment-268572