Checking and Information Value

It's now over three years ago that I wrote my last post on "testing and checking" - and I wrote three in total. I consider them a little 'rough' but I knew they contained some gems so revisited them recently - refs [1], [2] & [3].

James and Michael wrote a blog recently, clarifying and redefining what they meant by testing and checking, ref [4].

After reading it - and thinking about it - I felt that some emphasis was missing - for me. This doesn't talk directly to the definitions but more the application of them.

Note, James & Michael seem to apply this refined terminology within the frame of the Rapid Software Testing methodology, ref [11], but I think others will use it outside of that also.

Some History re-cap
After reading the original post, ref [5], I started wondering, "why?", what was the motivation behind making the distinction. I didn't experience the problem being described and couldn't see how the distinction might help me. As part of that I started wondering about how checks and checking was being framed and the extent to which it was being wrapped in testing.

My original questions:

• Why make a check when no testing is involved?
• How or why would a check be made without testing?

- ie why make a check and not analyse the result; why make a check without any thinking/analysis (testing) beforehand?

Note, the definition didn't say that a check wouldn't be evaluated, but it could be misinterpreted that a check without testing was useful (although I didn't think that was the intention).

So, at the time, I couldn't detach checking from testing - I couldn't see a situation where a check would be made without testing. That was my perspective and how I would advocate checking.

But I'm also very aware that there will be people/organisations that do some form of checking without any testing... How might that happen?

• "We always run this batch of scripts. When they pass then the SW is ready."

Note, I believe the definitions are trying to enable more accurate observation and discussion about elements that were testing or not.


Potential Trap

A check (or checking) is used in isolation without any test "thinking" (whether as analysis beforehand or afterwards.) This case is a form of the trap mentioned in ref [6].

On to Information Value
So, what was I missing? My early posts were looking for reasons / needs for the distinction - and I accepted that the distinction was useful (although, I admit, not immediately as I didn't see the utility in them in the beginning - that came later.) My remaining question was around emphasizing the importance of testing and the selection of a check as being rooted in testing.

During the last year I started thinking more and more about emphasizing information value of testing. This is partly trying to establish congruence between customer needs and testing goals, ref [7] & [8], partly about introducing test framing, ref [9], and framing checking from a testing perspective, ref [10].

So, where does this bring me?

Checking in isolation
Talking about checking in isolation has value when identifying the activity - whether as something that is planned or an observation in a process or activity.

• "On the upcoming release we'll reuse some checks from the last release - they will give feedback on basic data configuration - whether the data configured pre-upgrade is still visible after upgrade." (plan)
• "You're reporting a fault due to an automated script flagging an issue?" (activity) "Ok, has the result and activity been double-checked - that the flagging is correct?" (testing)
• "There is a bunch of automated scripts that run on every code change - and they never issue any problem indication. Is that ok, or is it time to review their content and purpose?" (process) 

Information Value of Checks and Checking
When considering whether to use a check or checking there is an implicit activity of comparing what you'd like to know about a product, what you can achieve (and how) and what each instance of a check might give. At least, I claim (with only anecdotal proof) that this is the case in many good testers. For example:

Wanted Information about product

Note, this example excludes people not making an evaluation about a check or whether or not to do any checking or what the check outcome says...

Part of this evaluation (sometimes estimation) is to understand the value of making a check - something cheap and easy might be preferable, but something expensive that returns a large information value might also be worth the effort. The key thing is the decision to make the check - because it will provide some information.

Example: In the past I've advocated 169hr tests (checks) - the result might be good/no good (pass/fail), but the value of it might be part regulatory PLUS a lot of additional data that can be mined/explored.

Why it might be useful to check or perform checking is then attempting to work out the value of the information it might give. This is information about the product "under test".

Information value can be estimated beforehand and/or assessed afterwards.

Checking as part of Testing
When information value (or at least its intention) connected to a check or checking is captured then the check or checking becomes part of testing.

A model of testing, checking and information value could be expressed as:

Testing ≥ Checking + Information value

In mathematical terms it could be written:

Testing ⊇ Checking + Information value

This means every instance of "Checking + Information value" is an element of Testing but also that "Testing" does not equal "Checking + Information value".

Checks (with no associated information value) ∉ Testing

Meaning checks without associated information value (either decision to make a check or analysis of a check outcome) are not part of testing.

Conclusion
Whether I think of checks, checking and testing inside the Rapid Software Testing methodology or not I think the terminology is useful. It is useful to identify artifacts of checking in the wild - whether in observed processes, activities or intended actions. It is useful to identify and categorize as any field scientist would categorize their observations.

In talking to people about processes and their actions it is very useful to describe and illustrate when a process was being followed, maybe even identifying assumptions that were not being stated and how they might confuse or muddy communication about what was really happening. It might be useful to highlight the extra work needed to make the activity into good testing.

In my worldview of testing there are no checks or checking in isolation, there is either an intention or idea beforehand about the usefulness of such a check or an analysis afterwards about the usefulness of the check or other assessment about the information it yields. Often, in the cases of good checking (implying good testing) both the beforehand intention/idea/hypothesis and the resulting assessment will both happen. That is my ambition where checks are used.

In the cases where checks are used without pre-determined intention or resultant analysis - and this does happen - my ambition is to raise the bar. In my worldview this is not testing. I want to raise the bar partly by awareness - noticing (and getting others to notice) this happens, partly by contrasting this to how it might look if we connect some information value to those checks. In so doing, understanding the real value (and maybe cost) of good testing. Note, there is no automatic raised cost by adding information value to checks to get testing, but rather a reduction in the overheads of "fake testing" (an activity which was maybe thought to be testing but wasn't).

I wish to raise the bar of checking by emphasizing the associated thinking attached to checking to identify information value (positive or negative), and so raise the bar of testing.

Thoughts, comments?

References
[1] The Tester's Headache: To test or not to test, just checking... http://testers-headache.blogspot.com/2009/08/to-test-or-not-to-test-just-checking.html
[2] The Tester's Headache: Sapient checking? http://testers-headache.blogspot.com/2009/09/sapient-checking.html
[3] The Tester's Headache: More notes on testing & checking http://testers-headache.blogspot.com/2009/09/more-notes-on-testing-checking.html
[4] Satisfice: Testing and Checking Refined http://www.satisfice.com/blog/archives/856
[5] Developsense: Testing vs. Checking http://www.developsense.com/2009/08/testing-vs-checking.html
[6] The Tester's Headache: Another Regression Test Trap http://testers-headache.blogspot.com/2013/03/another-regression-test-trap.html
[7] YorkyAbroad Slideshare: Testing Lessons From The Rolling Stones http://www.slideshare.net/YorkyAbroad/testing-lessons-from-the-rolling-stones
[8] The Tester's Headache: Mapping Information Value in Testing http://testers-headache.blogspot.com/2012/12/mapping-information-value-in-testing.html
[9] Developsense: Test Framing http://www.developsense.com/resources/TestFraming.pdf
[10] The Tester's Headache: Framing: Some Decision and Analysis Frames in Testing http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html
[11] Satisfice: Comment on Testing and Checking Refined http://www.satisfice.com/blog/archives/856/comment-page-1#comment-268572

Another Regression Test Trap

I mentioned in a previous post, ref [1], a trap to be aware of when thinking about regression testing. Time for another...

There is a lot of "rubbish" written about regression testing. As with other aspects of testing it is sometimes commoditized, packaged and "controlled".

Trap

Regression Testing is the act of re-running tests - that have previously been developed, designed or executed. Period.

The trap is that it stops there...

Sometimes, there is a contextual aspect included - which is addressed by stating that analysis of changes in the product, including the processes that were used for that change should be made. But in many cases - and indeed, in several pieces of literature discussing regression testing - the extent of this analysis is about setting the regression test scope from the previously developed or designed test ideas.

Hmmm, seems reasonable? Or does it? Let's look at a couple of commonly available sources of information.

Other Sources on Regression Testing:

Wikipedia, ref [2]: "Regression testing can be used to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change."

So, selecting a minimum set of tests - systematically - is an efficient test of a system, if that set provides adequate "coverage" of a change in the system. Mmm, good luck with that. This is bordering on pseudoscience, ref [4], - it's an untestable and vague claim. Having said that, the statement is qualified by "can" - hmm, ok, not so helpful though.

ISTQB glossary, ref [3]: "regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is
changed."

I can almost agree with this. Almost. Well, ok, not really. What's a big sticking point for me? "Ensure". My dictionary (New Oxford American to hand just now) states that ensure is either a guarantee or the action of making sure... Ok, so how can my testing guarantee or make sure that a defect has not been introduced. Why would it want to make sure that a defect hasn't been uncovered - wouldn't you really want to know about this????

Note, it's possible to find similar problematic definitions in testing text books.


And?

Ok, am I splitting hairs? Am I dissecting these definitions more than I should? Well, what level of clarity should I hold them too? Should I be satisfied that they give "surface" explanations and that I should not examine or look too deeply? The thing is, if a definition falls apart under scrutiny is it worth using? I can't use it and I don't advocate it they can't define something then the definition falls apart (or becomes ambiguous) under scrutiny is it worth using????


What's the problem?

My issues with typical regression scopes are connected to two often unstated assumptions:

1. An assumption that the regression scope that "worked" once is sufficient now - where the system has changed. (This means that the number of test ideas covered previously - whether expressed as instances (test cases) or not - are good enough for a legacy scope now.)
2. An assumption that a previous regression scope is somehow a conformance scope. (This means that an instance of a test idea - a test case - is correct now, in the new/modified system.)

Note, if these assumptions are made they should be stated. It could be adequate in some cases to assess the previous scope and determine that it is sufficient for the current needs. It could be useful to state that the previous regression scope is being treated as a conformance scope (or reference scope, or oracle) - if that's what the assessment of the current problem/system produces.

I think a post on how I'd approach a regression test problem is in order...

Tester Challenge
On a train journey returning from SWET4 to Stockholm I sat with James Bach and he tested me on "regression testing". He suggested there was one case where a previously executed test campaign could be repeated as a regression testing campaign. I gave him two.

I'll leave these as an exercise for the reader...

References
[1] Regression Test Trap #492 http://testers-headache.blogspot.com/2013/02/regression-test-trap-492.html
[2] Wikipedia: Regression testing http://en.wikipedia.org/wiki/Regression_test (phrase checked 29 April 2013)
[3] ISTQB: Standard glossary of terms used in Software Testing, version 2.2 http://www.istqb.org/downloads/finish/20/101.html
[4] Wikipedia: Pseudoscience http://en.wikipedia.org/wiki/Pseudoscience

Bonus Reference
[5] Checklist for identifying dubious technologies http://www.randi.org/images/commentary/200703/Checklist%20for%20identifying%20dubious%20technical%20processes%20and%20products.pdf

Regression Test Trap #492

I didn't call this "the" trap, as I think there are several. This is one aspect.

In June last year I had a little twitter rant/exploration (or transpective investigation, [1]) about regression testing. It was triggered when I saw a LinkedIn skill labelled as "Regression Testing". I wanted to know which skills that people considered specific to regression testing.

Now, one can take some of the LinkedIn labels with a pinch of salt, but I wanted to test the water about this one.

Note, I really enjoy these "bursts of activity" I have on twitter. There are many great minds that immediately question, probe, ask for explanation or example. I love it and am very appreciative to anyone that asks me to explain myself. It makes me stronger and clarifies my thoughts!

The twitter discussion dump, via twimemachine, in reverse order:

@Arborosa you have that problem in non-regression-specific testing also Thu Jun 28 06:54:20 +0000 2012@jamesmarcusbach Oh, the NAP is a great resource - was looking at this the other day http://t.co/Tjn3b18L Wed Jun 27 10:00:53 +0000 2012@jamesmarcusbach That's great, thanks - I need to spend more time on it - reminds me of Göranzon's work-> must dig that out again Wed Jun 27 09:49:44 +0000 2012@jamesmarcusbach True, it's one of my aids to avoid the trap of going from "application in context" to "application in general" Wed Jun 27 09:42:56 +0000 2012@jamesmarcusbach I try to distinguish between skills and application of skills, reminds me of "Dialogue Skills and Tacit Knowledge" Wed Jun 27 09:25:44 +0000 2012@jamesmarcusbach I call the workshops a feature walkthrough, see slides 20-27 in http://t.co/XnF5Ubnk - usual health warning with ppt slides Wed Jun 27 08:50:05 +0000 2012@jamesmarcusbach In my context, these skills are not specific to regression testing Wed Jun 27 08:37:44 +0000 2012@jamesmarcusbach Test idea maintenance - not specific to regression testing 4me - can be triggered by feature or test harness change/growth Wed Jun 27 08:36:42 +0000 2012@jamesmarcusbach Interviewing and analysis - I do this as part of feature interaction analysis - sometimes 1-2-1, sometimes as a workshop Wed Jun 27 08:36:00 +0000 2012@DuncNisbet yes, start from a picture of what you want to learn + where risks are - then let that point to what's "interesting" to know Tue Jun 26 22:08:01 +0000 2012@DuncNisbet I think of tests with a potential to find a problem as "interesting" - ideas to pursue/try out Tue Jun 26 22:00:29 +0000 2012@rahul_verma me too - I thought your questions were good and relevant! Tue Jun 26 19:10:03 +0000 2012@rahul_verma look forward to it Tue Jun 26 19:02:17 +0000 2012@rahul_verma yes, that's what I aim for Tue Jun 26 19:01:48 +0000 2012@rahul_verma and how many other types and application of testing were they no good at? :) Tue Jun 26 19:00:47 +0000 2012@rahul_verma I think people mix up activities, skills, application and results - and makes our job (communicating about a product) harder :( Tue Jun 26 18:57:20 +0000 2012@rahul_verma applying a skill successfully in an activity (RT) does not make that a skill specific to that activity Tue Jun 26 18:56:15 +0000 2012RT @rvansteenbergen: Want tweets about the latest news from testblogs? Follow @testingref #softwaretesting Tue Jun 26 18:45:00 +0000 2012@jamesmarcusbach I'd love to hear about the other things Tue Jun 26 18:14:31 +0000 2012@rahul_verma agree! Tue Jun 26 18:11:32 +0000 2012@rahul_verma on a certain level performance testing and regression testing do have an overlap of ambition Tue Jun 26 18:10:10 +0000 2012@rahul_verma performance testers focus on an element of a product's characteristics - to measure, monitor, feedback, recommend and help tune Tue Jun 26 18:09:02 +0000 2012@rahul_verma exact knowledge is tricky to define.... Tue Jun 26 18:03:28 +0000 2012@rahul_verma I'd say the purpose distinguishes it as regression testing, but not necessarily the skills used Tue Jun 26 18:02:42 +0000 2012@rahul_verma Are you asserting that regression testing is done by regression testers? Tue Jun 26 13:05:39 +0000 2012@rahul_verma What skillset in "regression testing" is specific to regression testing? Tue Jun 26 13:03:13 +0000 2012@rahul_verma Think skillset rather than domain knowledge Tue Jun 26 13:02:15 +0000 2012@rahul_verma Is that separate from good testing? Tue Jun 26 12:55:56 +0000 2012@thetestmile And is that separate from good testing? Tue Jun 26 12:55:34 +0000 2012@rahul_verma Are you saying you can be a good tester and not a good regression tester, or vice versa? Tue Jun 26 12:55:08 +0000 2012Now where's the "black viper testing" skillset button.... Tue Jun 26 12:42:55 +0000 2012"Regression testing" is a "skill" in linkedin. Really? Discuss... #faketestingmeasurement Tue Jun 26 12:41:39 +0000 2012

Trap

The trap is to think of the skills associated with good regression testing as being separate from good testing.

In my context and experience, I consider good testing to take into account aspects of new features that interact with legacy (existing) features or systems. This implies that "regression testing" is included. But if I think of specific skills needed for regression testing then these are just as applicable to "non-regression testing":

• Investigative skills about the domain (system)
• Discussing development and the existing system with developers/BA's/stakeholders
• Test Idea / Environment Analysis & Maintenance
• Reporting and discussing the status of the system
• etc, etc

So, is a "regression testing" skill separate or unique to regression testing? I assert not, and that it can be just as common in "non-regression testing" or any good testing.

To Be Continued

I have another trap about regression testing - coming soon.

References

[1] Satisfice: Transpective Dialogues for Learning http://www.satisfice.com/blog/archives/62

Other "traps":

[2] The Linear Filter Trap http://testers-headache.blogspot.com/2012/03/linear-filter-trap.html

[3] The Documented Process Trap http://testers-headache.blogspot.com/2012/06/documented-process-trap.html

[4] The Hindsight Trap http://testers-headache.blogspot.com/2012/03/hindsight-trap.html

Mapping Information Value in Testing

This has been a year where I’ve been thinking a lot about how testing information is communicated, some of the associated traps and some models to help explore this.

For SWET4 (Swedish Workshop on Exploratory Testing #4) I was prepared to talk about modelling information flow in testing and how I had used it. I’d called the topic “Modeling: Information-driven Testing VS Activity and Artifact-driven Testing”.

The drive of the proposed talk/discussion was about using models to illustrate traps associated with many projects - especially large projects, projects with large product bases or legacy systems. This comprised of two illustrative models. 

The first:

Model #1

In the diagram notice the flow of customer requirements that devolve into a project which, from a testing perspective, turns into some form of analysis, design, execution and reporting. Typically there may be some artifact from these activities. These typically produce part of the “answers” that the project and product are looking for (some of these “answers” may come in non-artifact form also, eg verbal reporting).

I assert that this is the view of the non-tester in many situations - whether from a project or product (owner) perspective. This is no bad thing in itself, but... After a number of repetitions it can happen that this is /the/ model for delivering on requirements -> that a project is set up we turn the handle with a number of given activities, produce a number of given artifacts and then deliver to the customer.

Some problems with this:

• The focus is on the containers - the activities and artifacts - rather than the contents. The contents (or content needs) should dictate the needed activities and artifacts, not the other way around*.

• The traps associated with Taylorism - splitting activities into unitary items (to optimize and measure) becomes a goal in itself. These look like targets, but can lead to local optimizations and process gaming (skewing) due to measurement goals.

• The model is used as an exemplar as it has succeeded with a (several) project(s) already.

• Familiarity with the model (system) is thought of as a closed system - meaning it’s processes and flows are defined and known to the extent that they are considered (by their practitioners) as complete.
• The previous two points contribute to the model becoming a best practice.

So, is this model wrong?

It depends... on how the model is used.

The activities and artifacts are not wrong in themselves. I am not saying these artifacts
cannot be produced or that these activities cannot be done.

The usage of it can be illustrative - or informative versus normative - to show what a “general” flow might look like. But I think there are several alternatives. It’s important to start at the needs end of the equation and build components that are congruent with those needs.


Mapping Information Flow


The second model:

Model #2

When I first presented the above model (#1) and contrasted it with an alternative (#2), in Sept 2012 to a group of my peers, I made it on a large whiteboard with one displayed above the other. I did this to point out that activities and artifacts (model #1) could be mapped into this model.

The second model takes the view of not encapsulating customer requirements, but tries to work with (analyse) what information the customer needs about the product (that they will receive). Part of this can be some traditional requirement labelling, but can also be thinking at a very early stage, “how will I demonstrate or show this to a customer”. This breaks down into a:

• Needs picture. This might be something close to a traditional requirement description - the items on a delivery checklist to a customer.

• Risk picture. What risks are associated with this product, due to it’s composition, development and it’s context (including legacy system).

• Information picture. What elements are important to tell the story to the customer and how?

Phasing: Sometimes products have dependencies to other product or third-party components and so phasing in time can be needed (from a project perspective).

The needs picture can be decomposed into information elements related to the new feature, the new system and the aspects of the legacy system.

The important thing to bear in mind with these elements is that they must be congruent to add to the bigger picture - that they add value to the element in themselves (eg test of a feature in isolation), add value to the whole (the new system) and their context (to the product that the customer will use - which might be different from a generic system).

This model also introduces elements of design thinking, see references, to move from the first model (how-> activity + artifact based) to a focus on the why and what value elements to help the needs pictures (of the project and the customer).


Usefulness of this model?

The information model has the purpose of identifying value to the business. We work with the information goals and elements that add value to the business. In this way, we align the development activity (and artifacts) to business goals. Then, maybe we devote less attention on activities (and artifacts) that don’t contribute towards a business goal (ie they add no information that is needed).

From a tester and testing perspective:
An aim with this model is to focus the daily activity wrt answering the question: “What information (and so business value) will my current task add for the customer.**

It forms a chain between daily activities and end goal. This could make reporting about testing (especially status reporting) easier or more accessible to stakeholders. The stakeholders know which thread of business value we are working on and how it contributes to the end goal.

It focusses reporting on information (business) value rather than something that is more abstract, eg test cases.

From a business and organisational perspective:
This is helpful in illustrating the difficulties and complexities of good testing. That requirements are not “things” in isolation, that the value chain in the whole development of the product is should be visible, to illustrate why something is being done or what omitting something might mean. It’s also a useful tool to show why “just test that” isn’t a necessarily simple task.

Background

Communication about testing. Since early 2010 I’ve been looking at different problems (traps) associated with communicating about testing. See references.

Before Let’s Test (May 2012) I had been thinking about communication about testing, especially to stakeholders and other interested parties. During Let’s Test I asked a number of questions related how testers should present information between different layers in the organisation -> ie that models for communication and analysis don’t translate so well between different layers of an organisation (it’s not their purpose, so how do we tackle that?)

In May 2012 I took a course on Value Communication - whilst the focus of it was about communicating key points (especially for execs) I started thinking about information flows in testing.


Work in Progress and Feedback

I presented these two models to James Bach as we sat on a train to Stockholm after SWET4. After some discussion about which problems I was trying to look at he commented that I was debugging the information flow. Yes, that was definitely a major aspect, especially when considering the perspectives of the non-testers in this flow, ie how non-testers perceive and interpret information flow to/from testers.

This model is a work in progress, elements in it and its applications. All comments and feedback are very welcome.

Notes

[*] Contrast this activity model with the idea of a test case - which can be thought of as an instantiation of a test idea with a given set of input parameters (which include expectations) - then when the test case is used as an exemplar it becomes a test idea constraint! In this sense the model (of activities and artifacts) constrains the development process. From a testing perspective it constrains the testing. 

[**] Customer, in this model, can be an external or internal customer. It could also be a stakeholder making a release decision.


References
[] Test Reporting to Non-Testers http://www.slideshare.net/YorkyAbroad/test-reporting-tonontesters2010
[] Challenges with Communicating Models http://testers-headache.blogspot.com/2012/07/challenges-with-communicating-models.html
[] Challenges with Communicating Models II http://testers-headache.blogspot.com/2012/07/challenges-with-communicating-models-ii.html
[] Silent Evidence in Testing http://testers-headache.blogspot.com/2012/03/silent-evidence-in-testing.html
[] Taylorism and Testing http://testers-headache.blogspot.com/2011/08/taylorism-and-testing.html
[] Wikipedia: Design Thinking http://en.wikipedia.org/wiki/Design_thinking

Testing Chain of Inference: A Model

Pass? What's a pass?

I've heard these questions recently and wonder how many think about this. Think about automated test suites: isn't a pass, a pass, a pass? Think also about regression testing - especially where tests are repeated - or where they are repeated in a scripted fashion. 

The following is an exploration of traps in thinking about test observations and test results. To help think about them I tried to model the process.  

Gigerenzer illustrated a chain of inference model, ref [1], which he used to illustrate potential mistakes in thinking when considering DNA evidence in legal trials.

Observed Pass -> Confirmed Pass Model

I used the chain of inference idea to apply to a "test pass":

Elements in the Model

Reported Pass. This is one piece of information on the road to making a decision about the test. It could be the output of an automated suite, or a scripted result. 

Product Consistency. Think: What product are you testing and on what level? This influences how much you will know about the product (or feature, or component, etc). Which features or services are blocked or available? Third-party products in the build - are their versions correct? Is any part of the system or product being simulated or emulated, and how does this reflect what you know about the product? What provisioned data is being used, and does that affect what you know about the product? What risk picture do you have of the product, and has the test information changed that?

Test Consistency. Think: Have my test framework or harness changed, and does this affect what I know about the product? Has the product changed in a way that may affect the testing I do? Has there been any change in typical user behavior, and if so, does this affect the information you think you're getting from the test (test observations). Is any behavior simulated, or not known, in the test?

Confirmed Pass. Think: Taking into account the test observations, the information about the product and test consistency can the result be deemed to be "good enough"? If not, is more information needed - maybe testing with different data, behavior or product components? (Note, "good enough" here means that the information from the test is sufficient - the test/s with the data, format, components and environment. It is not a template for an absolute guide (oracle) to a future "pass" - that would be falling for the representation error…)

Representation Error. A short cut in the evaluation of the result is made, specifically excluding information about product or test consistency. Think: This is equivalent to telling only a partial story about the product or testing.

Lucid Fallacy. Ref [2]. This is the simplification of the decision analysis (about the reported pass) to "a pass, is a pass, is a pass". Think: An automated suite, or a script, that produces a pass - that's a pass right? The assumption that an observed "pass" is a real "pass" is a simplification of the judgement about whether a test result might be good enough or not. 

Proxy Stakeholder Trap. Think: "Pass? That means the product is ok (to deliver, release, ship), right?" It is quite fine for a stakeholder to set their own "gate" about how to use a "pass". The trap here is for the tester who makes the jump and says, "the pass means the product is ok to…" - this is the trap of reading too much into the observation/s and transforming into a proxy stakeholder (whether by desire or accident).

This model helps visualize some significant ideas.

1. A reported pass is not the same as a confirmed pass
2. The labelled lines show traps in thinking about a test - shortcuts in thinking that may be fallible.
3. There is no direct/immediate link between a test being "OK" and a product being "OK"
4. Observing an automated/scripted "pass" implies that there needs to be a "decision or judgement" about a "pass" - to decide whether it is good enough or not.

Q: Are all reported passes confirmed in your environment? 

Test Observation -> Test Result Model

The above model can be used as a basis for thinking about the chain of inference in test results, reports and extending this thinking to exploratory approaches.

Elements in the Model

Test Observations. These are the notes, videos, memories or log files made during the testing of the product. They may cover the product, the test environment, the test data used, the procedures and routines applied, feelings about the product and the testing, anomalies and questions.

Product Elements. Think: Putting the components of the product into perspective. This is the product lens (view or frame, ref [3]) of the test observations. Are the test observations consistent with the product needs / mission / objectives?

Testing Elements. Think: The parts of the testing story come together - this is the test lens (frame) on the test observations. Are the test observations consistent with any of the models of test coverage? Do they raise new questions to investigate? Are there parts that can't be tested, or tested in the 

Project Elements. The aims of the project should ultimately align with those of the product, but it is not always the case. Therefore, the project lens (frame) needs to be applied to the test observations. Are they consistent with the immediate needs of the project?

Test Results. Think: Pulling the test observations together - both raw and with the perspectives of the project, product and testing elements - compiling and reflecting on these. Is there consensus on the results and their meaning or implication? Has the silent evidence, ref [4], been highlighted? What has been learnt? Are the results to be presented in a way tailored for the receivers?

Context-free Reporting. Think: When the thinking about the test observations and results are not consistent with project, product or testing elements then the context of the result is not involved. The result becomes context-agnostic or a context-free report.

Individual -> Multiple Test Observation Model

Combing the above two models gives:

Summary

This has been a look at some traps in thinking that can occur when thinking about test observations, test results and when implicitly making decisions about those.

1. A test observation and a judgement about a test observation are different.
2. A test result and a decision about a test result are different.
3. A test result and a feeling about a test result are different.

References

[1] Calculated Risks: How to know when numbers deceive you (Simon & Schuster, 2002, Gigerenzer)

[2] Wikipedia: Lucid Fallacy: http://en.wikipedia.org/wiki/Ludic_fallacy

[3] The Tester's Headache:  Framing: Some Decision and Analysis Frames in Testing

[4] The Tester's Headache: Silent Evidence in Testing

The Documented Process Trap

"I can't do this work well without a way of working..."


"If only we had a document describing how people should work they'd be much better...."


"It's very difficult to do this without a documented process..."

Any of these sound familiar? The third one might be true, but that says more about the person (or the conditioning they've been exposed to) than the process. It can be easy to get conditioned into this way of thinking.

Trap?
This is the belief that something can't be done unless it's documented. But where does this belief come from?

Beware, it's a trap.

It's a self-fulfilling trap. The more you hear it, the more some people tend to believe it, then it disables them so that they can't operate unless they have a documented process in front of them.

It can even become contagious.

Once you have a document (sometimes called a process description or way of working) telling you how to do things or what to do in what order, you start thinking about other aspects of your work that needs a document describing it...

Beware of this trap.

Note, sometimes process critical steps need ordering and documenting - this trap is not saying anything documented is bad, it's a warning about approaching new or original problems.

A logical extension of this is that test ideas, sometimes called test cases, are described in advance - so that we know what to do. Sound familiar?

Bad to document test ideas?
No. Awareness of this trap is stating that it is dangerous to pre-determine all steps to a solution before the solution is known. It can be very useful to write down a set of ideas in advance, but don't let the problem solving approach stop there - before you know if you're on the right track.


What about decisions or criteria for decisions? Should they be documented in advance so that you don't make mistakes?

Why document things in advance?
As aide-memoire, as checklists, or as decision tools? All of these are fine when used in the right spirit - as a help, but not necessarily definitive. Unfortunately, the move to something being definitive can be quite short. From aide to "standard" to "standard practice", or even "best practice". This is connected to the way we create norms, and then the dreaded cognitive ease (which we're all susceptible to).

Kahneman, ref [1], states that norms are built by repetition - when we see something "work", we start to think of it as "how it works". This is complicated by something called cognitive ease - once we start to "see how it works" it can become a natural default as it is much more difficult to think each time, "is that really how it works?".


Hypothesis
I proposed a hypothesis recently:

Many people naturally approach new problems in a heuristic manner, but due to repetition or familiarity this becomes a "standard" and the heuristic approach is lost or forgotten.

When we look at new or original problems we often don't know how to tackle them - we might take out our "toolkit" of approaches and try out some different ideas, we might think down different solutions depending on time, customer needs and available tools and people. We evaluate the results of some of those ideas, and either change or continue. There is rarely a pre-defined way to approach a problem.

Polya, ref [2], warns not to mix up heuristic reasoning with rigorous proof.


Awareness
Awareness of a trap won't avoid anyone falling into it. However, it is the first step in progressing away from unconscious incompetence, ref [3].

The next time you are presented with a new or original problem, ask yourself "am I thinking originally about this" or am I following a "standard" (or accepted practice) and should I question that.


References
[1] Thinking, Fast and Slow (Kahneman, 2011, FSG)
[2] How to Solve It: A New Aspect of Mathematical Method (Polya, 2004, Princeton University Press)
[3] Wikipedia: http://en.wikipedia.org/wiki/Four_stages_of_competence

Silent Evidence in Testing

A: All the test cases passed!
B: But the feature does not work for the customer...

I expect this, or something similar, is not a totally new type of statement to many people. Were the test cases sufficient, was there even any connection between the test cases and the feature, was some part of the scenario/system/network simulated and wasn't accounted for, etc, etc. The possible reasons are many!

Silent Evidence is a concept to highlight that we may have been working with limited information or that the information we gather may have limited scope for use. The effect is to act as a reminder to either (1) re-evaluate the basis of some of the decisions or (2) to act as a warning flag that the information is more limited than we want and so to warn about the usage of such information.

When I started reading Taleb's Black Swan, ref [1], at the end of 2009, I started seeing many similarities between the problems described and my experiences with software testing. One of the key problems that jumped out was to do with silent evidence, from which I started referring to silent evidence in testing.

Silent Evidence
Taleb describes this as the bias of focussing on what is presented or visible. He cites a story from Cicero:

Diagoras, a nonbeliever in the gods, was shown painted tablets bearing the portraits of some worshippers who prayed, then survived a subsequent shipwreck. The implication was that praying protects you from drowning. 

Diagoras asked, “Where are the pictures of those who prayed, then drowned?”

WYSIATI
Recently, whilst reading Kahneman's latest book, ref [2], I found that his description of WYSIATI extends the idea of silent evidence. WYSIATI stands for "what you see is all there is" - and this deals with making a decision on the available information. Many people make decisions on the available information - the distinction here is that it becomes an assumption that the information is good-enough, and not necessarily investigating if it is good-enough. This is manifested by associated problems:

• Overconfidence in the quality of the information present - the story is constructed from the information available. From a testing perspective - think about the elements of the testing not done and what that might mean for product risk. Do you consider any product risks from this perspective?
• Framing effects due to the way in which the information is presented. If a stakeholder associates a high pass rate as something "good" (and being the only piece of information to listen to) then starting the testing story with that information may be misleading for the whole story. See ref [3] for other examples of framing effects and silent evidence.
• Base-rate neglect: This is the weighting of information based on what is visible. Think about counting fault/bug reports and using them as a basis for judging testing or product quality. Without context the numbers give little useful information. Within context, they can be useful, but the point is how they are used and to what extent. See ref [3] for more examples. 

Silent Evidence: ENN
I use the term silent evidence from an analysis viewpoint, typically connected with evaluation of all the testing or even connected with root cause analysis - to examine assumptions that gave rise to the actions taken. (I use this not just for "testing problems" but also for process and organizational analysis.) This is useful to find patterns in testing decisions and how they might relate to project and stakeholder decisions.

I use the acronym ENN to remind me of the main aspects. They fall into the "before" and "after" aspects:

Excluded (before)
Not Covered (after)
Not Considered (after)

E: Excluded
This is the fact-finding, data part of the report on the information available and decisions taken before test execution. This considers the test scope or parts of areas that we rule out, or down-prioritize during an analysis of the test area. It can also include areas that will be tested by someone else, some other grouping or at some other stage.

Picture a functional map that modifies a feature, I might conclude that another feature which is mutually exclusive to the one being modified (can't be used/configured simultaneously), deserves much less attention. I might decide a sanity test of that area sufficient, test that the excluded interaction can't happen and leave it at that. (Remember, at this point I'm not excluding expanding the test scope - this might happen, depending upon the sanity, interaction or other testing.)

An extreme case: Testing mobile terminals for electromagnetic compatibility or equipment cabinets that can isolate the effects of a nuclear pinch are not done every time new software is loaded. Excluding this type of testing is not usually controversial - but do you know if it is relevant for the product?

A more usual case: There is a third-party-product (3PP) part of the system that is not available for the testing of your product. Whatever we say about the result of the testing we should make the lack of the 3PP visible, how we handled it (did we simulate something or restrict scope?), what risks the information might leave and if there are any parts to follow-up.

It's never as simple as, "the testing was successful"or "the test cases passed". This is drifting into the dangerous territory of "context-free reporting".

It's not just applicable to software test execution - think about a document review - a diagram has been added and I might consider unrelated parts of the document to only deserve a quick look through. Should I really check all the links in the references if nothing has changed? This is a scope setting to do with the author - they might check this themselves.

N: Not considered
This is the retrospective part of the report or analysis. What was missed or overlooked that should have had more of the available attention? What part of the product story didn't get presented in a good way?

This is very much a "learning from experience activity" - ok, how did we get here and, just as importantly, does it matter? Some problems are more fundamental than others, some are organizational (a certain process is always started after another, when maybe it should start in parallel or be joined with the first), some are slips that can't be pinpointed. The key is to look for major patterns - we are all human, so it's not to spend so much time to root out every little problem (that's not practical or efficient) - but to see if some assumptions get highlighted that we should be aware of next time.

Example: A customer experiences problems with configuration and provisioning during an update procedure. The problem here is not that this wasn't tested to some adequate level, but that it was tested and the information from that analysis resulted in a recommendation to stop/prevent such configuration and provisioning during upgrade. This information didn't make it into the related instruction for the update (and maybe even including a mechanism to prevent configuration).

In the example, the part not considered was either that the documentation update wasn't tested or highlighted to show potential problems with update so that a decision about how to handle the customer updates could be made.

N: Not covered
This is similar to the case for excluded testing - but it is usually that we had intended to have this part of the scope included. Something happened to change the scope along the way. It is very common for a time, tool or configuration constraint to fall into this category.

The not covered cases usually do get reported. However, it is sometimes hard to remember the circumstances in longer, drawn-out projects - so it is important to record and describe why testing decisions in scope are made. Record them along the way, so that they are visible in any reporting and for any follow-up retrospective.

The problem even occurs in shorter projects where the change happens early on, then the change gets "remembered" as always being there.

Example: Think of a particular type of tool needed to fulfill the scope of the testing for a product. For some reason this can't be available in the project (becomes visible early in the project). The project then starts to create a tool that can do part of the required functionality that the first tool would've covered. Everyone is very happy by the end of the project. But what about the 'hole' in functionality that couldn't be covered? Does this still affect the overall scope of the project and is this information being made visible?

And finally
Thinking about silent evidence helps me think about these traps, and maybe to even avoid some the next time around. Being aware of traps doesn't mean you won't fall for them, but it does help.

So, use the concept of silent evidence (or even WYSIATI) to keep your testing and product story more context-specific and less context-free.

References
[1] The Black Swan: The Impact of the Highly Improbable (Taleb; 2008, Penguin)
[2] Thinking, Fast and Slow (Kahneman; 2011, FSG)
[3] Test Reporting to Non-Testers (Morley; 2010, Iqnite Nordic)

The Linear Filter Trap

Or: Illustrating Systems Thinking with Proximate and Distal Analysis

I was having a discussion the other day where we were looking at some problems and discussing whether it was sufficient to treat symptoms... I remarked that by treating symptoms, rather than root causes [notes n1], and not allowing time to find the real problems then we would, in many cases, be fooling ourselves [notes n2].

Why? (I was asked)
People like to solve problems that they can see - which means we have a tendency to "fix" the problem we see in front of us. This occurs more often if the problem appears to have a "straightforward" fix -> I think of this as a form of cognitive ease in action. Digging for root causes is a challenging activity, and we sometimes want to believe that the cause we identify is good enough to fix. For another example of cognitive ease, with best practices, see ref [2].

An illustration of this - that I have seen in one form or another - is that we settle for the first solution without understanding (or trying to) the root cause. There is no guarantee that fixing a symptom will make the problem better. Many times, the problem improves for a while, but then re-occurs in another form. Now there is a "new" problem to solve, which usually has the same (or similar) root cause, so from the system perspective it's ineffective [notes n3].

Ineffective, because many problems in processes and organisations are often non-linear, but we often try to solve "linear" problems, and...


Linear Filter
I expanded, that I think of this from a systems thinking perspective as applying a "linear filter" to a "non-linear system".

What? (I was asked again) Linear vs Non-Linear?
Non-linear -> multiple interactions affect the output vs linear -> the output is directly proportional to the input. So the application here is that there are usually multiple causes for a problem -> when I perform an assessment after a root cause analysis (RCA) activity I usually take them root cause by root cause, in the order that we think will have the biggest impact on improvement.

Ok, time for a....

A Real-life example
A fault (bug) report was written by a customer -> initial RCA shows that some "test" was not executed that would (in theory) have caught the problem. This is a symptom and a "linear" view of the problem. The "linear filter trap" is to then consider this as the root (or most important) problem. [notes n6]

Digging deeper shows that the team had it's priorities changed during execution, to make an early drop (resulting in some negative and alternative use-cases being delivered later). This, in itself, is not a problem but the communication that was associated with the "early drop" didn't reflect the change.

In this case, some of the underlying problems were a set of communication issues:

• Partly in the team connected with their story [notes n4] (especially their testing silent evidence [notes n5]), 
• Partly connected with the stakeholder that changed priorities and may have had a duty to follow the change of expectations through the delivery chain and what that might mean at those different stages, and 
• Communication with the customer to ensure that they are aware of the staged delivery, and what means to them.

Another example of a root cause analysis can be found in ref [3].

And finally
Tackling and fixing symptoms is a very natural activity - very human. But it is not always enough. Sometimes it is enough - it depends, of course, on the scale of the problem and the cost of investigating the underlying problems and tackling those. Sometimes, the underlying problems cannot be fixed and it is sufficient with easing the symptoms.

But I believe, as good testers, it is important to understand the difference between symptoms and root causes, especially where it affects either the testing we do or affects the perception of the testing we do. This is important where a perception of "testing or testers missed something"... So,

Be aware of the linear filter trap!

Notes
[n1] In philosophy and sociology, root causes and symptoms are usually referred to as distal and proximate causes, see ref [1] for more background.

[n2] Slightly naughty of me, playing on the fact that most people don't like to think that they are fooling themselves, but that's a different story...

[n3] The times when it might be effective are when we (some stakeholder) is prepared to take the cost of fixing some problem now. A problem here is that stakeholders that are project-driven have, by the nature of the task, a propensity to see only as far as the end of the project. A product owner may have a different perspective - bear this is mind when someone is deciding whether it's a project problem or a product problem - or even a line organisation problem.

[n4] Story here means the story about the product and the story about the testing of the product.

[n5] Here, testing silent evidence refers to the elements not tested and thus not reported - their significance is assumed to be not important. For further background see ref [4].

-->edit-->
[n6] I should add that the problem with the trap in this example is that I have seen this in the past trigger one of two responses: (1) A perception that the testers are at fault, which becomes a myth/rumour with a life of its own ; (2) A knee-jerk reaction to implement some extra oversight of the test displicine or team as a whole -> in the worst case it becomes a desire to introduce some additional "quality gate". This is a good example of when reacting to the perceived symptom is both ineffective and counter-productive for the organisation.

References
[1] Wikipedia: Proximate and ultimate causation
[2] The Tester's Headache: Best Practices - Smoke and Mirrors or Cognitive Fluency?
[3] The Tester's Headache: Problem Analysis - Mind Maps and Thinking
[4] The Tester's Headache: Mind The Information Gap

The Hindsight Trap

Root cause analysis (RCA) activities are an important part of many retrospectives - whether as part of  a team activity, analysis of test reports, analysis of fault (bug) reports or as standalone activities to investigate organizational and process problems.

I read Rikard Edgren's post the other day, on some good ISTQB definitions, here, and it reminded me of a common trap that people sometimes fall into with documented claims or requirements. (I hasten to add that Rikard didn't appear to fall into this trap!)

The most common form of this trap in testing circles (where documentation is involved) usually crops up as a question, "why didn't you/someone spot this problem earlier (as it's specified in document X)", or statement, "this requirement should been tested because it's specified in document X".

The Trap
The problem with these types of statements and questions is that there is an important assumption being made: They are using documented specifications or claims in their current form and assuming that was the same form that was an "input" into some tester's/team's mind.

In my experience this assumption occurs more often when:

• The one making the analysis doesn't understand the problems (challenges) that testers face with making documented and undocumented requirements visible.
• The one making the analysis assumes that testers have the same information as everyone else.
• The one making the analysis assumes that test problems are "owned" solely by testers. 

Trap Avoidance
Ask yourself at least one question (but preferably more) about documented claims / specifications:

• Was this information available to the tester at the time of testing?
• Silent evidence? Was this information available but not used/tested for a particular reason?

In the first bullet the onus is on the one making the analysis. In the second bullet the onus is on good test reporting (story-telling) - highlighting areas not done (for whatever reason),  more on silent evidence in ref [1].


Further Analysis
Potential for these problems exist where code design and test are separated in different teams. But, more generally, they exist where communication problems (or lack of transparency) exists - I see these most often where parts of teams receive external requests that isn't "hand-shaken" within the team.

One way to help make these problems visible (whether from an RCA or team retrospective viewpoint) is to consider the frames that different roles have with respect to the problem, how does the developer look at the documented claim and how does the tester - any external influence that changes their priorities? More in ref [2].


So, being knowledgable after the fact (with hindsight) usually misses potentially significant parts of the story. And remember, a statement is very often not the whole story -> good story-telling takes effort!

Hindsight trap - sound familiar?




References

[1] Mind the Information Gap
[2] Framing: Some Decision and Analysis Frames in Testing