Best Practices - Smoke and Mirrors or Cognitive Fluency?

" #softwaretesting #testing #cognition "

I saw a post on the STC, here, about best-in-context practices. I started thinking about 'best-in-show' (a la crufts) and best practices in general.

New ideas, terms or phrases usually trigger the same question: 'what problem is this trying to solve?'

BP in General

Best practices. Why tout them? It's a sales gimmick - 'here's my way of doing it and I think everyone should do it. And I can show you why it will solve your problems too'. 

It's a sales pitch - and sometimes the best salesman wins. Or, it's a marketing gimmick - if you create enough exposure to the 'best practice' then it gets assumed to be 'best' because there is so much discussion about it.

Systems Theory

Applying 'systems theory' - creating a best practice is like creating a local optimization - there's a danger it will sub-optimize for the whole system, ref [5]. Best practices are inherently localized.

Think of the Spanish Inquisition (the real one and the monty python version) or '1984' and the thought police, ref [7]. These organisations would probably have been thought to be quite effective to the ones that created them - that's the paradox of local optimizations, there's always at least one angle from which they look very good (or even 'best'). 

Good for the whole though? Here, of course, a lot depends on your idea of good and the frame in which you think of the problem - and the volume of your voice or marketing.

Unfair comparisons? Problem with framing?

Thinking in frames and framing the problem and choice! A problem with 'best practices' could be that:

• The problem has not been thought through to achieve a consensus of what practice to adopt - then someone makes a choice
• The problem and it's nature is not understood (ie in the next project the problem is different - this is hard work… - let's stick with what we have)
• A practice is used and gets labelled 'best' afterwords (maybe not by the original practice decision maker)

In the third case, it's use implies it's a 'best practice' even if it's not. Think from a typical/average manager's perspective - why would they use anything that wasn't best, so if it's in use now and isn't causing huge pain (or seems good enough) then they can be tempted to call it a best practice.

A Snapshot in Time and Good Enough

Personally, I think of best practices as ephemeral - a dragonfly that lives for a short time, and then it's gone - ie 'best' is like taking a still picture with a camera - there (at that timestamp) it's best - but it might not apply anymore….. That snapshot might not be 'best' if I started comparing with other snapshots…..

So, why search for 'best'? Why not search for 'good enough'? Or 'good enough for the problem in front of me'? To me, that would imply active thinking…. But achieving consensus about which practice to use might be simpler than you think - but working on your assumptions is needed. For example:

• What problem is trying to be addressed with a best practice?
• Is it a project manager/company that wants to create a 'standard' that they don't want to maintain?
• Is this a money-saving approach? Again maintenance.
• Is it telling the people not to think?

Maybe by paying people not to think (lower paid) then a practice needs to be adopted that is low-maintenance. This seem to be a dangerous game… 

Perhaps companies tolerate them as long as the black swan event can't be traced to their choice of practice (or reification of it being 'the best'). Maybe they genuinely don't realise any better. Maybe the difference in practices is judged to be too small to not warrant the need for constant evaluation or judgement. (I suspect this is a default - coupled with cognitive fluency, see below)

But if this was a default option shouldn't there be less advocating/marketing/discussion of best practices? Well, progress implies (to many) re-invention or creation of new ideas, therefore 'discovering' a new best practice is quite marketable.

Smoke and Mirrors

Is the idea of a 'best' practice an illusion? Software development (including testing) is knowledge work - what's the 'best' way to think? 

Is it not the case of an application of an industrial metaphor to knowledge-based work? Mmm!!

Is it a problem with language?

• "Practice makes perfect." 

Implies there is a 'best' - this is also an example of cognitive ease (see below) and will be more easily remembered as being a 'good' guide.

• "Practice does not make perfect. Only perfect practice makes perfect."

Not achieving 'the best' is probably politically incorrect in business circles - so there is pressure to say this or that is 'best'. But, remember, this is knowledge work.

Think of any award or recognition in the sciences - we don't say 'X is the best economist/physicist' - they are usually identified for their contribution. In the same way, we should be particular about any 'practice' that we use - what is it good and not so good for??? 

If you can't think of issues with the practice then you probably haven't thought hard enough.

Is it a cognitive problem?

The sticking point is 'best' - it's a superlative. It's natural that any manager would like to advocate and advertise that they are doing the 'best' possible. It becomes even worse if they have had a 'best practice' in the past - then it is harder to move away from such a practice as this is hard work convincing their peers why this is necessary.

Behavioural Economists and Psychologists also have a related theory - it's called cognitive fluency, ref [2], or cognitive ease, ref [3] - and it's the condition when something is easier to think about affects our choice. It has been noted that when something is easier to think about then it might be more readily believed to be true (or correct, or remembered - depending on context) - ie there builds an assumption (bias) that it is true (correct or something seen before). 

This ease of thought can be obtained via familiarity (repeat messages), bolder and easier to read type and even colour. 

So, if anyone (manager/stakeholder) has ever had success (however they gauge that) with a best practice in the past, then they are more likely to think of best practices as being a good thing, and their particular best practice in particular.

It is much easier to take an off-the-shelf practice for use in the next project rather than think it through and check whether it is good enough or not. 

The opposite of cognitive fluency - cognitive disfluency - also holds true, it acts as a warning signal. So, forcing someone to re-evaluate what a 'good enough' practice is is always going to be harder than taking a ready-made practice off the shelf.

Assumptions

'Best practices' don't usually display their magnitude of pre-requisites, conditions, exclusions and terms of use. It's like the various "terms of use" that come with many SW applications - many don't read them. 

Why, then, should a project manager read and be aware of all the conditions of use for a 'best practice'? It's hard work. Managers/stakeholders usually assume (or hope) that knowledge workers are 'doing their best' under the circumstances.

And finally...

The next time someone talks about a best practice, go easy on them - they are following an evolutionary pattern ('if it's familiar it hasn't eaten me') therefore, the onus is on us to highlight the shortcomings of the particular practice (if there are any to be found) or judging why it's fit for purpose. 

That activity is important - it's like going through a pre-flight checklist - you see the potential warning signs when you can still act upon them!

Also, never present anything as 'best' for knowledge work - cognitive fluency will mean it becomes a best practice next time. Avoid superlatives (best) and indications of certainty or insurance, ref [4].

References

[1] Framing: Some Decision and Analysis Frames in Testing

[2] http://www.boston.com/bostonglobe/ideas/articles/2010/01/31/easy__true/

[3] Thinking, Fast and Slow (Kahneman, 2011, FSG)

[4] The Certainty Effect and Insurance

[5] Thinking in Systems: A Primer (Meadows, 2008, Chelsea Green Publishing)

[6] Flikr: Crufts Best in Show Roll of Honour

[7] http://en.wikipedia.org/wiki/Thoughtcrime

[8] http://www.softwaretestingclub.com/profiles/blogs/my-take-on-best-practices

Our survey said...

" #interpretation #fun #context #cognition "

Whilst compiling material for some other work I stumbled across some old Family Fortunes and Family Feud 'funny/strange' answers on YouTube*.

I've recently being doing a lot of thinking around framing, ref [5], and the problems it can cause and solve and I started thinking about different causes for the unexpected answers.

For communication analysis I use two types of exercise, (1) frame analysis and (2) word and meaning substitution.

Frame analysis

• What are the aspects that might be important to each person involved in the communication? This usually revolves around situational context of either the one asking the question (presenting the problem) or the one answering the question (presenting a solution). Here there is scope for a range of cognitive and interpretation mistakes.

Word and meaning substitution

• A well-known example of this is the "Mary had a little lamb" exercise, described in "Are you lights on", ref [1], and is a demonstration of how changing the emphasis of a word in a sentence, or replacing a word with a similar meaning (from a dictionary or thesaurus), can change the meaning of the sentence. Therefore, if both parties in the communication intend different word emphasis (in word placement or interpretation) then there is a possibility of confusion.

So, what can appear as confusing or even amusing answers can, with the right perspective, have a certain logic. In the list** below I've made an attempt at finding the perspective behind the answer, in red.

The questions are typically prefixed with "We asked 100 people to name..."

Q: Something a husband and wife should have separate of

A: Parents

Logical answer(?) but maybe not along the intended lines of the questioner.

Q: A planet you recognize just by looking at a picture of it

A: The Moon

Confusion of definition of planet with 'celestial body' (something in space with an orbit)

Q: A month of spring

A: Summer

Slip of the ear, of->after(?), ref [4]

Q: A word that starts with the letter Q

A: Cute

Q. Name a part of the body beginning with 'N'

A. Knee

Phonetic interpretation

Q: The movie where John Travolta gave his most memorable performance

A: The John Travolta Biography

'Most memorable performance' to the questioner had a potentially different meaning. To the answerer either it was interpretted as a film where he featured the most, or he wanted to give an amusing answer.

Q: Something you wouldn't use if it was dirty

A: Toilet paper

Amusing and logical answer(?)

Q: A signer of the Declaration of Independence

A: Thomas Edison

Slip of the tongue, specifically noun substitution, ref [4]

Q: Something that comes in twelves

A: Dozens 

Could be logical interpretation but not something the questioner was intending(?)

Q: A sophisticated city.

A: Japan

Misinterpretation (or even slip of the ear) of city for destination.

Q: A kind of bear

A: Papa Bear

Recency effect(?) - had recently been reading or exposed to children's stories (?)

Q. Name a number you have to memorise

A. 7

Misinterpretation of 'memorise' as 'favourite' or 'memorable'(?)

Q. Name something in the garden that's green

A. Shed

Context-specific to the answerer(?)

Q. Name something that flies that doesn't have an engine

A. A bicycle with wings

'Logical' and specific answer - but the questioner could have maybe clarified the question with a 'commonly known item'.

Or, recency effect - flugtag, ref [6].

Q. Name something you might be allergic to

A. Skiing

'Alergic' -> 'don't like'(?)

Q. Name a famous bridge

A. The bridge over troubled waters

Interpreted as 'something well-known with bridge in it'(?)

Q. Name something you do in the bathroom

A. Decorate

Specific to the answerer's context.

Q. Name an animal you might see at the zoo

A. A dog

Generics. Potential that the answerer has not interpreted the the question as 'generally seen and residing in the zoo'.

Q. Name a kind of ache

A. Fillet 'O' Fish (?)

Brain-freeze or 'slip of the ear'(?)

Q. Name a food that can be brown or white

A. Potato

Answerer framed the question as a food which could be presented as brown or white(?)

Q. Name a famous Scotsman

A. Jock

'Slip of the ear' -> 'a common nickname'(?)

Q. Name a non-living object with legs

A. Plant

Maybe thinking of a plant on a plant stand(?)

Q. Name a domestic animal

A. Leopard

Misinterpretation of 'domestic'(?)

Q. Name a way of cooking fish

A. Cod

'way' misinterpreted as 'type'(?)

Analysis Notes

• Context - some answers are specific to the answerer and not the questioner. Example traps might be (1) Understanding and interpretation, (2) Word association problems or (3) Relating everything to ones own experience or circumstances.

• Recency effects, ref [3] - the interpretation associated with a word was used in a different context, giving a skewed answer. In testing this occasionally results in skewed emphasis of the risk determination - see tester framing problems in ref [5].

• Skipping and changing words in sentences - to actually hear a different question - sometimes grouped under 'slips of the ear'. In testing this might result in an incorrect solution application, similar to framing problems but can also be 'straightforward' slips that result in some faulty analysis - missing some key input parameter for example.

• Other framing effects can be caused by the previous question, previous answer or even some realisation that a previous answer was wrong/silly and so inducing more stress in the answerer.

• Stress can mean that sometimes when you're trying to react you don't actually listen to the whole message or question. This can be time pressure or other stresses. Be aware of this potential problem.

• Anchoring effects, ref [2] - focusing on a word and giving an association with that word (rather than focusing on the whole question). In testing this typically results in confirmatory testing.

• Generic statements can create confusion. These are generic statements as part of the answers - this is where the question can be confused between giving an example of a specific kind and categorizing the answer into a grouping. Opposite of the answerer-specific problem. More on this in another post...

• Don't rule out brain-freezes either - these can be multi-word substitution or paragrammatism, ref [4], which result in nonsense responses.

And finally...

This is a good exercise and quite instructive for those working in software testing - it's a good illustration of how what might be seen as an obvious or simple answer can actually diverge from the expectations of the stakeholder or even customer.

Be alert for not just for confusing messages but also the potential for confusing answers. In this way you might know when to re-affirm your interpretation back to the stakeholder or customer.


References

* If you want to see the clips you can search youtube for "family fortunes answers" or "family feud answers" or "game show stupid answers".
** Lists compiled from

http://www.funny-haha.co.uk/Joke.asp?J=283

http://www.businessballs.com/familyfortunesanswers.htm

http://www.stupidgsa.com/american/family-feud/

[1] Are Your Lights On?: How to Figure Out What the Problem Really Is (Gause and Weinberg, Dorset House, 1990)
[2] Anchoring effects: http://en.wikipedia.org/wiki/Anchoring
[3] Recency effects: http://en.wikipedia.org/wiki/Serial_position_effect

[4] Um...: Slips, Stumbles, and Verbal Blunders, and What They Mean (Erard, Panteon, 2007)

[5] http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html

[6] http://en.wikipedia.org/wiki/Red_Bull_Flugtag

Testing: Do you train like an athlete?

" #softwaretesting #testing "

I just read this analysis piece on the BBC site about the pressures involved in sprinting, especially the start and run-up to the start.

As I read through it I found myself mentally ticking off links to the testing world:

• There is no perfect start
• Appearance and presentation is part of the message
• Pressure kills concentration
• Reacting at the right time
• Distractions affecting focus

No perfect start?
The interview contrasts two sprinters with different techniques and how their physical make-up presents different problems at starting and how they handle that. In the software development world this translates to there is no best practice. Every problem and solution is unique - what works for one athlete (or product) does not necessarily work for another.

Presentation of the message
In the article the example is given of Linford Christie displaying his superior physique to other competitors before a race. The intention was partly - I'm prepared and ready.

In the testing world, the message and form in which we give that message is important. Successful message styles are usually truthful, not unduly biased by numbers and consistent. Part of this goes towards building your brand.

Building this trust in your team and stakeholders is vital to successful story-telling.

Pressure kills concentration
All athletes respond and cope with pressure in different ways. Some pressure is exerted by other athletes, some is created by the athlete themselves (their own expectations) and their surroundings.

In the testing world - sometimes the pressure is external - created by teams or stakeholders, sometimes with unrealistic expectations of testing and sometimes because they don't handle/distribute the pressure so well.

This reminded me of the chapter in Pragmatic Thinking and Learning, "pressure kills cognition" which gives examples of overt pressure disrupting thinking. Be on the look-out for this - it's not the easiest thing to deal with when you're on the receiving end, but if anything, don't pass on the pressure - or at least understand what effect it might have.

Reaction times
In athletics the false start is feared - especially for the sprints. The athletes are coiled like a spring and are trying to react as quickly as possible. But, they don't want to react too quickly, and not too late either - they want something that's good enough for them. The interview demonstrates the difficulty here with the game of slapsies (look here and here).

In testing, the similarity might be when to raise or highlight a problem, call in help for the investigation. You don't want to be crying wolf for every issue, just as you don't want to be doing some lone investigation "too long" (and potentially getting stuck). Here is where pairing or having a colleague (tester or developer), that you can run something by, is very useful. Even at a daily stand-up meeting mention what you're currently investigating - sometimes there is someone who says "check X" or "have you got Y set in your configuration".

There is a routine and balance to find here - something that takes both practice and goes hand in hand with your message / brand.

Distractions affect focus
Athletes get distracted by the antics of other athletes, sometimes by the crowd or acoustics in the stadium. They have different ways of dealing with this - some try to get into and stay 'in the zone', some go and lie down and stare at something, whilst others continuously move around to avoid tensing up.

Distractions play a big part in any work environment also, as well as the ways we try and remedy them. It might be the problem of multi-tasking - getting many high-priority demands on your attention and not being able to decide which to devote attention to - or really how to stay focussed on the task at hand.

Sometimes, it's more efficient to simplify the problem into smaller component parts, and so have a feeling of manageability and being able to see progress quicker. Small wins keep attention and focus rather than long drawn-out slogs wading through mud.

Sometimes it's about removing distractions - don't check email for the next hour, close unnecessary browsers (with their flash animations that catch the eye - or use a flash blocker) - reduce the number of open windows to the minimum needed for the task.

Sometimes it's right to de-focus on the problem, step back and look at the wider picture. This helps you relate the problem to it's situational context, re-evaluate why you're doing something, maybe even bring in a fresh pair of eyes to help. This sometimes gives new information or re-affirms the original scope, then you can re-focus on the problem (with any new insights and information).

Are you a person that whilst talking to someone must always answer a phone call (no matter who it's from)? Or do you treat phone calls like someone coming up to you in the corridor whilst you're already in a conversation - usually they'd wait to interrupt your ongoing conversation - so why should it be different with a phone call? (The exemption here is if you're waiting for some urgent or important information which would lead you to interrupt the conversation.)

Awareness
There are lessons to learn all around - especially from non software testing disciplines. The key is to be able to recognise your potential problem areas - whether it's to do with message presentation, knowing when to react, handling distractions or being aware that pressure can have a detrimental affect on performance.

Awareness is an important first step in problem solving - whether you can solve the issue or not - understanding factors that affect your "testing performance" is key!

Framing: Some Decision and Analysis Frames in Testing

" #softwaretesting #testing "

What is a Frame?

The following is from Tversky and Kahneman's description of a decision frame, ref [1],:

We use the term "decision frame" to refer to the decision-maker's conception of the acts, outcomes, and contingencies associated with a particular choice. 

The frame that a decision-maker adopts is controlled partly by the formulation of the problem and partly by the norms, habits, and personal characteristics o f the decision-maker

When using a decision frame to analyze a problem and come to a decision they call this framing. So, I'll refer to a frame as relating to a decision (or analysis) frame.

Factors at play

As mentioned, many different factors affect how we analyze problems, including:

Temperament/Emotions

• Anger
• Happy
• Optimistic
• Pessimistic
• Tiredness
• Fear (e.g. of failure)

Experience

• Lessons from past situations - own experience and feedback
• What has been learnt recently
• Complacency due familiarity

Strategy

• Your own vs someone else's
• Aggressive
• Military campaign - lots of detailed planning
• Reactive

The factors and the weight given to them might be different for:

• Stakeholder view ("Upgrade needs to be reliable", "Of the new feature set only x is live in the quarter")
• Tester view ("Which risks are most important to look at now?")
• Developer view ("I did a fix, can you test that first?")

The stakeholder, developer, tester and any other role in the project has a set view on priorities and aims with the project - agendas maybe - and one challenge is in trying to tie these together, or at least understand the differences and how they impact our communication. They all may have similar product goals but their interpretations to their work may be different - their influences and micro-decisions will be different meaning that transparency in communication is important. 

But, there's a catch - the way we present information can affect its interpretation - depending upon the frame that a stakeholder is adopting.

Think of a frame as a filter through which someone looks at a problem - they're taking in lots of data but only the data that gets through the filter gets attention (the rest may end up in the subconscious for later or isn't absorbed), "I have my show-stopper filter on today so I don't notice the progress the team has made…" 

So, being aware of the potential different types of frames that each project member might have as well as some traps associated with frame formulation is important.

Stakeholder Frames

Might include:

• Emphasizing minimum time for product delivery
• Emphasizing short iteration times and delivering quickly
• Trying to minimize cost of product development (cost of testing?)
• Emphasizing future re-use of the development environment (tendency to worship automation?)
• Aiming for a reduced future maintenance cost

Tester Frames

Might include:

• Emphasizing the favourite test approach
• Emphasizing areas of greatest risk (to?)
• Emphasizing the last successful heuristic that found a show-stopper bug
• Emphasizing focus on the data configuration that found the most bugs the last time
• Emphasizing conformance to a standard over a different aspect of the product
• Emphasizing the backlog item that seems the most interesting
• Emphasizing widespread regression as "fear of failure / breaking legacy" affects analysis
• Emphasizing feature richness over stability

Note, in isolation, some of these frames may be good, but they might not necessarily be good enough.

Framing Problems in Testing

Functional Blindness or Selective Perception

Russo and Schoemaker called it Functional Blindness. This is the tendency to frame problems from your own area of work or study. Dearborn and Simon called this Selective Perception, ref [3], where they noted that managers often focus their attention on areas that they are familiar with - sales executives focussing on sales as a top priority and production executives focussing on production.

In testing this may translate into:

• Testers with mainly performance test experience focussing on those areas
• Recent customer support experience leading to a preference to operational and configuration aspects
• A generalist spreading effort evenly in many areas

Sunk-Cost Fallacy

This is the tendency to factor in previous investments to the framing of the problem, link. A good example is James Bach's Golden Elephant Syndrome, ref [4].

In testing this may translate into:

• The latest favourite tool or framework of the execs must be used as there has been so much investment in it.

Over-Confidence

As we've seen above there can be many different ways of framing the problem. It's important to be aware of this. There is a trap that testers can think they've done everything they need - their model/s was the most adequate in this situation. 

Here the warning is against complacency - re-evaluate periodically and tell the story against that assessment. It may be that an issue you find during testing affects some of your initial assumptions - the approach might be good, but maybe it could be better. 

(It might be that you can't change course/approach even if you wanted to, but that's good information for reporting to the stakeholder - areas for further investigation.)

Whatever your model, it's one model. Is it good enough for now? What does it lack - what product blind spots does it have?

Measurements and Numbers

Decision frames and framing sometimes uses a way of measuring whether the frame is good or useful - or whether alternatives are equal. There is a danger here when numbers and measurements get involved.

In business and everyday life there can be occasions when figures and measurements are presented as absolutes  and other times when they're presented are relative figures. They can be misleading in both cases, especially when not used consistently. 

Project stakeholders are probably very familiar with looking at project cost and overrun in absolute and relative terms - depending on how they want the information to shine.

So it's very easy for testers to be drawn into the numbers game - and even play it in terms of absolute or relative figures.

• "This week we have covered 50% of the product"
• "Our bug reports have increased 400% compared to the previous project"
• "The number of tests to run is about 60% of the last project"
• "5 bug reports have been implemented in this drop"
• "All pre-defined tests have been run"

As you can (hopefully) see this is just data - not necessarily information that can be interpreted. So, beware of number usage traps in the problem analysis and formulation - both in those given to you and in those you send out,

Another aspect of problems with numbers and decision framing can be thought of as the certainty effect, ref [6]. This can affect how we frame problems - and even how we should communicate.

Frames and Framing Need Maintenance

Analyze and periodically check that your assumptions are correct. Sometimes the emphasis of the project changes - the problem to solve changes. Is the frame still right or correct? Are the parameters of the problem still the same, are the reference points and ways in which to measure or judge the frame - are they the same - if not, time to re-evaluate.

Working with Frames

• What frames do you and your project / organization start with? (Subconcious default)
• Are there alternative frames to consider? How many were investigated?
• Look at what each frame includes and excludes
• What is the best frame fit for the situation / project? (Do all involved agree on the 'good enough' frame?)

References

[1] The Framing of Decisions and the Psychology of Choice (Tversky & Kahneman, Science Vol 211, No. 4481)

[2] Decision Traps: The Ten Barriers to Brilliant Decision-Making and How to Overcome Them (Russo, Schoemaker, Fireside,1990)

[3] Selective Perception: A Note on the Departmental Identifications of Executives (Dearborn, Simon, Sociometry Vol 21, No 2, June 1958)

[4] James Bach "Golden Elephant" Syndrome (Weinberg, Perfect Software: And Other Illusion about Testing, Dorset House, 2008, p. 101)

[5] Calculated Risks: How to Know When Numbers Deceive You (Gigerenzer, Simon and Schuster, 1986)

[6] The Certainty Effect and Insurance

Fault Localisation and Search Satisfaction

" #softwaretesting #testing "

A recent meet-up with Christin Wiedemann, Oscar Cosmo and Daniel Berggren to talk testing was a very enjoyable and thought-provoking session. It reminded me of a problem that I'd recently being involved with, where I'd observed the problem of Search Satisfaction.

Background
A tricky performance problem existed and it was getting more and more "stakeholder" attention. The problem description (bug report) was quite detailed around the network for the part of the system under test (SUT) in the picture. The bug report described a fairly major problem in the SUT and the investigation had been ongoing for some time when I eventually came into the loop.

The ongoing investigation and fault localisation was made more difficult by the fact that several different problems were being observed in the whole network simultaneously (multiple problems in each network element) and so rooting out the real problem was quite tricky.

Several rounds of extracting different log and trace information to localise the fault and create patch builds had been made. Then there was some form of inertia - little or no progress.

For some reason I was involved in the loop.

Search Satisfaction
When reading "How Doctors Think", ref [1], I encountered the notion of search satisfaction. This is where a diagnosis is made based on the observed data (or really a subset of the complete symptoms), no further testing or searching is done. The search is satisfied and the diagnosis made.

When I read the bug report for the problem above, and all the associated notes, I found lots of evidence pointing to the SUT. There was a lot of information about faults in the SUT and actions that had been taken to correct them and re-test. However, there was something missing - for me. The main problem/issue didn't seem to be related to the faults that had been corrected.

The description I read in the bug report would justify some of the associated faults being corrected (in an effort to localise the main problem) but not really the most serious fault. For that there needed to be more information on the surrounding network elements - I suspected an interaction problem and to understand that I needed information on the behaviour in the rest of the network.

I raised these issues with the people directly involved. They took another look, now with a different perspective and after some further rounds of localisation (now involving a wider search) the issue was found. A fault in another network element which was masked by the interaction between several elements.

So, in this example the faults being worked on were driven partly by the data available and partly by what could be fixed. The actions on those faults were correct. The search for fixes was satisfied by some of the symptoms that could be seen. But did those symptoms explain the severity of the main problem? Sometimes that's an important question - I can observe symptom X, Y & Z, but do they explain the main problem?

Search Satisfaction - How?
Search satisfaction is not just about fulfilling a search and not wanting to continue search. We learn to see patterns, get comfortable with patterns and recognize patterns. This has been demonstrated in different studies.

In one case, ref [2], it was found that Americans and Chinese subjects focus on different aspects of a picture. These habits of perception are thought to be influenced by the environment and society that they live in - people learn what they're used to and that is the default habit.

Another study, ref [3], showed an experiment where kittens had learned to perceive their environments in different ways (by being deprived of horizontal or vertical perception) and then had problems when they needed to use the perception they'd been deprived of.

Another favourite perspective of mine - which I use when thinking about open-ended tests - is the example in this picture from hubble, here. Read the comment. Sometimes if you think you won't see anything, you might not see anything - so sometimes you need to keep looking!

So, whether seeing a pattern in what one observes when testing, those patterns can be something that has been learnt. It fits the observation. But is it the only explanation, or is there a different observation (perspective) that might contradict the hypothesis?

Observation
In the example that I observed there were two factors that kept the search "satisfied" and constrained. These were a fixed focus and constant stakeholder attention.

Too focussed:
De-Focus and Re-Focus techniques can help avoid this problem. Sometimes you have to step back and observe what's going on around the periphery to then make some different sense of the detail. This is sometimes easier said than done. Sometimes it needs a fresh pair of eyes to trigger this.

Stakeholder "attention":
This can actually induce search satisfaction. Someone wants an answer (or diagnosis) and you have an answer staring right at you. This is more difficult to avoid, but a first step to avoiding this is realizing that you might be operating with limited data.

Lessons from Science
Don't stick to confirmatory approaches. Think about disconfirmatory evidence - how could my theory be proved wrong or incomplete? Do I have enough information? Can I think of a way in which it might not be a sufficient (or good enough) theory?

References

1. How Doctors Think (Groopman, Houghton Mifflin 2007)
2. Cultural variation in eye movements during scene perception (Chua, Boland and Nisbett, PNAS 2005)
3. Development of the brain depends on the visual environment (Blakemore and Cooper, Nature 1970)

Problem Analysis - Mind Maps & Thinking

Credit to Peter Haworth-Langford, Christin Wiedemann and Oscar Cosmo, with whom I was chatting back in December...

The topic of mind maps was discussed (in various usages) and I realised that maybe I had a usage that I need to write about and digest...

I frequently get asked to do a root cause analysis type of activity. Usually it's connected with faults reported against a product, but other times it might be to analyse a situation or process. In all cases I usually have a main question in my head, "what is the real problem?"

Below is an example of using these approaches for a root cause analysis of a fault. The analysis is used as a learning opportunity - to see if there is anything that needs to be changed or improved, including project and team structures and aspects of how they work.


Example
A typical root cause analysis (RCA) in my shop might look like the following. This is one example and there are many other variants.

The example "presumes" there was some missing testing (which might reasonably have been expected to happen), but it could just as easily be missing design analysis, design coding error or any of a range of other issues.

I have two main techniques that I use to help reveal information - and so help me on my way to "the real problem". Note, I say this as though there is only one, but I'm well aware there are usually many different competing aspects.

The first is the 5-whys and the second is what I think of as the "are your lights on?" approach and I use them in distinct ways:

• 5-whys - extracting raw data
• Lights-on - attaching meaning to the analysis

The 5-Whys
This is the forensic search for raw data - collecting the different aspects of the problem - what the problem was, its solution, aspects related to team analysis, system design, aspects related to test design and execution, aspects of team dynamics, changes in timing and priorities that might any of these parts.

Actually, I don't restrict the analysis to "5" - it can be 2, 3, or 8 - as many as are needed to get a "good enough" picture or level of granularity (part of this is an understanding of how deep we need to dig).  Extending the above example, it might become:

Lights-On
This approach is heavily based on the Gause & Weinberg book, "Are your lights on?", which I recommended highly. This is an approach that helps answer:

• What is the problem?
• To whom is it a problem?
• Whose problem is it?
• Does the problem need solving?

Usually, by working through these aspects of the problem it is possible to determine the extent of the issue, if it is an issue that should be addressed, if this is something that can be done now and by who.

Sometimes the problem appears due to a change in project priority or timing, something doesn't get completed and, crucially, the customer is not aware that it shouldn't use feature X in this first drop, or that it has certain limitations.

Communication (or the absence of it) is, occasionally, a root cause in itself!


A Third Way
I said I had two main ways, well recently I've realised that I'm using a third way - I say recently, as I've really only attached (or discovered) the terminology for it: Framing.


Framing
Framing is heavily based around the work of Tversky & Kahneman, with some application by Russo.

This is the filter that is used to look at the problem. In the above example, I might look at the issue from the project perspective: what priorities existed on the project (stakeholder or customer), were there aspects outside the control or remit of the project, did any of the project aspects change and with what timing?

There might be team frames: how did the internal and external communication to and from the team work; was there a project change that didn't filter into the team; information availability, assumptions and other limitations.

Finally, there might be aspects that work on project, team or individual level: risk averse or risk taking attitudes; attitudes of uncertainty, lack of support or overconfidence; or a combination of many factors.

So, applying some framing aspects to this example might give:

The framing of the problem - and using multiple frames - helps to put the issues into perspective, the situational context - what did it mean at this particular point in time, in this particular project, in this particular team set-up, for this particular customer, etc, etc.


Full Circle
The framing aspect now fits quite well into my RCA approach, or any problem analysis approach:

• Raw data is gathered: 5-whys is one approach
• Situational context: Framing, ideally using a number of frames.
• Meaning and decision: Lights-on

Conclusion
Mind maps help in a number of aspects here: visualisation and recording of thinking and exploration.

However, the big aspect is the thinking - using a range of tools to explore the problem, not limiting analysis to the problem artifact (a team didn't do xyz...) but adding in the situational aspects and weighing up the problem with the surrounding information to get to a better understanding of it (framing). Then the information can be used to help understand if it really is a problem that needs solving and by who (Lights-On).


References

1. Gause & Weinberg, "Are Your Lights On?: How to Figure Out What the Problem Really Is" (Dorset House, 1990)
2. Kahneman & Tversky, "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981)
3. Russo, "Decision Traps: The Ten Barriers to Decision-Making and How to Overcome Them" (Fireside, 1990)
4. 5 Whys, (wikipedia)

The Certainty Effect and Insurance

" #softwaretesting #testing "

At #swet2 I gave a lightning talk on an aspect of Framing (more on this in future posts) and thought I'd jot down some of the details.

On the way to the peer conference I'd read Tversky and Kahneman's "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981), and was struck by the description of what they described as the certainty effect. Or really the potential application in testing.

The Certainty Effect
This was labelled by Tversky and Kahneman (in a 1979 article in Ecometrica on Prospect Theory) on a paradox observed by French economist Maurice Allais in 1953. Expected utility theory predicts that a choice is made dependent on it's expected probability. The example from the 1979 article is:

Problem 1:

Choose between

Choice A 

A return of 2500 with a probability of .33

A return of 2400 with a probability of .66

A return of 0 with a probability of .01

Choice B

A certain (guaranteed) return of 2400

In a sample of 72 respondents 18% chose A and 82% chose B.

This is in line with expected utility theory. It's also known as a risk averse approach. Now in the second problem the certainty was removed:

Problem 2:

Choose between

Choice C 

A return of 2500 with a probability of .33

A return of 0 with a probability of .67

Choice D 

A return of 2400 with a probability of .34

A return of 0 with a probability of .66

Now in this case, of the 72 respondents 83% chose C and 17% chose D. This is not in line with expected utility theory and is what is labelled as the certainty effect. That is, there was a disproportionate weighting on an outcome when it was certain - as this weighting wasn't reflected when the outcome was not certain.

This is related to how the question is asked (framed).

Insurance
A further example of this type of framing affecting choice can be seen in attitudes to risk - typically when looking at insurance there is a difference in attitudes that is displayed between risk reduction and risk elimination, for example consider:

A vaccine that is effective in half of cases.

vs

A vaccine that is fully effective in 1 of 2 cases.

Then there is a disproportionate preference for the second formulation.

Applied to Testing?
By default, stakeholders want certainty. They tend to want risk elimination rather than risk reduction. They link this to cost (just like in different levels of insurance) and think that by working with cost they will get closer to certainty.

This is a problem for testers - or really how they are communicating with stakeholders.

Testers can't work in terms of certainty (without very tight restrictions and lots and lots of explanations and assumptions). Therefore, given the two possibilities of talking about risk elimination and risk reduction testers should talk in terms of risk reduction.

Additionally, the certainty effect tells us that typical decisions and choices can be skewed (disproportionately) when the risk or probability moves away from certainty (guarantee).

When handling the message and understanding expectations towards stakeholders consider:

• Be consistent - never talk in terms of (or give the impression that you can deliver) certainty.
• Be aware that when something is not certain then attitudes to risk and decision choices don't always follow expected weighting of probabilities.

Certainty, insurance and talking to stakeholders - it's not always logical.

Did you understand the question?

" #softwaretesting #testing #cognition "

I just took a short quiz on "Science fiction vs science fact" (here it is, (link), go and take it - it'll take 5 mins).

Well, I stank!

But long before I got to the end of the quiz I realised that I wasn't sure about the intentions of the quizmaster - were they phrasing the questions ambiguously (maybe to trap or fool people), or was the subject matter naturally close to the edge of plausibility?

Anyway, it seemed like I was both unsure of how to evaluate the questions but also how to evaluate the questioner - so really I didn't understand the question (it's context you could say) and whether it was important to deliberate long over the questions. Of course, it was just a bit of fun (trivia) so I plowed on, but I was aware of all these questions and potential thinking traps I was falling into...

Availability and anchoring biases - Ah, I heard of this in the news recently - or did I? Then connecting that with another question... Thinking something sounds plausible and then not wanting to move too far away from that opinion.

Where there were a range of topics being discussed then it's easy to fall for the recency effect so you don't dwell on the question and realize you were tricked.

There is, of course, a whole topic on taking exams - including not dwelling too long to save time to revisit the question - but that's another story... If you follow the wikipedia link then you'll probably be able to find examples of lots of different biases in the way you take the test (or answer the question) - one of them being "reading too much into the data bias..." (almost the self-serving bias.)


Testing?

Yes, I immediately started thinking along a couple of testing-related lines.

• Did I understand the question/requirement?
• Did I understand the context behind the question/requirement?

Having the stakeholder on-hand is always very useful to clarify and clear out any misunderstandings (by either stakeholder or yourself). Sometimes that's not possible - as in the case of the above multiple-choice test - but usually the things that matter in testing will have a stakeholder will be available at some point in time.

If you have a stakeholder (or proxy) available then you can follow-up with a whole range of questions to get to the bottom of the problem.

A very important aspect is your own frame - what's your attitude to the problem, but also to understand that how the information is presented (or by whom) can affect your response.

For example, if you're not on amicable terms with the stakeholder you might adopt an aggressive questioning attitude and not be receptive to the information to be able to react/respond with useful  follow-up questions. 

Or: You're feeling very tired (or not as alert as usual) and so you miss some implication in the question (requirement) - and act on the first layer of information: Yes, we can send a man to Mars because we can build a rocket and life-support system (but how much has his/her muscles deteriorated by the time they return to Earth - and so how much recovery time is needed, permanent damage(?) etc, etc..)

Can you guarantee that this correction package will work?

Yes, I've heard that question in the past. Working out where to start tackling that question is a whole different post - but really there is a whole different bunch of questions that the questioner/stakeholder has and he/she expresses the "simple" (compressed) question to me - but really I need to get behind the question and understand what their "real" problem is - one way is to use Gause & Weinberg's "context-free questions" from "Exploring Requirements" (here's a transcription from Michael Bolton). Another way is to use some of the techniques from "Are your lights on?" (Gause & Weinberg again.)

Framing

Most of this (for me) boils down to framing and how that influences both our problem analysis, information intake, problem exploration and ultimately decision making. We all have it - mostly without realising the affect it plays. But the important aspect is to (try to) be aware of it and some of the problems that it can cause - then we have a better chance of answering the question (requirement) in the real spirit that it was asked!

By the way, it did occur to me that this could be misconstrued as another "exam-bashing" link - but that's not the intention :) If that thought occurred to you, then that maybe says something about your frame.


Are you aware of your own frames?

Oh, this was my first transcription from 750words - thanks to Alan Page for tweeting about his use - I'm brain-dumping regularly now!

Links:

Quiz: 

• http://www.bbc.co.uk/news/science-environment-12758575

Context-free questions: 

• http://www.developsense.com/blog/2010/11/context-free-questions-for-testing/

Framing:

• http://en.wikipedia.org/wiki/Framing_(social_sciences)
• http://en.wikipedia.org/wiki/Framing_effect_(psychology)