Our survey said...

" #interpretation #fun #context #cognition "

Whilst compiling material for some other work I stumbled across some old Family Fortunes and Family Feud 'funny/strange' answers on YouTube*.

I've recently being doing a lot of thinking around framing, ref [5], and the problems it can cause and solve and I started thinking about different causes for the unexpected answers.

For communication analysis I use two types of exercise, (1) frame analysis and (2) word and meaning substitution.

Frame analysis

• What are the aspects that might be important to each person involved in the communication? This usually revolves around situational context of either the one asking the question (presenting the problem) or the one answering the question (presenting a solution). Here there is scope for a range of cognitive and interpretation mistakes.

Word and meaning substitution

• A well-known example of this is the "Mary had a little lamb" exercise, described in "Are you lights on", ref [1], and is a demonstration of how changing the emphasis of a word in a sentence, or replacing a word with a similar meaning (from a dictionary or thesaurus), can change the meaning of the sentence. Therefore, if both parties in the communication intend different word emphasis (in word placement or interpretation) then there is a possibility of confusion.

So, what can appear as confusing or even amusing answers can, with the right perspective, have a certain logic. In the list** below I've made an attempt at finding the perspective behind the answer, in red.

The questions are typically prefixed with "We asked 100 people to name..."

Q: Something a husband and wife should have separate of

A: Parents

Logical answer(?) but maybe not along the intended lines of the questioner.

Q: A planet you recognize just by looking at a picture of it

A: The Moon

Confusion of definition of planet with 'celestial body' (something in space with an orbit)

Q: A month of spring

A: Summer

Slip of the ear, of->after(?), ref [4]

Q: A word that starts with the letter Q

A: Cute

Q. Name a part of the body beginning with 'N'

A. Knee

Phonetic interpretation

Q: The movie where John Travolta gave his most memorable performance

A: The John Travolta Biography

'Most memorable performance' to the questioner had a potentially different meaning. To the answerer either it was interpretted as a film where he featured the most, or he wanted to give an amusing answer.

Q: Something you wouldn't use if it was dirty

A: Toilet paper

Amusing and logical answer(?)

Q: A signer of the Declaration of Independence

A: Thomas Edison

Slip of the tongue, specifically noun substitution, ref [4]

Q: Something that comes in twelves

A: Dozens 

Could be logical interpretation but not something the questioner was intending(?)

Q: A sophisticated city.

A: Japan

Misinterpretation (or even slip of the ear) of city for destination.

Q: A kind of bear

A: Papa Bear

Recency effect(?) - had recently been reading or exposed to children's stories (?)

Q. Name a number you have to memorise

A. 7

Misinterpretation of 'memorise' as 'favourite' or 'memorable'(?)

Q. Name something in the garden that's green

A. Shed

Context-specific to the answerer(?)

Q. Name something that flies that doesn't have an engine

A. A bicycle with wings

'Logical' and specific answer - but the questioner could have maybe clarified the question with a 'commonly known item'.

Or, recency effect - flugtag, ref [6].

Q. Name something you might be allergic to

A. Skiing

'Alergic' -> 'don't like'(?)

Q. Name a famous bridge

A. The bridge over troubled waters

Interpreted as 'something well-known with bridge in it'(?)

Q. Name something you do in the bathroom

A. Decorate

Specific to the answerer's context.

Q. Name an animal you might see at the zoo

A. A dog

Generics. Potential that the answerer has not interpreted the the question as 'generally seen and residing in the zoo'.

Q. Name a kind of ache

A. Fillet 'O' Fish (?)

Brain-freeze or 'slip of the ear'(?)

Q. Name a food that can be brown or white

A. Potato

Answerer framed the question as a food which could be presented as brown or white(?)

Q. Name a famous Scotsman

A. Jock

'Slip of the ear' -> 'a common nickname'(?)

Q. Name a non-living object with legs

A. Plant

Maybe thinking of a plant on a plant stand(?)

Q. Name a domestic animal

A. Leopard

Misinterpretation of 'domestic'(?)

Q. Name a way of cooking fish

A. Cod

'way' misinterpreted as 'type'(?)

Analysis Notes

• Context - some answers are specific to the answerer and not the questioner. Example traps might be (1) Understanding and interpretation, (2) Word association problems or (3) Relating everything to ones own experience or circumstances.

• Recency effects, ref [3] - the interpretation associated with a word was used in a different context, giving a skewed answer. In testing this occasionally results in skewed emphasis of the risk determination - see tester framing problems in ref [5].

• Skipping and changing words in sentences - to actually hear a different question - sometimes grouped under 'slips of the ear'. In testing this might result in an incorrect solution application, similar to framing problems but can also be 'straightforward' slips that result in some faulty analysis - missing some key input parameter for example.

• Other framing effects can be caused by the previous question, previous answer or even some realisation that a previous answer was wrong/silly and so inducing more stress in the answerer.

• Stress can mean that sometimes when you're trying to react you don't actually listen to the whole message or question. This can be time pressure or other stresses. Be aware of this potential problem.

• Anchoring effects, ref [2] - focusing on a word and giving an association with that word (rather than focusing on the whole question). In testing this typically results in confirmatory testing.

• Generic statements can create confusion. These are generic statements as part of the answers - this is where the question can be confused between giving an example of a specific kind and categorizing the answer into a grouping. Opposite of the answerer-specific problem. More on this in another post...

• Don't rule out brain-freezes either - these can be multi-word substitution or paragrammatism, ref [4], which result in nonsense responses.

And finally...

This is a good exercise and quite instructive for those working in software testing - it's a good illustration of how what might be seen as an obvious or simple answer can actually diverge from the expectations of the stakeholder or even customer.

Be alert for not just for confusing messages but also the potential for confusing answers. In this way you might know when to re-affirm your interpretation back to the stakeholder or customer.


References

* If you want to see the clips you can search youtube for "family fortunes answers" or "family feud answers" or "game show stupid answers".
** Lists compiled from

http://www.funny-haha.co.uk/Joke.asp?J=283

http://www.businessballs.com/familyfortunesanswers.htm

http://www.stupidgsa.com/american/family-feud/

[1] Are Your Lights On?: How to Figure Out What the Problem Really Is (Gause and Weinberg, Dorset House, 1990)
[2] Anchoring effects: http://en.wikipedia.org/wiki/Anchoring
[3] Recency effects: http://en.wikipedia.org/wiki/Serial_position_effect

[4] Um...: Slips, Stumbles, and Verbal Blunders, and What They Mean (Erard, Panteon, 2007)

[5] http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html

[6] http://en.wikipedia.org/wiki/Red_Bull_Flugtag

Framing: Some Decision and Analysis Frames in Testing

" #softwaretesting #testing "

What is a Frame?

The following is from Tversky and Kahneman's description of a decision frame, ref [1],:

We use the term "decision frame" to refer to the decision-maker's conception of the acts, outcomes, and contingencies associated with a particular choice. 

The frame that a decision-maker adopts is controlled partly by the formulation of the problem and partly by the norms, habits, and personal characteristics o f the decision-maker

When using a decision frame to analyze a problem and come to a decision they call this framing. So, I'll refer to a frame as relating to a decision (or analysis) frame.

Factors at play

As mentioned, many different factors affect how we analyze problems, including:

Temperament/Emotions

• Anger
• Happy
• Optimistic
• Pessimistic
• Tiredness
• Fear (e.g. of failure)

Experience

• Lessons from past situations - own experience and feedback
• What has been learnt recently
• Complacency due familiarity

Strategy

• Your own vs someone else's
• Aggressive
• Military campaign - lots of detailed planning
• Reactive

The factors and the weight given to them might be different for:

• Stakeholder view ("Upgrade needs to be reliable", "Of the new feature set only x is live in the quarter")
• Tester view ("Which risks are most important to look at now?")
• Developer view ("I did a fix, can you test that first?")

The stakeholder, developer, tester and any other role in the project has a set view on priorities and aims with the project - agendas maybe - and one challenge is in trying to tie these together, or at least understand the differences and how they impact our communication. They all may have similar product goals but their interpretations to their work may be different - their influences and micro-decisions will be different meaning that transparency in communication is important. 

But, there's a catch - the way we present information can affect its interpretation - depending upon the frame that a stakeholder is adopting.

Think of a frame as a filter through which someone looks at a problem - they're taking in lots of data but only the data that gets through the filter gets attention (the rest may end up in the subconscious for later or isn't absorbed), "I have my show-stopper filter on today so I don't notice the progress the team has made…" 

So, being aware of the potential different types of frames that each project member might have as well as some traps associated with frame formulation is important.

Stakeholder Frames

Might include:

• Emphasizing minimum time for product delivery
• Emphasizing short iteration times and delivering quickly
• Trying to minimize cost of product development (cost of testing?)
• Emphasizing future re-use of the development environment (tendency to worship automation?)
• Aiming for a reduced future maintenance cost

Tester Frames

Might include:

• Emphasizing the favourite test approach
• Emphasizing areas of greatest risk (to?)
• Emphasizing the last successful heuristic that found a show-stopper bug
• Emphasizing focus on the data configuration that found the most bugs the last time
• Emphasizing conformance to a standard over a different aspect of the product
• Emphasizing the backlog item that seems the most interesting
• Emphasizing widespread regression as "fear of failure / breaking legacy" affects analysis
• Emphasizing feature richness over stability

Note, in isolation, some of these frames may be good, but they might not necessarily be good enough.

Framing Problems in Testing

Functional Blindness or Selective Perception

Russo and Schoemaker called it Functional Blindness. This is the tendency to frame problems from your own area of work or study. Dearborn and Simon called this Selective Perception, ref [3], where they noted that managers often focus their attention on areas that they are familiar with - sales executives focussing on sales as a top priority and production executives focussing on production.

In testing this may translate into:

• Testers with mainly performance test experience focussing on those areas
• Recent customer support experience leading to a preference to operational and configuration aspects
• A generalist spreading effort evenly in many areas

Sunk-Cost Fallacy

This is the tendency to factor in previous investments to the framing of the problem, link. A good example is James Bach's Golden Elephant Syndrome, ref [4].

In testing this may translate into:

• The latest favourite tool or framework of the execs must be used as there has been so much investment in it.

Over-Confidence

As we've seen above there can be many different ways of framing the problem. It's important to be aware of this. There is a trap that testers can think they've done everything they need - their model/s was the most adequate in this situation. 

Here the warning is against complacency - re-evaluate periodically and tell the story against that assessment. It may be that an issue you find during testing affects some of your initial assumptions - the approach might be good, but maybe it could be better. 

(It might be that you can't change course/approach even if you wanted to, but that's good information for reporting to the stakeholder - areas for further investigation.)

Whatever your model, it's one model. Is it good enough for now? What does it lack - what product blind spots does it have?

Measurements and Numbers

Decision frames and framing sometimes uses a way of measuring whether the frame is good or useful - or whether alternatives are equal. There is a danger here when numbers and measurements get involved.

In business and everyday life there can be occasions when figures and measurements are presented as absolutes  and other times when they're presented are relative figures. They can be misleading in both cases, especially when not used consistently. 

Project stakeholders are probably very familiar with looking at project cost and overrun in absolute and relative terms - depending on how they want the information to shine.

So it's very easy for testers to be drawn into the numbers game - and even play it in terms of absolute or relative figures.

• "This week we have covered 50% of the product"
• "Our bug reports have increased 400% compared to the previous project"
• "The number of tests to run is about 60% of the last project"
• "5 bug reports have been implemented in this drop"
• "All pre-defined tests have been run"

As you can (hopefully) see this is just data - not necessarily information that can be interpreted. So, beware of number usage traps in the problem analysis and formulation - both in those given to you and in those you send out,

Another aspect of problems with numbers and decision framing can be thought of as the certainty effect, ref [6]. This can affect how we frame problems - and even how we should communicate.

Frames and Framing Need Maintenance

Analyze and periodically check that your assumptions are correct. Sometimes the emphasis of the project changes - the problem to solve changes. Is the frame still right or correct? Are the parameters of the problem still the same, are the reference points and ways in which to measure or judge the frame - are they the same - if not, time to re-evaluate.

Working with Frames

• What frames do you and your project / organization start with? (Subconcious default)
• Are there alternative frames to consider? How many were investigated?
• Look at what each frame includes and excludes
• What is the best frame fit for the situation / project? (Do all involved agree on the 'good enough' frame?)

References

[1] The Framing of Decisions and the Psychology of Choice (Tversky & Kahneman, Science Vol 211, No. 4481)

[2] Decision Traps: The Ten Barriers to Brilliant Decision-Making and How to Overcome Them (Russo, Schoemaker, Fireside,1990)

[3] Selective Perception: A Note on the Departmental Identifications of Executives (Dearborn, Simon, Sociometry Vol 21, No 2, June 1958)

[4] James Bach "Golden Elephant" Syndrome (Weinberg, Perfect Software: And Other Illusion about Testing, Dorset House, 2008, p. 101)

[5] Calculated Risks: How to Know When Numbers Deceive You (Gigerenzer, Simon and Schuster, 1986)

[6] The Certainty Effect and Insurance

Problem Analysis - Mind Maps & Thinking

Credit to Peter Haworth-Langford, Christin Wiedemann and Oscar Cosmo, with whom I was chatting back in December...

The topic of mind maps was discussed (in various usages) and I realised that maybe I had a usage that I need to write about and digest...

I frequently get asked to do a root cause analysis type of activity. Usually it's connected with faults reported against a product, but other times it might be to analyse a situation or process. In all cases I usually have a main question in my head, "what is the real problem?"

Below is an example of using these approaches for a root cause analysis of a fault. The analysis is used as a learning opportunity - to see if there is anything that needs to be changed or improved, including project and team structures and aspects of how they work.


Example
A typical root cause analysis (RCA) in my shop might look like the following. This is one example and there are many other variants.

The example "presumes" there was some missing testing (which might reasonably have been expected to happen), but it could just as easily be missing design analysis, design coding error or any of a range of other issues.

I have two main techniques that I use to help reveal information - and so help me on my way to "the real problem". Note, I say this as though there is only one, but I'm well aware there are usually many different competing aspects.

The first is the 5-whys and the second is what I think of as the "are your lights on?" approach and I use them in distinct ways:

• 5-whys - extracting raw data
• Lights-on - attaching meaning to the analysis

The 5-Whys
This is the forensic search for raw data - collecting the different aspects of the problem - what the problem was, its solution, aspects related to team analysis, system design, aspects related to test design and execution, aspects of team dynamics, changes in timing and priorities that might any of these parts.

Actually, I don't restrict the analysis to "5" - it can be 2, 3, or 8 - as many as are needed to get a "good enough" picture or level of granularity (part of this is an understanding of how deep we need to dig).  Extending the above example, it might become:

Lights-On
This approach is heavily based on the Gause & Weinberg book, "Are your lights on?", which I recommended highly. This is an approach that helps answer:

• What is the problem?
• To whom is it a problem?
• Whose problem is it?
• Does the problem need solving?

Usually, by working through these aspects of the problem it is possible to determine the extent of the issue, if it is an issue that should be addressed, if this is something that can be done now and by who.

Sometimes the problem appears due to a change in project priority or timing, something doesn't get completed and, crucially, the customer is not aware that it shouldn't use feature X in this first drop, or that it has certain limitations.

Communication (or the absence of it) is, occasionally, a root cause in itself!


A Third Way
I said I had two main ways, well recently I've realised that I'm using a third way - I say recently, as I've really only attached (or discovered) the terminology for it: Framing.


Framing
Framing is heavily based around the work of Tversky & Kahneman, with some application by Russo.

This is the filter that is used to look at the problem. In the above example, I might look at the issue from the project perspective: what priorities existed on the project (stakeholder or customer), were there aspects outside the control or remit of the project, did any of the project aspects change and with what timing?

There might be team frames: how did the internal and external communication to and from the team work; was there a project change that didn't filter into the team; information availability, assumptions and other limitations.

Finally, there might be aspects that work on project, team or individual level: risk averse or risk taking attitudes; attitudes of uncertainty, lack of support or overconfidence; or a combination of many factors.

So, applying some framing aspects to this example might give:

The framing of the problem - and using multiple frames - helps to put the issues into perspective, the situational context - what did it mean at this particular point in time, in this particular project, in this particular team set-up, for this particular customer, etc, etc.


Full Circle
The framing aspect now fits quite well into my RCA approach, or any problem analysis approach:

• Raw data is gathered: 5-whys is one approach
• Situational context: Framing, ideally using a number of frames.
• Meaning and decision: Lights-on

Conclusion
Mind maps help in a number of aspects here: visualisation and recording of thinking and exploration.

However, the big aspect is the thinking - using a range of tools to explore the problem, not limiting analysis to the problem artifact (a team didn't do xyz...) but adding in the situational aspects and weighing up the problem with the surrounding information to get to a better understanding of it (framing). Then the information can be used to help understand if it really is a problem that needs solving and by who (Lights-On).


References

1. Gause & Weinberg, "Are Your Lights On?: How to Figure Out What the Problem Really Is" (Dorset House, 1990)
2. Kahneman & Tversky, "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981)
3. Russo, "Decision Traps: The Ten Barriers to Decision-Making and How to Overcome Them" (Fireside, 1990)
4. 5 Whys, (wikipedia)

The Certainty Effect and Insurance

" #softwaretesting #testing "

At #swet2 I gave a lightning talk on an aspect of Framing (more on this in future posts) and thought I'd jot down some of the details.

On the way to the peer conference I'd read Tversky and Kahneman's "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981), and was struck by the description of what they described as the certainty effect. Or really the potential application in testing.

The Certainty Effect
This was labelled by Tversky and Kahneman (in a 1979 article in Ecometrica on Prospect Theory) on a paradox observed by French economist Maurice Allais in 1953. Expected utility theory predicts that a choice is made dependent on it's expected probability. The example from the 1979 article is:

Problem 1:

Choose between

Choice A 

A return of 2500 with a probability of .33

A return of 2400 with a probability of .66

A return of 0 with a probability of .01

Choice B

A certain (guaranteed) return of 2400

In a sample of 72 respondents 18% chose A and 82% chose B.

This is in line with expected utility theory. It's also known as a risk averse approach. Now in the second problem the certainty was removed:

Problem 2:

Choose between

Choice C 

A return of 2500 with a probability of .33

A return of 0 with a probability of .67

Choice D 

A return of 2400 with a probability of .34

A return of 0 with a probability of .66

Now in this case, of the 72 respondents 83% chose C and 17% chose D. This is not in line with expected utility theory and is what is labelled as the certainty effect. That is, there was a disproportionate weighting on an outcome when it was certain - as this weighting wasn't reflected when the outcome was not certain.

This is related to how the question is asked (framed).

Insurance
A further example of this type of framing affecting choice can be seen in attitudes to risk - typically when looking at insurance there is a difference in attitudes that is displayed between risk reduction and risk elimination, for example consider:

A vaccine that is effective in half of cases.

vs

A vaccine that is fully effective in 1 of 2 cases.

Then there is a disproportionate preference for the second formulation.

Applied to Testing?
By default, stakeholders want certainty. They tend to want risk elimination rather than risk reduction. They link this to cost (just like in different levels of insurance) and think that by working with cost they will get closer to certainty.

This is a problem for testers - or really how they are communicating with stakeholders.

Testers can't work in terms of certainty (without very tight restrictions and lots and lots of explanations and assumptions). Therefore, given the two possibilities of talking about risk elimination and risk reduction testers should talk in terms of risk reduction.

Additionally, the certainty effect tells us that typical decisions and choices can be skewed (disproportionately) when the risk or probability moves away from certainty (guarantee).

When handling the message and understanding expectations towards stakeholders consider:

• Be consistent - never talk in terms of (or give the impression that you can deliver) certainty.
• Be aware that when something is not certain then attitudes to risk and decision choices don't always follow expected weighting of probabilities.

Certainty, insurance and talking to stakeholders - it's not always logical.

Automation: Oh What A Lovely Burden!

" #softwaretesting #testing "

Do you have a suite of automated test cases? Have you looked at them lately? Do they seem to be growing out of control? Have they needed some update? Does it seem that you occasionally see a problem and think, 'why didn't the test suite catch that?'

If so, then maybe you have a 'lovely burden'.

One of the things with automated suites is that they are not guaranteed to maintain themselves.... Another is that they do not always tell the tester (or stakeholder) exactly what they're doing (and not doing). The information (results) that they give can be sometimes interpreted for something more than what they actually are.

Automated test suites can sometimes give very good information about changes you've made in the system, a lot of times they give very good feedback, sometimes they catch a catastrophic change.

However, they can sometimes lull 'people in software development' into a false sense of security. Wait! The test suites are not evil, as such, so how can that be?

Well, in addition to automated test suites not maintaining themselves and not guaranteeing a lot of  things - they are combined with people (whether testers, project, line or other stakeholders) into an idea of a 'holy suite'.

Why are they 'holy', untouchable and must be maintained (as though we form museums and living exhibits of test suites and frameworks)? Well, part of it is the "Golden elephant" problem (James Bach in Weinberg's "Perfect Software and other illusions about testing"). Another part of it is that people (testers, developers and stakeholders) can become detached from what the test suites are doing - something that has been around for a long time, might be 'left alone' until it breaks.

Oops!

Sometimes test suites are not maintained or refactored for several reasons. It may be a judgement call, sometimes it's not possible to easily see where the point of diminishing returns is reached, sometimes vanity (yes, we didn't see that they had an 'end-of-life' 5 years ago, but even so, I don't want to look like I couldn't plan 5 years ahead....) Projects usually have difficulty seeing clearly to the end of the project (at the beginning), so why should it be any different with any artifacts that are produced along the way (like test suites).

If I were not aware of a lot of the above problems I (mr stakeholder) might say that we need to plan better... But, as testers interested in contributing to working software products we should help contribute to the better understanding and use of automated test suites.

How?

Look at test suites regularly (or at least more than never) for:

• Relevance
• Signs for reaching the point of diminishing returns
• The Test Suite "Frame"

Relevance

• Is the test suite doing what it needs to do? Are there redundant test cases/scripts? Possibly - do you know where or which ones?

• Are there cases (scripts) that never (or hardly ever fail)? Are there scripts that fail when there are always others that fail? This might show a pattern in (1) the system architecture - weak links are highlighted - this is good, but how do you react to it? ; (2) the test suite and the way it is configured - different tests funnel through the same route (is this intentional?)

• The test suite is just one view of the software - maybe it's a real view or an artificial view (due to behaviour changes). Which is it?

• Is it static - same data and behaviour model - or is it dynamic? If it's not dynamic do you (someone) inject dynamism in some way (e.g. change cases in and out, rotate cases used, ordering, data fuzzing, etc..) Do you have any refresh or re-evaluation routines for the suite?

Point of diminishing return

Think about the current cost of maintaining the test suite.

How many backlog items are there? Do the backlog items that 'need' implementing grow at a greater rate than can be supported with the current budget (time or people), is the architecture reaching it's viable limit, do you know or have thought about what the viable limit is?

• Who's the test suite 'product owner', and how are the decisions about what goes in made?

It's important to understand what the automation suite is costing you now - this is an ongoing cost-benefit analysis - which is probably not done in a very transparent way. Not only should the current costs of maintenance be balanced against the benefits that the suite gives, but also more subtle items.

These more subtle items include the cost of the assumptions made about the suite - from the stakeholder perspective. How many decisions are based on inaccurate or incomplete information about what the test suite is giving? This is an area that is rarely appreciated, never mind understood or researched. Ooops!

The Test Suite Frame

Thinking about the test suite has several frames (models or filters in which people see the problem and interpret the information connected with it.) Some of these 'angles' might be:

1. What's the intention with the automated suite? Expectations?

• Is it the same intention that the stakeholder has? If not, do they realize this? Do the stakeholders think it is all-singing-and-dancing - and if so, how do you bridge that expectation?
• What about new stakeholders that 'inherit' a legacy suite? How do they get to know all the intricacies of the automated suites? They probably don't ask for them, so how does the tester communicate that?

2. Are there gaps to be filled? Planned or known about? (Maintenance plans)

3. The test suite will only give a limited view - do you actively counter this in some way?

4. Risks associated with the test suite - in terms of what it says and doesn't say? (How do you translate results into information further downstream?)

• What assumptions are built into the test suite?
• Happy path only?

These are just some of the most obvious frames.

And finally...

Ask yourself some basic questions (and just because they are basic doesn't mean they are easy to answer):

What assumptions are built into the test suite, what does it tell you, what doesn't it tell you, what expectations exist on it and how they are matched, or mitigated, how much reliance is placed on the suite, what risks exist with it and how they are monitored (evaluated)?

If you don't have the answer (or a view on these questions) then you have a potential burden.

You might think 'oh what a lovely burden' or you might think 'I'm a tester, get me out of here', or alternatively you might start thinking about which type of questions that need tackling now (soon) to ensure that the stakeholders are getting the information they need - and, importantly, understand the information that they are not getting. Then you/they can start wondering how much it will cost to get the extra (other) information and whether it's worth it.

But, ultimately you'll be working with automation in a responsible way.

Yes, sometimes it can be a 'lovely burden'...

Did you understand the question?

" #softwaretesting #testing #cognition "

I just took a short quiz on "Science fiction vs science fact" (here it is, (link), go and take it - it'll take 5 mins).

Well, I stank!

But long before I got to the end of the quiz I realised that I wasn't sure about the intentions of the quizmaster - were they phrasing the questions ambiguously (maybe to trap or fool people), or was the subject matter naturally close to the edge of plausibility?

Anyway, it seemed like I was both unsure of how to evaluate the questions but also how to evaluate the questioner - so really I didn't understand the question (it's context you could say) and whether it was important to deliberate long over the questions. Of course, it was just a bit of fun (trivia) so I plowed on, but I was aware of all these questions and potential thinking traps I was falling into...

Availability and anchoring biases - Ah, I heard of this in the news recently - or did I? Then connecting that with another question... Thinking something sounds plausible and then not wanting to move too far away from that opinion.

Where there were a range of topics being discussed then it's easy to fall for the recency effect so you don't dwell on the question and realize you were tricked.

There is, of course, a whole topic on taking exams - including not dwelling too long to save time to revisit the question - but that's another story... If you follow the wikipedia link then you'll probably be able to find examples of lots of different biases in the way you take the test (or answer the question) - one of them being "reading too much into the data bias..." (almost the self-serving bias.)


Testing?

Yes, I immediately started thinking along a couple of testing-related lines.

• Did I understand the question/requirement?
• Did I understand the context behind the question/requirement?

Having the stakeholder on-hand is always very useful to clarify and clear out any misunderstandings (by either stakeholder or yourself). Sometimes that's not possible - as in the case of the above multiple-choice test - but usually the things that matter in testing will have a stakeholder will be available at some point in time.

If you have a stakeholder (or proxy) available then you can follow-up with a whole range of questions to get to the bottom of the problem.

A very important aspect is your own frame - what's your attitude to the problem, but also to understand that how the information is presented (or by whom) can affect your response.

For example, if you're not on amicable terms with the stakeholder you might adopt an aggressive questioning attitude and not be receptive to the information to be able to react/respond with useful  follow-up questions. 

Or: You're feeling very tired (or not as alert as usual) and so you miss some implication in the question (requirement) - and act on the first layer of information: Yes, we can send a man to Mars because we can build a rocket and life-support system (but how much has his/her muscles deteriorated by the time they return to Earth - and so how much recovery time is needed, permanent damage(?) etc, etc..)

Can you guarantee that this correction package will work?

Yes, I've heard that question in the past. Working out where to start tackling that question is a whole different post - but really there is a whole different bunch of questions that the questioner/stakeholder has and he/she expresses the "simple" (compressed) question to me - but really I need to get behind the question and understand what their "real" problem is - one way is to use Gause & Weinberg's "context-free questions" from "Exploring Requirements" (here's a transcription from Michael Bolton). Another way is to use some of the techniques from "Are your lights on?" (Gause & Weinberg again.)

Framing

Most of this (for me) boils down to framing and how that influences both our problem analysis, information intake, problem exploration and ultimately decision making. We all have it - mostly without realising the affect it plays. But the important aspect is to (try to) be aware of it and some of the problems that it can cause - then we have a better chance of answering the question (requirement) in the real spirit that it was asked!

By the way, it did occur to me that this could be misconstrued as another "exam-bashing" link - but that's not the intention :) If that thought occurred to you, then that maybe says something about your frame.


Are you aware of your own frames?

Oh, this was my first transcription from 750words - thanks to Alan Page for tweeting about his use - I'm brain-dumping regularly now!

Links:

Quiz: 

• http://www.bbc.co.uk/news/science-environment-12758575

Context-free questions: 

• http://www.developsense.com/blog/2010/11/context-free-questions-for-testing/

Framing:

• http://en.wikipedia.org/wiki/Framing_(social_sciences)
• http://en.wikipedia.org/wiki/Framing_effect_(psychology)