Did you understand the question?

" #softwaretesting #testing #cognition "

I just took a short quiz on "Science fiction vs science fact" (here it is, (link), go and take it - it'll take 5 mins).

Well, I stank!

But long before I got to the end of the quiz I realised that I wasn't sure about the intentions of the quizmaster - were they phrasing the questions ambiguously (maybe to trap or fool people), or was the subject matter naturally close to the edge of plausibility?

Anyway, it seemed like I was both unsure of how to evaluate the questions but also how to evaluate the questioner - so really I didn't understand the question (it's context you could say) and whether it was important to deliberate long over the questions. Of course, it was just a bit of fun (trivia) so I plowed on, but I was aware of all these questions and potential thinking traps I was falling into...

Availability and anchoring biases - Ah, I heard of this in the news recently - or did I? Then connecting that with another question... Thinking something sounds plausible and then not wanting to move too far away from that opinion.

Where there were a range of topics being discussed then it's easy to fall for the recency effect so you don't dwell on the question and realize you were tricked.

There is, of course, a whole topic on taking exams - including not dwelling too long to save time to revisit the question - but that's another story... If you follow the wikipedia link then you'll probably be able to find examples of lots of different biases in the way you take the test (or answer the question) - one of them being "reading too much into the data bias..." (almost the self-serving bias.)


Testing?

Yes, I immediately started thinking along a couple of testing-related lines.

• Did I understand the question/requirement?
• Did I understand the context behind the question/requirement?

Having the stakeholder on-hand is always very useful to clarify and clear out any misunderstandings (by either stakeholder or yourself). Sometimes that's not possible - as in the case of the above multiple-choice test - but usually the things that matter in testing will have a stakeholder will be available at some point in time.

If you have a stakeholder (or proxy) available then you can follow-up with a whole range of questions to get to the bottom of the problem.

A very important aspect is your own frame - what's your attitude to the problem, but also to understand that how the information is presented (or by whom) can affect your response.

For example, if you're not on amicable terms with the stakeholder you might adopt an aggressive questioning attitude and not be receptive to the information to be able to react/respond with useful  follow-up questions. 

Or: You're feeling very tired (or not as alert as usual) and so you miss some implication in the question (requirement) - and act on the first layer of information: Yes, we can send a man to Mars because we can build a rocket and life-support system (but how much has his/her muscles deteriorated by the time they return to Earth - and so how much recovery time is needed, permanent damage(?) etc, etc..)

Can you guarantee that this correction package will work?

Yes, I've heard that question in the past. Working out where to start tackling that question is a whole different post - but really there is a whole different bunch of questions that the questioner/stakeholder has and he/she expresses the "simple" (compressed) question to me - but really I need to get behind the question and understand what their "real" problem is - one way is to use Gause & Weinberg's "context-free questions" from "Exploring Requirements" (here's a transcription from Michael Bolton). Another way is to use some of the techniques from "Are your lights on?" (Gause & Weinberg again.)

Framing

Most of this (for me) boils down to framing and how that influences both our problem analysis, information intake, problem exploration and ultimately decision making. We all have it - mostly without realising the affect it plays. But the important aspect is to (try to) be aware of it and some of the problems that it can cause - then we have a better chance of answering the question (requirement) in the real spirit that it was asked!

By the way, it did occur to me that this could be misconstrued as another "exam-bashing" link - but that's not the intention :) If that thought occurred to you, then that maybe says something about your frame.


Are you aware of your own frames?

Oh, this was my first transcription from 750words - thanks to Alan Page for tweeting about his use - I'm brain-dumping regularly now!

Links:

Quiz: 

• http://www.bbc.co.uk/news/science-environment-12758575

Context-free questions: 

• http://www.developsense.com/blog/2010/11/context-free-questions-for-testing/

Framing:

• http://en.wikipedia.org/wiki/Framing_(social_sciences)
• http://en.wikipedia.org/wiki/Framing_effect_(psychology)