SW Testing: A Mystery and a Puzzle

 #qa #testing #softwaretesting !

Mystery and Puzzle? What's the difference.

According to Treverton (from Malcom Gladwell's article) a puzzle is where you don't have enough information to come to some conclusion or recommendation. A mystery is where there may be too much information to really get to the core of the problem and understand it.

I quite like this distinction between the two - I say that without looking up dictionary definitions because I can readily agree with it - and use it.

The typical day of a tester is full of puzzles and mysteries.


Puzzles

Sometimes the tester is faced by a puzzle: There is not enough information to determine is something is working "correctly", where the fault may be (problems with debug implementations) or ambiguous descriptions (the ambiguity is one of the faults.)

In these cases it takes a combination of the research scientist and investigative reporter mentality to persevere and dig/research/test/debug for the additional (missing) information. This is the information needed to enable you to make sense of the findings.

The next person in the chain might make some different sense - or there might not be enough information for them - then it might come back to you to dig for more information to help solve the stakeholders puzzle.


Mysteries

A mystery is a dangerous trap that hits testers quite frequently. This is where there is too much information to make sense or some judgement about what is going on. It might be that several variables have altered at the same time and so reading the "headline" figures is not enough - there is too much noise.

As Weinberg says too much testing can create to much information to cope with - a good counter-argument against too much testing.

So, how to find the way through a mystery?

Sometimes it requires the domain knowledge to help. This is someone that has the baggage of "oh this combination of results can indicate x or y, trying investigating with feature z deactivated..."

Sometimes it's very good to have a fresh pair of eyes on the case - just to get that different perspective. This is a perfect opportunity to bounce ideas around with teammates, maybe even request a pair-testing session.

At other times, it might require walking away from the problem for a while - taking a break - and letting the subconcious start processing the information (the subconcious mind processes data at a much higher rate than the "concious" mind, according to Norretranders.)


Puzzle or Mystery?

So, how do you know that you have too much or too little information to deal with? Start by reducing the information inputs that you have to a minimum - the point where you can make an assumption, "based on the results I can hypothesize A - and then test against it." If you can't get to this minimum then you probably don't have enough information.

I feel a whole series building up around tackling test puzzles and mysteries... To be continued.

However, as far as puzzles and mysteries go I like Churchill's quote (when talking about Russia), "It is a riddle, wrapped in a mystery, inside an enigma" - I have days like that sometimes when trying to deduce what's going on during traffic testing...

Have you thought about the puzzles and mysteries you face in your daily work?