On Thinking about Heuristic Discovery

How do you spot new heuristics?

How do you identify new heuristics?

These were questions that were “touched on” during parts of SWET6* (a small part in the open season of James Bach’s discussion and during a car journey from SWET6).

“Pause & Reflect” / “System 2 Re-Insertion”

During the open season of James Bach’s topic on documenting heuristics from a test activity I remarked that I’d spotted an heuristic that he didn’t appear to notice. It was the activity of pause and reflection: putting down the work for a while, revisiting, correcting and re-working. This can cycle through after bouncing ideas around with colleagues or getting their feedback, followed by then further ‘pause and reflection’ cycles until the result was deemed good enough. 

Reflection

Due to the nature of how the activity and report had been made it incurred a series of pauses (interruptions or breaks) which then (I assert) meant that some of the parameters of the context (or frame through which you look at the work) has to be remembered, reviewed or picked-up again - this can mean that the frame gets slightly altered, i.e. “you look at it with a fresh pair of eyes” and, hey presto, see something different or new.

I do not claim to be the first to spot the power of this activity - but I think of it as re-activating the System 2 mode of thought (according to Kahneman, ref [1] this is more deliberate and takes conscious effort to use). The action of putting the work (report) aside for a time and then revisiting means that some of the context is forgotten and so has to be remembered when the work is picked up again. This is the re-analysis in the system 2 mode of thought.

Noticing New Heuristics

The pause and reflection about the observations is very important. It gives the basis for new pattern recognition. Either you notice something different that you don’t recognise or that is a little different. Typically this might be a procedure that you followed, for example:

• I tend to find problems of type X when I do A, B & C, 
• I tend to find race condition problems when I alter (shorten and lengthen) the timing between certain test steps.
• I tend to notice new/different patterns when I pause and reflect on a set of actions and compare with previous times I did some similar action.

Blink Comparator / Trawl and Compare

The activity of reflection and noticing new patterns is something I think of as a blink comparator test. A blink comparator test is something astronomers used to use to find new stars. A reference pattern is used to compare with a current observation. It’s like trawling through previous experience (observations) and comparing with the latest experience. In the example above it might be:

• When I find problems of type X (in the past) what was the same/similar to the current actions (A, B & C)?
• When I have found race conditions in the past were there any timing differences in the actions I made in the test steps? (When I change the timing between steps - for the same types of steps - I find race conditions more often.)
• When I notice new patterns is there some key step that is common? (Pause and reflect,)

Visualization

Sometime in January I tried to sketch this connection between Pause & Reflect and how I noticed new heuristics. This is my latest version that I think reflects the process I tend to follow:

A larger version of this model can be viewed here.

"Pause & Reflect Heuristic"

The model is best read entering the “Pause & Reflect Heuristic” box. 

Here observations of actions (or reviewing a test report) is done - with a pause & reflection in-between - so the two “Solution / Result” clouds may be different. The “Evaluate” cloud is where a difference is noticed. In some cases this feeds back as re-work, in other cases it causes a question which then feeds into the “Proto-pattern” cloud. 

Blink Comparator Test / Trawl and Compare

This is the step where a search of the difference (comparator item) is made. The search might be a re-analysis of similar experiences to understand what caused the different result. The result / assessment might be that it was a random action, or something outside my control, that caused the difference**. At other times it might be, “step X was made when the system was in state Y” - that might be enough to give me a new useful rule of thumb (heuristic).

The next step would be to search if it was something in use elsewhere or known already. If new, then classify (name and describe) and publish (talk about or discuss it).

Question, Test, Evaluate

But the work doesn’t stop there. When using this heuristic new observations about it’s use should be gathered. Does the heuristic work as before? Is there a new (maybe more subtle) pattern or action that makes it useful. This might result in a new heuristic, a refined heuristic or a restriction to fewer applications. This is the “Question, Test, Evaluate” yellow box. It is analogous to the scientific method where the heuristic is the object (hypothesis) tested.

Testing the model

I’m currently collecting new observations of noticing new patterns and trying to construct ways I can test this model.

In the meantime, comments and suggestions are very welcome.

References

[1] Thinking, Fast and Slow [Kahneman; 2012; Farrar]

*SWET6 was at Hönö Hotell, Öckerö, Sweden, 19-20 October 2013, Attendees: Martin Jansson, Steve Öberg, Saam Koroorian, Mikael Jönsson, Anders Bjelkfelt, Marcus Möllenborg, Klaus Nohlås, Simon Morley, Henrik Emilsson, James Bach

**Note, a “different result” might be, when I tackled the problem now I got a different result than previously. What made the difference? This can applies to systems of people interaction too.