A Thought Experiment on Definitions

Thought experiments are a very powerful tool. You probably use them a lot without realising! Any time you wonder "what would happen if..." or "what is a possible consequence of that?" then you're making your own mini thought experiment.

Einstein used them to develop his ideas around relativity. A recommended documentary on this here.

I recently posed a thought experiment on twitter

thought experiment: what happens when one agrees on purpose behind a definition but not agree on it's usage..

There are a number of layers to this question.

• Purpose
• Definition
• Agreement
• Usage

Purpose
This is the "why?" question. What problem are you trying to solve, and does the definition and usage examples help solve it?
So, what might happen if a usage of a definition doesn't appear to agree with it's intent, i.e. they are not congruous.

Definition
This is getting into the correctness and relevance area. Is the definition too narrow or broad? Is it circular? Is it complex to understand. Is there some guidance to help understanding?

Agreement
Complex or obscure definitions may be harder to agree with. Is the definition accessible, useable and congruous. Is there controversy or disagreement? Is that due to the purpose-definition-usage parts not being in synch? Is the definition generally accepted - de facto agreement?

Usage
Is it clear how such a definition would and wouldn't be used? Are there any examples, or patterns and anti-patterns of usage somewhere - or indeed any guidance at all.

It's not necessary for a definition to have usage examples or guidance. But it might help the case. Think about dictionaries - do they often, usually or seldom include examples of usage or guidance notes? (I think the answer would, of course, vary with the dictionary used.) This question would seem to be more relevant if the definition is complex or is difficult to accept.

What might symptoms of non agreement between definition and usage look like?

• Dislike of the definition (fit for purpose? relevance?)
• Aversion or uneasiness with the definition (understanding, clarity?)
• Misuse of the definition (understanding, clarity?)
• Non-use (relevance, clarity, understanding?)

Conclusion

To me there are a number of consequences if such a contradiction crops up between usage and definition.

• The definition is not clear or complete.
• The usage of the definition is not clear or illustrated.
• The definition is misunderstood.
• The definition is communicated in a way that doesn't align with the definition.
• There is resistance to the definition and/or usage - emotional response.
• There is resistance to the definition and/or usage - different paradigm.
• There is resistance to the definition and/or usage - different dictionary references.
• There is resistance to the definition and/or usage - frames of reference.
• There is resistance to the definition and/or usage - little value add visible.
• A combination of the above or even something else.

So, good definitions are generally robust. Unfortunately in the world of software testing many definitions would fail a lot of these tests above. Go look in the ISTQB Standard Glossary of Terms used in Software Testing and try it. Do you find any terms that "don't add value"? 

Example?
Ok, so if I wanted play the school ground bully and pick on the weak I'd start with the ISTQB glossary, but I have higher intellectual ambitions, so...

I've been thinking about checking recently, let's try there.

Checking
I would say I have had a certain uneasiness with the definition - for reasons I don't think I've always been able to articulate. This could boil down to my understanding or the clarity of the definition or something else.

It could be that this feeling is also reflected elsewhere - as recently appeared on the software testing club. The reasons others may give for their "Icky feeling" may be unconnected from my observations, but it would be interesting for them to give their reasons.

Ok, so let's take the checking definition from RST:

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

• “evaluations” as a noun refers to the product of the evaluation, which in the context of checking is going to be an artifact of some kind; a string of bits.
• “algorithmic” means that it can be expressed explicitly in a way that a tool could perform.
• “specific observations” means that the observation process results in a string of bits (otherwise, the algorithmic decision rules could not operate on them).

 Now let's apply it to a stochastic process - eg speech recognition.

According to the definition I can make specific observations (samples of audio) and apply an algorithm to them (for example a speech recognition algorithm). The interesting thing here is that the result is non deterministic (due to speech/accent/pronunciation variation - making the test data design problem difficult) and is going to need some engagement - both for input threshold parameters and analysis of the output. I might get a boolean output (match/no match) or I might get a range (78% match) - and that is a function of the input parameters and the specific observations I ran the algorithm with.

Now the actual algorithm that is making the comparisons is the "checking" part of the process. But this becomes a very small part of the whole - because I need to put effort (more effort and time than the algorithm takes) beforehand and afterwards.

To make this example fit into the current definition I'd have to have all possible samples for certain speech snippets (infinite) or I'd have to define the sample population (this is the test design part of the process - by implication this is part of "testing"). (I won't get into the problematics of the sampling mechanism I use.) So, I'm narrowing the checking part of the whole even more.

So, the question becomes (for me) - should I only use checks where I am certain of the wanted outcome - i.e. a binary answer (which might be "yes/no", "pass/fail", "above 78% threshold/not above 78% threshold"). And here's the problem - I'm quite happy to use scripts as change checkers - or early/leading indicators - they are a mechanism to draw my attention to a result and then ask a question, "should I investigate more or what does this result tell me?". As soon as I am paying attention to the result or thinking about it I am not checking anymore - that's testing.

In this example, checking becomes a very small part of the whole - compared with all the other parts of requirement and test analysis, test design, test set-up and result analysis that make up testing. Then I wonder what value it really adds.

Am I using the definition incorrectly? I don't see any usage examples anywhere, so maybe the definition is incomplete. Or maybe guidance is incomplete. Or maybe the terminology is just not useful for me.

Divergent thought: In the definition of checking it's not clear to me if the algorithm can be a non deterministic algorithm. It could be read in that way - then here's another thought experiment --> what would the consequences of that be?

If I was to revisit the purpose and intent behind this definition I'm not sure that it achieves what it wanted. The checking part is quite small - the other activities in testing are not described so the importance of checking seems to be artificially increased. This is a problem! To me, it would be better to list different tactics of test execution and highlight that checking is one of them.

So, in this example, the "checking" is a very small part of the whole and falls into (for me) a very narrow definition, with a certain amount of ambiguity. (It's narrow as it is contrasted with testing. This is analogous to a "testing vs test design post".) The definition is incomplete and/or incongruous (no usage example and generates confusion and discussion) and fails to add value (as it seems to artificially inflate the importance of checking in relation to other testing activities).

Note, it's taken me quite a while to come to this conclusion - I have needed to put an amount of time thinking around this. It's certainly not an obvious conclusion. And I can also understand if others don't have the time, energy or inclination to do this type of thought journey and treat it as a heuristic to help in their communication. And I also understand that this term is helpful for some people and they have success in using it with their stakeholders - again if this heuristic communication works for you - fine.

Final word
It seems to me that there are many definitions around in the testing and software testing community that could benefit from this type of approach. Do you agree? Which would you try it on first?

Potentially Related Posts

The Conway Heuristic
Communication Heuristic: Use Cases
Thoughts around the label "Checking"

New "What's it all about?"

Since I started this blog in April 2009 I had the word "quality" placed somewhere near the top of the page - specifically in the "What's it all about" box. I've just made a change from:

The Tester's Headache is sometimes about issues connected to the balancing act of executing a test project on Time, on Budget and with the right Quality. Other times it's a reflection on general testing issues of the day.

to:

The Tester's Headache is sometimes about issues connected to the balancing act of executing a test project on Time, on Budget and trying to meet the right expectations. Other times it's a reflection on general testing issues of the day.

I've been doing a bit of sporadic writing/thinking about this recently, the occasional tweet and the odd comment on selected blog posts. There will be more to come soon (I hope - all to be revealed soon) on my thoughts around the word 'quality' in the testing-specific domain.

I'll still occasionally use the #qa hashtag on twitter - hey, I can't start a revolution/re-think from the outside can I? Or can I?

Thoughts, comments? Provocative? Thought-provoking or sleep-invoking?

Problems and Context Driving

I got involved in a small twitter exchange about problems, bugs and perception of the problems the other day with Andy Glover and Markus Gärtner.

Problem Perception
There was a view expressed that a user's perception of a problem is a problem.

I'd agree with this in most "normal" cases. But then there was a knock at the door and Mr. D. Advocate lent me his hat.

So putting the devil's advocate hat on I thought about how this view might not be enough. Or borrowing De Bono's phrase, "good, but not enough".

Cars
My way of looking at this for a counter-example was to think of driving a car. If I press a button, or depress a lever, expecting a certain response or action and something totally different occurs is this a bug? It might be, depending on what the button/lever was and the response.

If I'd pressed the button marked AC and the radio came on - I might think, "there's a problem here".

If I'd pressed a lever for the windscreen wipers and the indicator blinkers started then I might double-check the markings on the lever.

Blink Testing
BTW, Anne-Marie Charrett labelled this lever mix-up as an alternative form of "blink" testing!

Further Ramblings
I then started having flashbacks to my own encounters with cars in different countries and how I'd understood the issues:

• Being stuck in an underground carpark at SFO, engine running, facing a wall, unable to reverse as I couldn't release the parking brakes - there were two and one was hidden. Manual to the rescue. Perception issue about where I'd thought the parking brake release would be.

• Driving to San Jose (same car) and a heavy rain shower starting - how the heck do I switch the wipers on? Pull over and get the manual out. Perception problem about where the lever normally is.

• Opening a taxi car door in Japan - taboo. Soon followed by the next of trying to tip the driver. Applying customs of one culture to another. My problem.

• Driving someone round a roundabout (left hand side of road) when they'd only ever experienced driving on the right hand side. Were the muffled screams a "problem" or just a perception issue?

• Trying to park in Greece - not on the pavement, unlike everyone else. Problem with local customs using my perception of a norm.

• Sitting in the back of a taxi on the way to the office outside Athens going round a mountain pass whilst the driver is reading a broadsheet. Is my anxiousness a cultural thing?

• Being surprised by overtaking customs in Greece. Flashing lights before the maneuvour starts. Irritation and confusion. My problem, not being familiar with the local customs.

• Driving round the motorways of France and Italy - minimal gaps between cars - my problem, perception problem?

• Driving in Sweden - coming face-to-face with a moose at speed. My problem. Obviously the moose has right of way!

Image by Steffe via Flickr

Problem vs Perception?

Problems can have fixed interpretations (this is an agreed issue) and areas of vagueness. Is this my perception or interpretation?

As testers I think we try to root out and understand what our perceptions are and then understand whether they are reasonable, on track or in need of further investigation.

Some of this might be working against a well documented expectation or at other times not knowing what to expect - I deal with both extremes.

One way I handle this is to keep an open (and skeptical) mind and then work out what my hypothesis (interpretation according to the observation) is and compare that with the product owner's.

Sometimes there's no clear-cut right or wrong interpretation.

But it helps to keep an open mind with perception (and borrow Mr. D.A.'s hat now and then).

Have you had any problems with perception lately?

Analogy! He's got an "ology" (almost)!

I find myself using analogy more and more these days. I like it for a number of reasons:-

• It helps get the message across
• I like humour
• I like to approach the problem from different angles

Getting the message across
Sometimes the message is difficult. Not everyone is tuned-in to your way of thinking. Tor Norretranders wrote in the User Illusion that people need to go through a "synch'ing" process before "real" communication takes place. This is a bit like a protocol handshake to establish that both sides are ready to transmit and receive.

Analogy plays a part here. It helps find a common ground, history, context, experience or whatever you might want to call it.

Humour and off-the-wall ideas
I'm a divergent thinker. It's in my nature to look at problems from different angles. I work a lot with strategy decisions - sometimes that means taking a new/radical/different view about something or wanting to try something "new" out. One of these "problems" is communication. Getting people on-board and getting your message across and gaining acceptance for the message

Humour can be like an ice-breaker for new ideas, a door-opener. It allows the salesman to get his foot in the door and start a dialogue. Humour is not the subject or main attraction it's just the warm-up routine or the introduction.

Traps
There are pitfalls or traps associated with analogy. It should be clear what the subject/purpose is - that it isn't the analogy itself - "why is he talking about comfy chairs again?" There is a danger with being too divergent, too tangential, that you actually lose the audience....

This has happened in the past and will probably happen in the future. This is where perseverance comes in (and trying alternative approaches.) If you want to get your message across you have to work at it!

I sometimes joke that if someone isn't agreeing with me then I haven't discussed the topic with them enough.

Sounds arrogant?

Absolutely not - you find common ground - the discussion becomes a dialogue (from the  ancient Greek meaning where the group discovers insights that the individual cannot achieve) - and everyone learns together.

So, when I think about testing problems and concepts I use analogy a lot. I've used it in the past when thinking about testing myths, communication and I'll be using it (probably coupled with a bit of left-field humour) in the future.

I have plenty of humour candidates and problems/concepts that could do with a new look - so keep an eye out for the analogy-ometer ticking away!

So, when did you last look at something from a different angle?

Are you a divergent thinker?

#softwaretesting #teamwork

Almost sounds unpleasant doesn't it? Something that was frowned upon in Victorian times? Well, I am a divergent thinker, and more...

Problem solving and learning fascinates me - especially the interaction with teams. The how and why people learn in different ways and how they approach different problems. Luckily, sitting in a software test engineering environment, I get to think about this, put it into practice and continue to learn and develop daily.

Whenever I sit down with someone to look at a problem - whether it's brainstorming, bouncing some ideas around or to help someone out - I'm intrigued by the different approaches and paths people take to tackle a problem.

Problem solving is like going on a journey - lots of different ways to do it (paths to take) - hopefully arriving at the same destination (the problem in question is solved.) This has a very close analogy to the way people think and learn - lots of different ways to do it, to hopefully achieve the same result.

So, what am I?

Many years ago I was involved in a self-managed learning program. Out of that my instructor and myself came to the conclusion that I was a divergent thinker. Okayyyy I thought...

Is that all?

No, we also concluded that I was an emergent (constructive) learner. Not bad, but what does all this mean?

Testing relevance?

Some key attributes are:

Emergent Learners

- Building knowledge from the bottom-up

- Diving-in to learn from experience rather than from a book

- Evolutionary - adapting to change and being self-organising

- More DIY-learning...

Divergent Thinkers

- Brain-storming

- Creativity

- Looking at the problem from several angles -> "Big picture" & "out of the box

Sounds great to have these qualities in a test team.

Of course, there are drawbacks with these. Emergent learners may have holes in their knowledge as they haven't gone via the classroom - but I think the ability to adapt and learn makes-up for this.

This doesn't mean that one type of learning/approach is better than another - as with any team, it needs a healthy mix of all types - then everyone is backing each other up... That includes mixing high-flyers and less-able members - it's the team that counts, not the team hero.

Similarly, I wouldn't want a team of only brainstormers (or even brain showers!) - all that whiteboard and post-it activity - they'd never do anything!

In practice, is it true? Has it worked?

In early 2005 when I joined an organisation as a system integrator - with part of the role being to establish a system integration process and organisation - I was the only tester not to be sent on a training course for the HW/platform we were using. My manager's explanation: "we're time stretched and you're the sort of person that will learn more without the course..."

Well, I didn't completely agree - it was like a bit of tightrope walking without a safety net - but I pulled on my waders (or was it diving suit) and jumped in at the deep end.

Horrible mix of metaphors!

True enough, I adapted and evolved both a strategy and process - this attitude fitted nicely as the org was an incremental development project in start-up, so there was plenty of change, modification and adaption.

Does this mean these traits are more suited to incremental development? Yes, they're good to have but again I think it's the healthy mix of competences that makes the project work. Everyone has elements of all learning styles - just that some are more practiced than others...

Is that all then Ted?

Recognising and understanding the differences is what's important. As a team leader it's very handy to know (or guesstimate) these styles in different team members because adapting the presentation to the individual wins every time! Also tasks can be shuffled around to match strengths, or even exercise/improve weaker areas.

As an aside, if you're interested in mind-mapping (will appeal more to the divergent thinkers) there's a lot of good software tools out there. One I've been playing with recently is bubbl. Quite nice.

Don't confuse emergent learning with emergence - this is another term I'm interested in - more to do with how teams can self-organise...