" #softwaretesting #testing #cognition "
I saw a post on the STC, here, about best-in-context practices. I started thinking about 'best-in-show' (a la crufts) and best practices in general.
New ideas, terms or phrases usually trigger the same question: 'what problem is this trying to solve?'
BP in General
Best practices. Why tout them? It's a sales gimmick - 'here's my way of doing it and I think everyone should do it. And I can show you why it will solve your problems too'.
It's a sales pitch - and sometimes the best salesman wins. Or, it's a marketing gimmick - if you create enough exposure to the 'best practice' then it gets assumed to be 'best' because there is so much discussion about it.
Systems Theory
Applying 'systems theory' - creating a best practice is like creating a local optimization - there's a danger it will sub-optimize for the whole system, ref [5]. Best practices are inherently localized.
Think of the Spanish Inquisition (the real one and the monty python version) or '1984' and the thought police, ref [7]. These organisations would probably have been thought to be quite effective to the ones that created them - that's the paradox of local optimizations, there's always at least one angle from which they look very good (or even 'best').
Good for the whole though? Here, of course, a lot depends on your idea of good and the frame in which you think of the problem - and the volume of your voice or marketing.
Unfair comparisons? Problem with framing?
Thinking in frames and framing the problem and choice! A problem with 'best practices' could be that:
- The problem has not been thought through to achieve a consensus of what practice to adopt - then someone makes a choice
- The problem and it's nature is not understood (ie in the next project the problem is different - this is hard work… - let's stick with what we have)
- A practice is used and gets labelled 'best' afterwords (maybe not by the original practice decision maker)
In the third case, it's use implies it's a 'best practice' even if it's not. Think from a typical/average manager's perspective - why would they use anything that wasn't best, so if it's in use now and isn't causing huge pain (or seems good enough) then they can be tempted to call it a best practice.
A Snapshot in Time and Good Enough
Personally, I think of best practices as ephemeral - a dragonfly that lives for a short time, and then it's gone - ie 'best' is like taking a still picture with a camera - there (at that timestamp) it's best - but it might not apply anymore….. That snapshot might not be 'best' if I started comparing with other snapshots…..
So, why search for 'best'? Why not search for 'good enough'? Or 'good enough for the problem in front of me'? To me, that would imply active thinking…. But achieving consensus about which practice to use might be simpler than you think - but working on your assumptions is needed. For example:
- What problem is trying to be addressed with a best practice?
- Is it a project manager/company that wants to create a 'standard' that they don't want to maintain?
- Is this a money-saving approach? Again maintenance.
- Is it telling the people not to think?
Maybe by paying people not to think (lower paid) then a practice needs to be adopted that is low-maintenance. This seem to be a dangerous game…
Perhaps companies tolerate them as long as the black swan event can't be traced to their choice of practice (or reification of it being 'the best'). Maybe they genuinely don't realise any better. Maybe the difference in practices is judged to be too small to not warrant the need for constant evaluation or judgement. (I suspect this is a default - coupled with cognitive fluency, see below)
But if this was a default option shouldn't there be less advocating/marketing/discussion of best practices? Well, progress implies (to many) re-invention or creation of new ideas, therefore 'discovering' a new best practice is quite marketable.
Smoke and Mirrors
Is the idea of a 'best' practice an illusion? Software development (including testing) is knowledge work - what's the 'best' way to think?
Is it not the case of an application of an industrial metaphor to knowledge-based work? Mmm!!
Is it a problem with language?
- "Practice makes perfect."
Implies there is a 'best' - this is also an example of cognitive ease (see below) and will be more easily remembered as being a 'good' guide.
- "Practice does not make perfect. Only perfect practice makes perfect."
Not achieving 'the best' is probably politically incorrect in business circles - so there is pressure to say this or that is 'best'. But, remember, this is knowledge work.
Think of any award or recognition in the sciences - we don't say 'X is the best economist/physicist' - they are usually identified for their contribution. In the same way, we should be particular about any 'practice' that we use - what is it good and not so good for???
If you can't think of issues with the practice then you probably haven't thought hard enough.
Is it a cognitive problem?
The sticking point is 'best' - it's a superlative. It's natural that any manager would like to advocate and advertise that they are doing the 'best' possible. It becomes even worse if they have had a 'best practice' in the past - then it is harder to move away from such a practice as this is hard work convincing their peers why this is necessary.
Behavioural Economists and Psychologists also have a related theory - it's called cognitive fluency, ref [2], or cognitive ease, ref [3] - and it's the condition when something is easier to think about affects our choice. It has been noted that when something is easier to think about then it might be more readily believed to be true (or correct, or remembered - depending on context) - ie there builds an assumption (bias) that it is true (correct or something seen before).
This ease of thought can be obtained via familiarity (repeat messages), bolder and easier to read type and even colour.
So, if anyone (manager/stakeholder) has ever had success (however they gauge that) with a best practice in the past, then they are more likely to think of best practices as being a good thing, and their particular best practice in particular.
It is much easier to take an off-the-shelf practice for use in the next project rather than think it through and check whether it is good enough or not.
The opposite of cognitive fluency - cognitive disfluency - also holds true, it acts as a warning signal. So, forcing someone to re-evaluate what a 'good enough' practice is is always going to be harder than taking a ready-made practice off the shelf.
Assumptions
'Best practices' don't usually display their magnitude of pre-requisites, conditions, exclusions and terms of use. It's like the various "terms of use" that come with many SW applications - many don't read them.
Why, then, should a project manager read and be aware of all the conditions of use for a 'best practice'? It's hard work. Managers/stakeholders usually assume (or hope) that knowledge workers are 'doing their best' under the circumstances.
And finally...
The next time someone talks about a best practice, go easy on them - they are following an evolutionary pattern ('if it's familiar it hasn't eaten me') therefore, the onus is on us to highlight the shortcomings of the particular practice (if there are any to be found) or judging why it's fit for purpose.
That activity is important - it's like going through a pre-flight checklist - you see the potential warning signs when you can still act upon them!
Also, never present anything as 'best' for knowledge work - cognitive fluency will mean it becomes a best practice next time. Avoid superlatives (best) and indications of certainty or insurance, ref [4].
References
[3] Thinking, Fast and Slow (Kahneman, 2011, FSG)
[5] Thinking in Systems: A Primer (Meadows, 2008, Chelsea Green Publishing)
[6] Flikr: Crufts Best in Show Roll of Honour
Great post Simon, lots of useful ideas there.
ReplyDeleteThe points about people needing to 'think' are very stand-out for me, it's what people don't do or they assume others have. Best Practice is adopted often because there's a lot of noise around it, by noise I meet marketing and communications stuff.
Mark
The same could be said about any product, service, idea or methodology, right?
ReplyDeleteCertifications are marketed as having the ability to produce quality testers which isn't always the case. It doesn't necessarily mean the information learned for the certifications has no merit.
What about exploratory testing? Perhaps at its grassroots, there is value, but now everyone is grabbing hold of buzz words and using them to drive blog traffic. Does that mean there's no value in an exploratory approach? Clearly not.
@Anonymous
ReplyDeleteI'm not quite sure what you're saying…
"The same could be said about any product, service, idea or methodology, right?"
Which part of the post were you referring to? Marketing? Yes, there are dangers with any marketing or sales effort. How do you confront or cope with that?
"Certifications are marketed as having the ability to produce quality testers which isn't always the case. It doesn't necessarily mean the information learned for the certifications has no merit."
Do you believe that the certification alone produces the quality tester? How? What learning are you referring to? Do you have experience of learning something from a software testing certification that could only be leant there? In my experience, here, I learnt nothing new. Earlier in the year I asked Dorothy Graham about the motivations of certifications compared to their original goals - you can judge, here, whether or not my questions were answered.
"What about exploratory testing? Perhaps at its grassroots, there is value, but now everyone is grabbing hold of buzz words and using them to drive blog traffic. Does that mean there's no value in an exploratory approach? Clearly not."
I'm not sure what you're saying here either, can you help me out? In your second sentence you seem to be saying that ET is used as name magic by everyone? If you see someone using ET for the sake of it and without any substance then you should call them out about it. I hope you do.
You seem to be implying that I'm taking any buzzword or popular phrase and saying that it can't be used. I'm fully against name magic, yes - and I called-out the usage of BP and hypothesized some reasons why it is sticky (has a staying factor):
(1) The psychological connotation of 'best' -> how that becomes something to aspire to and also makes it difficult to move away from using it;
(2) the fact that people use a phrase/label without thinking about it's assumptions or limitations (not specific to BP);
(3) it's 'sitckiness' due to something called cognitive ease (or fluency) which makes it desirable (especially if you don't want to think through it's implications);
(4) and I speculated about it's application to knowledge work not being appropriate.
Feel free to come back with additions or clarifications. If you want to take the discussion offline a contact email is under "What's it all about?"
Thanks for this. I've been meaning to come back and read it properly for a couple of weeks, and our exchange on Twitter about Kahneman's book reminded me.
ReplyDeleteThese are all important points for testers to bear in mind. We have to be flexible and questioning, never assuming that we have got a perfect process that will guide us to the correct outcome.
The link with behavioural economics is very interesting. I studied Economics at university. Something that made a huge and lasting impression on me was the realisation that you could never truly master the subject, or any part of it. All you could do was peel away layers of misunderstanding.
Each successive year of the degree would entail probing deeper, going beyond the superficial understanding we'd acquired up to that point. We'd learn to analyse, probe and understand more. We never fooled ourselves into thinking we'd cracked it. Perhaps the single most important thing I learnt at university was that asking the right questions was more important than thinking I had the right answers. Once your mind starts to work that way you feel very uncomfortable trying to manipulate a problem to fit a predefined "best practice".