The Tester's Headache

Model Based Testing: Some Assumptions and Traps

2012-01-09T00:31:00.000+01:00

" #softwaretesting #testing #modelling "

Subtitle: Model Based Testing: What Aren't You Telling Me?

Full Disclosure
I worked on a project that used 'formal' model based design and 'formal' model based test in 1996 (I lead the test team). It was a very successful project, we found lots and lots of issues earlier in the design chain than we'd ever achieved previously. The resulting software did the job that was intended with very few downstream faults.

So, I've seen some examples of formal design and test models up close, and understand some of the benefits and issues they raise. I have been able to use them in the past to explore early design models, and find ambiguities early.

Now
In December I had a very interesting discussion - about model based testing - I was there as a 'customer' and model based testing was being 'sold'. I made my skepticism known before the meeting, by my assertion that all testing is based on models - and that some people are better at observing, constructing, expressing and examining them than others. I also stated I'd seen the power of modeling techniques that allowed testing earlier but that they are far from the complete picture.

Traps
Modeling and Model Based Testing has a certain lure, a siren call if you will. But it's not all plain sailing. So, one of my questions was, "what aren't you telling me?" This was quite naturally met with, "what?" "Ok, what assumptions have you built into the story I'm hearing?" "What?"

Trap #1: Are the assumptions about the model visible?

Completeness and Correctness
"Formal methods, and formal specifications" - these produce models that are "correct". Not correct in the sense that they are fault free, but usually syntactically correct - that the grammar holds together. However, "correct" is a sticky word. Say it in the presence of a manager and it's hard to undo the preconceptions that get stirred up.

Trap #2: There is a short distance from formal models are correct (grammatically) to 'the model is correct'.

I probed this idea of completeness, where I was able to find that due to the modelling separating business and implementation/platform logic then it would inevitably mean a group of implementation type activities could not be modeled (like installation, start-up/shutdown/restart, upgrade/update procedures - messy, but crucial for customers in operation).

The implementation of the test model I was looking at was able to be used in 2 main ways: 1: to test against a design model; and 2: to be used to generate scripts for execution on a target system.

I questioned the need for the 2nd step - "tell me why that is useful..." - this showed an assumption that the test model is correct. My assertion was that it's useful but not necessarily correct. This raised a question to me from my "salesman" if I trusted my devised "test cases" (i.e. are they "correct") more than those generated from the model. My answer being that I didn't believe they were "correct" either, but useful.

I stated that the idea of being "correct" - whether in terms of a model (a legacy from the idea of a formal stated model being grammatically "correct") or manually devised test cases (ideas) - was a misleading discussion. Both could be useful, but ultimately fallible.

Trap #3: Models can gain a 'halo effect', ref [1] - meaning that you exclude other sources.

Bias
Assumptions sometime abound. Beware of assumptions!

A not uncommon assumption: The design model is more complex (detailed) than the test model (and by implication more complicated and demands more expertise.)

This was presented in the meeting, to which I was able to demonstrate that the way the modeling had been implemented inevitably meant that the test model would need to include more information than for any corresponding design model. Ooops!

Trap #4: The model simplifies the test space modeling.

Useful?
With all of my skepticism and testing of the assumptions and ideas presented to me, I did actually find some useful elements to it.

One, being a visualization of a complex test space - to aid exploration of it.

Another being as an aid to separate business and implementation logic of scripted and automated test frameworks (a bunch of these being necessary as code checkers for legacy systems).

Note, these are context specific to my working domain.

Recap
Be careful about people assuming the model contains everything (trap #2) and that it becomes a sole source of test ideas (trap #3). If the model doesn't make clear what assumptions it is built on, then they need to be clear and understood somewhere (trap #1). And finally, beware of simple models (trap #3 and #4) as something that advertises the need for less 'thought' - they may be ok, but not always (as I showed).

I had an interesting discussion - and was thanked for my skepticism and the way I'd highlighted problems with the business case. To me it was just another session testing...

Reference

[1] http://en.wikipedia.org/wiki/Halo_effect

Some SWET3 thoughts

2011-11-21T23:37:00.001+01:00

" #softwaretesting #testing #swet3 #peer #conference "

Date: 19-20 November, 2011
Place: Ringsjöstrand, Skåne, Sweden
Weather: Fog

Attendees: Johan Jonasson, Ola Hyltén, Anders Claesson, Oscar Cosmo, Petter Mattsson, Rikard Edgren, Henrik Andersson, Robert Bergqvist, Maria Kedemo, Sigge Birgisson, Simon Morley

I spent another weekend with a group of thinking testers that made up SWET3 (Swedish Workshop on Exploratory Testing 3). An intense and well-organised weekend - many thanks to Henke & Robban for all the work!

SWET runs according to the LAWST model - although we didn't have any 'rat hole' cards, and white was the new red - confusing enough that meant some wrote 'red' on their white cards - there's a lesson there somewhere.

Rikard's Talk: Teaching Testing Basics with Open Source Applications
First up was Rikard with a talk about some of his experiences with teaching testing to students in Karlstad - an interesting talk and a wide-ranging discussion followed. My notes showed I had the following questions/comments:

Which basic techniques do the students learn the easiest?
Any reflections on the learning methods the work better or not so well?
What is the energy in the students like? Is there any relationship to their enthusiasm?
When you work with bug isolation was there any assessment of the student's capabilities before the exercise started?
What patterns do you see in the students and how do the patterns compare with testers out in the 'real world'?
"They want to be be testers" - Is that the way it is for all from the beginning or have you seen it change during the course?
How do you teach them the heuristic test strategy model?
How do they learn quality criteria / how do they learn stakeholder relations?
What problems does teaching with open source applications give?
How do you handle the balance between listening to the teacher (as a beginner) and encouraging a level of scepticism in the students?
How do you handle your own teaching needs/learning?
Any issues with terminology in the class, eg "tests vs checks"?

Johan's talk: Teaching Testing Online
Johan discussed the AST online courses - both from the perspective of a student and as an instructor. There was a lot of discussion comparing pros and cons of the course, covering a wide range of aspects from it's timing, pace, the involvement of the instructors, content, issues and benefits with peer review, time zones, issues with online presence and more. Some questions, from my notes, were:

Have you disagreed with anything in the class material?
What parts of the class material would you change even if you didn't disagree with it?
Are there any dangers for groupthink?
Is there no way to jump straight to the Test Design course if you can demonstrate the right pre-requisite learnings?

My Talk: Mindset Changes: Changing the direction of the oil tanker

My own talk was centred on some of the issues with changing thinking, PR and discussion of and about testing in a large organisation. I may reflect and describe this in more detail in a later post, but some observations were:

The tester view/perception and the non-tester view/perception of testing (in large organisations) is linked and influenced by each other (a circle of dependency).
Good communication is key: without it it doesn't matter what sort of testing you do.
"Many bugs" gets translated into "poor testing" too easily - we need to educate how complex good testing really is.
Testing (in general) needs reframing away from a predominantly confirmatory approach to a more investigative approach (which might include a confirmatory aspect).
Maps for coverage, just like dashboards, are fallible and heuristic in nature.
Test cases and fault/bug reports are heuristics - they may give some useful information under certain conditions but are fallible.
I called the approach to changing views as a Machiavelli approach - meaning subtle and hiding the up-front meaning. The reasoning being to avoid labels and the associated cognitive disfluency - which is present (in my experience) in large organisations.

In a discussion about mapping coverage as a form of visual reporting/awareness - I reinforced the point that maps can be wrong, but they can be useful.

In the following feature map - the components of the feature are drawn out in black, and some test progress in some of the components is indicated in red. I could imagine such a map could leave the observer with a satisfied (or even happy) feeling...

My improvised coverage map

Great discussions, great experiences and lots of food for thought - just the right ingredients for a successful peer conference!

Other SWET Reports:

Notes from SWET3 (Rikard)
SWET3 thoughts through mind maps (Sigge)

Let's Test!

2011-11-12T00:32:00.001+01:00

" #softwaretesting #testing #LetsTest "

Let's Test 2012, 7-9 May, Runö Conference Centre, north of Stockholm

Site: http://lets-test.com/

Five guys got together and formed a committee to organise a test conference. Not just any five guys, and not just any test conference!

Henrik Andersson, Henrik Emilsson, Jonas Jonasson, Ola Hytén and Torbjörn Ryber are the organisers. I know them from SWET1, SWET2 & SWET3 - they don't just know testing, they talk a lot about things you see in "good testing".

They got together and it looks like they've put together a special conference. Three days of tutorials, keynotes, talks, test lab and more. The concept of having the conference and accommodation all in one place - a peninsula in an archipelago just north of Stockholm (of all cool places!) - makes it into an extended, and pretty special, peer conference.

Peer conference: with the emphasis on meeting peers and having contact with them for the whole three days. Conference with the emphasis on conferring! Cool concept, cool location, cool set-up -> cool conference.

Oh, what else is special about this? It's being called the first European context-driven test conference. Some people get scared or worried by the idea of "context-driven testing", but come on: isn't that just a declaration of doing appropriate testing - with your "eyes wide open", brain in gear and eliciting valuable information about the product(?) - yes, it's difficult to do well, you have to work at it and continue to learn.

So, if you like learning, are partial to good discussion and want to meet like-minded people then I think this is a very interesting conference to attend.

A test conference, by testers for testers - pretty special!

Oh, and there's one week left for submissions (call for papers closes 13 December).

Best Practices - Smoke and Mirrors or Cognitive Fluency?

2011-11-03T00:10:00.000+01:00

" #softwaretesting #testing #cognition "

I saw a post on the STC, here, about best-in-context practices. I started thinking about 'best-in-show' (a la crufts) and best practices in general.

New ideas, terms or phrases usually trigger the same question: 'what problem is this trying to solve?'

BP in General

Best practices. Why tout them? It's a sales gimmick - 'here's my way of doing it and I think everyone should do it. And I can show you why it will solve your problems too'.

It's a sales pitch - and sometimes the best salesman wins. Or, it's a marketing gimmick - if you create enough exposure to the 'best practice' then it gets assumed to be 'best' because there is so much discussion about it.

Systems Theory

Applying 'systems theory' - creating a best practice is like creating a local optimization - there's a danger it will sub-optimize for the whole system, ref [5]. Best practices are inherently localized.

Think of the Spanish Inquisition (the real one and the monty python version) or '1984' and the thought police, ref [7]. These organisations would probably have been thought to be quite effective to the ones that created them - that's the paradox of local optimizations, there's always at least one angle from which they look very good (or even 'best').

Good for the whole though? Here, of course, a lot depends on your idea of good and the frame in which you think of the problem - and the volume of your voice or marketing.

Unfair comparisons? Problem with framing?

Thinking in frames and framing the problem and choice! A problem with 'best practices' could be that:

The problem has not been thought through to achieve a consensus of what practice to adopt - then someone makes a choice
The problem and it's nature is not understood (ie in the next project the problem is different - this is hard work… - let's stick with what we have)
A practice is used and gets labelled 'best' afterwords (maybe not by the original practice decision maker)

In the third case, it's use implies it's a 'best practice' even if it's not. Think from a typical/average manager's perspective - why would they use anything that wasn't best, so if it's in use now and isn't causing huge pain (or seems good enough) then they can be tempted to call it a best practice.

A Snapshot in Time and Good Enough

Personally, I think of best practices as ephemeral - a dragonfly that lives for a short time, and then it's gone - ie 'best' is like taking a still picture with a camera - there (at that timestamp) it's best - but it might not apply anymore….. That snapshot might not be 'best' if I started comparing with other snapshots…..

So, why search for 'best'? Why not search for 'good enough'? Or 'good enough for the problem in front of me'? To me, that would imply active thinking…. But achieving consensus about which practice to use might be simpler than you think - but working on your assumptions is needed. For example:

What problem is trying to be addressed with a best practice?
Is it a project manager/company that wants to create a 'standard' that they don't want to maintain?
Is this a money-saving approach? Again maintenance.
Is it telling the people not to think?

Maybe by paying people not to think (lower paid) then a practice needs to be adopted that is low-maintenance. This seem to be a dangerous game…

Perhaps companies tolerate them as long as the black swan event can't be traced to their choice of practice (or reification of it being 'the best'). Maybe they genuinely don't realise any better. Maybe the difference in practices is judged to be too small to not warrant the need for constant evaluation or judgement. (I suspect this is a default - coupled with cognitive fluency, see below)

But if this was a default option shouldn't there be less advocating/marketing/discussion of best practices? Well, progress implies (to many) re-invention or creation of new ideas, therefore 'discovering' a new best practice is quite marketable.

Smoke and Mirrors

Is the idea of a 'best' practice an illusion? Software development (including testing) is knowledge work - what's the 'best' way to think?

Is it not the case of an application of an industrial metaphor to knowledge-based work? Mmm!!

Is it a problem with language?

"Practice makes perfect."

Implies there is a 'best' - this is also an example of cognitive ease (see below) and will be more easily remembered as being a 'good' guide.

"Practice does not make perfect. Only perfect practice makes perfect."

Not achieving 'the best' is probably politically incorrect in business circles - so there is pressure to say this or that is 'best'. But, remember, this is knowledge work.

Think of any award or recognition in the sciences - we don't say 'X is the best economist/physicist' - they are usually identified for their contribution. In the same way, we should be particular about any 'practice' that we use - what is it good and not so good for???

If you can't think of issues with the practice then you probably haven't thought hard enough.

Is it a cognitive problem?

The sticking point is 'best' - it's a superlative. It's natural that any manager would like to advocate and advertise that they are doing the 'best' possible. It becomes even worse if they have had a 'best practice' in the past - then it is harder to move away from such a practice as this is hard work convincing their peers why this is necessary.

Behavioural Economists and Psychologists also have a related theory - it's called cognitive fluency, ref [2], or cognitive ease, ref [3] - and it's the condition when something is easier to think about affects our choice. It has been noted that when something is easier to think about then it might be more readily believed to be true (or correct, or remembered - depending on context) - ie there builds an assumption (bias) that it is true (correct or something seen before).

This ease of thought can be obtained via familiarity (repeat messages), bolder and easier to read type and even colour.

So, if anyone (manager/stakeholder) has ever had success (however they gauge that) with a best practice in the past, then they are more likely to think of best practices as being a good thing, and their particular best practice in particular.

It is much easier to take an off-the-shelf practice for use in the next project rather than think it through and check whether it is good enough or not.

The opposite of cognitive fluency - cognitive disfluency - also holds true, it acts as a warning signal. So, forcing someone to re-evaluate what a 'good enough' practice is is always going to be harder than taking a ready-made practice off the shelf.

Assumptions

'Best practices' don't usually display their magnitude of pre-requisites, conditions, exclusions and terms of use. It's like the various "terms of use" that come with many SW applications - many don't read them.

Why, then, should a project manager read and be aware of all the conditions of use for a 'best practice'? It's hard work. Managers/stakeholders usually assume (or hope) that knowledge workers are 'doing their best' under the circumstances.

And finally...

The next time someone talks about a best practice, go easy on them - they are following an evolutionary pattern ('if it's familiar it hasn't eaten me') therefore, the onus is on us to highlight the shortcomings of the particular practice (if there are any to be found) or judging why it's fit for purpose.

That activity is important - it's like going through a pre-flight checklist - you see the potential warning signs when you can still act upon them!

Also, never present anything as 'best' for knowledge work - cognitive fluency will mean it becomes a best practice next time. Avoid superlatives (best) and indications of certainty or insurance, ref [4].

References

[1] Framing: Some Decision and Analysis Frames in Testing

[2] http://www.boston.com/bostonglobe/ideas/articles/2010/01/31/easy__true/

[3] Thinking, Fast and Slow (Kahneman, 2011, FSG)

[4] The Certainty Effect and Insurance

[5] Thinking in Systems: A Primer (Meadows, 2008, Chelsea Green Publishing)

[6] Flikr: Crufts Best in Show Roll of Honour

[7] http://en.wikipedia.org/wiki/Thoughtcrime

[8] http://www.softwaretestingclub.com/profiles/blogs/my-take-on-best-practices

Carnival of Testers #27

2011-11-02T02:59:00.001+01:00

" #softwaretesting #testing "

October, Halloween, pranks, scary stuff, ritual and various forms of deity-worship is nothing new to software testing.

But, hopefully, in the selection below there are no ghouls, just good writing! Judge for yourselves...

Natural?

Phi Kirkham made a series of interviews with contributing authors of a new software testing-related book. Here, was a good example with Catherine Powell.

Assumptions, pre-conceptions and hidden interpretations are rife in any social interaction. They're amplified by software. Adam Knight gave an example, here, of how simple it is to overlook pre-conceptions.

The first GATE workshop took place at the beginning of October and got a good write-up from Markus Gärtner, here. Eusebiu Blindu's thoughts on it, here.

If you're in the vicinity Phil Kirkham was announcing an STC meetup in Chicago, here.

Jeff Nyman had some perspectives on quality assurance and what's obvious, here.

A comprehensive report from PNSQC by Nathalie de Vries, here.

Duncan Nisbet, here, gave a run-down of his last 12 months - actual against expectations. Interesting reflections.

A good post about the beta book program, here, from Michael Larsen.

Alan Richardson lists his inspiration for his model of software testing, here.

Retrospectives, de-briefs and reflections are important in any team or testing activity. Lisa Crispin describes how fortune cookies were used as a trigger in a team retrospective, here.

Unnatural?

Regression testing - done well, done badly, done at all, done with any thought? Anne-Marie Charrett explored some issues with aspects of regression testing, here.

"All that testing is getting in the way of quality"? A halloween prank? Eric Jacobson wrote his reflections on the STARWest talk, here. Ben Kelly made a good walk-through of the points in Eric's post, here. Good comments in both!

Supernatural?

Halloween. Is quality dead or undead? Trish Khoo had some thoughts, here.

Men in black or bogey men? Mr TestSheep has some thoughts on them, here. (Well done, NZ in RWC 2011!)

And finally, to wrap-up October and all things ghoulish, Claire Moss wrote about the undead attributes of quality, here.

Until the next time....

Carnival of Testers #26

2011-10-02T19:57:00.000+02:00

" #softwaretesting #testing "

This month there were many flavours on communication...

Joe Strazzere encountered an interesting problem, here, with re-communication of his words without crediting Joe. An important issue that all should be on the lookout for.

A mindmap is always an interesting communication medium. James Bach's CAST2011 mindmap. Here, Adam Knight maps his Eurostar talk.

Rikard Edgren used book form to communicate his ideas and influences around test design, here. Well worth a read!

Good thoughts around conferences, gathering and communicating here from Phil Kirkham.

Communicating about the part testing plays is at the heart of Elisabeth Hendrickson's post, here.

Simon Schrijver compiled a list of free testing magazines, here.

Ben Kelly's take on framing and communication is worth reading, here.

Some interesting points that Eric Jacobson noted, here, on an internal presentation around more effective tester communication.

Testing asshole? Marlena Compton gives some hints to the communication imperatives that will be part of her talk at PNSQC.

Some interesting articles from Joris Meets and how he relates them to software testing, some of the issues discussed are (at the root) about how we relate to (and communicate about) software testing.

Michael Bolton wrote about representing testing problems as testing results as a way to communicate about the product.

Until the next time...

Our survey said...

2011-09-25T15:19:00.001+02:00

" #interpretation #fun #context #cognition "

Whilst compiling material for some other work I stumbled across some old Family Fortunes and Family Feud 'funny/strange' answers on YouTube*.

I've recently being doing a lot of thinking around framing, ref [5], and the problems it can cause and solve and I started thinking about different causes for the unexpected answers.

For communication analysis I use two types of exercise, (1) frame analysis and (2) word and meaning substitution.

Frame analysis

What are the aspects that might be important to each person involved in the communication? This usually revolves around situational context of either the one asking the question (presenting the problem) or the one answering the question (presenting a solution). Here there is scope for a range of cognitive and interpretation mistakes.

Word and meaning substitution

A well-known example of this is the "Mary had a little lamb" exercise, described in "Are you lights on", ref [1], and is a demonstration of how changing the emphasis of a word in a sentence, or replacing a word with a similar meaning (from a dictionary or thesaurus), can change the meaning of the sentence. Therefore, if both parties in the communication intend different word emphasis (in word placement or interpretation) then there is a possibility of confusion.

So, what can appear as confusing or even amusing answers can, with the right perspective, have a certain logic. In the list** below I've made an attempt at finding the perspective behind the answer, in red.

The questions are typically prefixed with "We asked 100 people to name..."

Q: Something a husband and wife should have separate of

A: Parents

Logical answer(?) but maybe not along the intended lines of the questioner.

Q: A planet you recognize just by looking at a picture of it

A: The Moon

Confusion of definition of planet with 'celestial body' (something in space with an orbit)

Q: A month of spring

A: Summer

Slip of the ear, of->after(?), ref [4]

Q: A word that starts with the letter Q

A: Cute

Q. Name a part of the body beginning with 'N'

A. Knee

Phonetic interpretation

Q: The movie where John Travolta gave his most memorable performance

A: The John Travolta Biography

'Most memorable performance' to the questioner had a potentially different meaning. To the answerer either it was interpretted as a film where he featured the most, or he wanted to give an amusing answer.

Q: Something you wouldn't use if it was dirty

A: Toilet paper

Amusing and logical answer(?)

Q: A signer of the Declaration of Independence

A: Thomas Edison

Slip of the tongue, specifically noun substitution, ref [4]

Q: Something that comes in twelves

A: Dozens

Could be logical interpretation but not something the questioner was intending(?)

Q: A sophisticated city.

A: Japan

Misinterpretation (or even slip of the ear) of city for destination.

Q: A kind of bear

A: Papa Bear

Recency effect(?) - had recently been reading or exposed to children's stories (?)

Q. Name a number you have to memorise

A. 7

Misinterpretation of 'memorise' as 'favourite' or 'memorable'(?)

Q. Name something in the garden that's green

A. Shed

Context-specific to the answerer(?)

Q. Name something that flies that doesn't have an engine

A. A bicycle with wings

'Logical' and specific answer - but the questioner could have maybe clarified the question with a 'commonly known item'.

Or, recency effect - flugtag, ref [6].

Q. Name something you might be allergic to

A. Skiing

'Alergic' -> 'don't like'(?)

Q. Name a famous bridge

A. The bridge over troubled waters

Interpreted as 'something well-known with bridge in it'(?)

Q. Name something you do in the bathroom

A. Decorate

Specific to the answerer's context.

Q. Name an animal you might see at the zoo

A. A dog

Generics. Potential that the answerer has not interpreted the the question as 'generally seen and residing in the zoo'.

Q. Name a kind of ache

A. Fillet 'O' Fish (?)

Brain-freeze or 'slip of the ear'(?)

Q. Name a food that can be brown or white

A. Potato

Answerer framed the question as a food which could be presented as brown or white(?)

Q. Name a famous Scotsman

A. Jock

'Slip of the ear' -> 'a common nickname'(?)

Q. Name a non-living object with legs

A. Plant

Maybe thinking of a plant on a plant stand(?)

Q. Name a domestic animal

A. Leopard

Misinterpretation of 'domestic'(?)

Q. Name a way of cooking fish

A. Cod

'way' misinterpreted as 'type'(?)

Analysis Notes

Context - some answers are specific to the answerer and not the questioner. Example traps might be (1) Understanding and interpretation, (2) Word association problems or (3) Relating everything to ones own experience or circumstances.

Recency effects, ref [3] - the interpretation associated with a word was used in a different context, giving a skewed answer. In testing this occasionally results in skewed emphasis of the risk determination - see tester framing problems in ref [5].

Skipping and changing words in sentences - to actually hear a different question - sometimes grouped under 'slips of the ear'. In testing this might result in an incorrect solution application, similar to framing problems but can also be 'straightforward' slips that result in some faulty analysis - missing some key input parameter for example.

Other framing effects can be caused by the previous question, previous answer or even some realisation that a previous answer was wrong/silly and so inducing more stress in the answerer.

Stress can mean that sometimes when you're trying to react you don't actually listen to the whole message or question. This can be time pressure or other stresses. Be aware of this potential problem.

Anchoring effects, ref [2] - focusing on a word and giving an association with that word (rather than focusing on the whole question). In testing this typically results in confirmatory testing.

Generic statements can create confusion. These are generic statements as part of the answers - this is where the question can be confused between giving an example of a specific kind and categorizing the answer into a grouping. Opposite of the answerer-specific problem. More on this in another post...

Don't rule out brain-freezes either - these can be multi-word substitution or paragrammatism, ref [4], which result in nonsense responses.

And finally...

This is a good exercise and quite instructive for those working in software testing - it's a good illustration of how what might be seen as an obvious or simple answer can actually diverge from the expectations of the stakeholder or even customer.

Be alert for not just for confusing messages but also the potential for confusing answers. In this way you might know when to re-affirm your interpretation back to the stakeholder or customer.

References

* If you want to see the clips you can search youtube for "family fortunes answers" or "family feud answers" or "game show stupid answers".
** Lists compiled from

http://www.funny-haha.co.uk/Joke.asp?J=283

http://www.businessballs.com/familyfortunesanswers.htm

http://www.stupidgsa.com/american/family-feud/

[1] Are Your Lights On?: How to Figure Out What the Problem Really Is (Gause and Weinberg, Dorset House, 1990)
[2] Anchoring effects: http://en.wikipedia.org/wiki/Anchoring
[3] Recency effects: http://en.wikipedia.org/wiki/Serial_position_effect

[4] Um...: Slips, Stumbles, and Verbal Blunders, and What They Mean (Erard, Panteon, 2007)

[5] http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html

[6] http://en.wikipedia.org/wiki/Red_Bull_Flugtag

Book Frames and Threads Updated

2011-09-17T10:39:00.000+02:00

" #softwaretesting #testing "

Earlier in the year I tracked some of the influences to my software testing learning, here. I say some, as I exclude blogs and online articles - that is an update for a later time.

The map is intended to track the influences - whether it is something I have in my active reference library or something that sits in the anti-library, reminding me that however much I absorb and use there is always a great deal more out there that I haven't looked at that could be useful. It's a very physical way of reminding me that I don't know everything - and that's good!

In the map I try to track

whether the publication was discovered during my own research or via the test community (either recommendation or discussion),
if the book/article was directly influenced/referenced by another publication (linked by a blue line)
if it's ongoing whether it is active or not
changes from the last version (in pick/read) -whether it has moved (with a line to track) or a new addition

This is a useful reference for me to

track some elements of my learning, influences and research
track what has been current (the changes under the latest period and what that might mean for me)
remind me that I don't track all my influences (blogs, some other online material and publications)
remind me that there is still plenty to research
show that items are added and removed from the anti-library

Future additions that I'm thinking about:

include some dates (addition and/or completion)
include more community influences
include more notes around the publication (what I learnt or felt about it - you can learn without agreeing with the publication)
include a frequency of re-use (how often I re-use it for reference) - a type of star rating for use/re-use

The map is below - I'm more than happy to receive comments on recommendations for publications and sources and even additional information that I might find useful to include.

If it encourages you to do something similar I'd love to hear about it.

Enjoy!

Testing: Do you train like an athlete?

2011-09-04T12:12:00.000+02:00

" #softwaretesting #testing "

I just read this analysis piece on the BBC site about the pressures involved in sprinting, especially the start and run-up to the start.

As I read through it I found myself mentally ticking off links to the testing world:

There is no perfect start
Appearance and presentation is part of the message
Pressure kills concentration
Reacting at the right time
Distractions affecting focus

No perfect start?
The interview contrasts two sprinters with different techniques and how their physical make-up presents different problems at starting and how they handle that. In the software development world this translates to there is no best practice. Every problem and solution is unique - what works for one athlete (or product) does not necessarily work for another.

Presentation of the message
In the article the example is given of Linford Christie displaying his superior physique to other competitors before a race. The intention was partly - I'm prepared and ready.

In the testing world, the message and form in which we give that message is important. Successful message styles are usually truthful, not unduly biased by numbers and consistent. Part of this goes towards building your brand.

Building this trust in your team and stakeholders is vital to successful story-telling.

Pressure kills concentration
All athletes respond and cope with pressure in different ways. Some pressure is exerted by other athletes, some is created by the athlete themselves (their own expectations) and their surroundings.

In the testing world - sometimes the pressure is external - created by teams or stakeholders, sometimes with unrealistic expectations of testing and sometimes because they don't handle/distribute the pressure so well.

This reminded me of the chapter in Pragmatic Thinking and Learning, "pressure kills cognition" which gives examples of overt pressure disrupting thinking. Be on the look-out for this - it's not the easiest thing to deal with when you're on the receiving end, but if anything, don't pass on the pressure - or at least understand what effect it might have.

Reaction times
In athletics the false start is feared - especially for the sprints. The athletes are coiled like a spring and are trying to react as quickly as possible. But, they don't want to react too quickly, and not too late either - they want something that's good enough for them. The interview demonstrates the difficulty here with the game of slapsies (look here and here).

In testing, the similarity might be when to raise or highlight a problem, call in help for the investigation. You don't want to be crying wolf for every issue, just as you don't want to be doing some lone investigation "too long" (and potentially getting stuck). Here is where pairing or having a colleague (tester or developer), that you can run something by, is very useful. Even at a daily stand-up meeting mention what you're currently investigating - sometimes there is someone who says "check X" or "have you got Y set in your configuration".

There is a routine and balance to find here - something that takes both practice and goes hand in hand with your message / brand.

Distractions affect focus
Athletes get distracted by the antics of other athletes, sometimes by the crowd or acoustics in the stadium. They have different ways of dealing with this - some try to get into and stay 'in the zone', some go and lie down and stare at something, whilst others continuously move around to avoid tensing up.

Distractions play a big part in any work environment also, as well as the ways we try and remedy them. It might be the problem of multi-tasking - getting many high-priority demands on your attention and not being able to decide which to devote attention to - or really how to stay focussed on the task at hand.

Sometimes, it's more efficient to simplify the problem into smaller component parts, and so have a feeling of manageability and being able to see progress quicker. Small wins keep attention and focus rather than long drawn-out slogs wading through mud.

Sometimes it's about removing distractions - don't check email for the next hour, close unnecessary browsers (with their flash animations that catch the eye - or use a flash blocker) - reduce the number of open windows to the minimum needed for the task.

Sometimes it's right to de-focus on the problem, step back and look at the wider picture. This helps you relate the problem to it's situational context, re-evaluate why you're doing something, maybe even bring in a fresh pair of eyes to help. This sometimes gives new information or re-affirms the original scope, then you can re-focus on the problem (with any new insights and information).

Are you a person that whilst talking to someone must always answer a phone call (no matter who it's from)? Or do you treat phone calls like someone coming up to you in the corridor whilst you're already in a conversation - usually they'd wait to interrupt your ongoing conversation - so why should it be different with a phone call? (The exemption here is if you're waiting for some urgent or important information which would lead you to interrupt the conversation.)

Awareness
There are lessons to learn all around - especially from non software testing disciplines. The key is to be able to recognise your potential problem areas - whether it's to do with message presentation, knowing when to react, handling distractions or being aware that pressure can have a detrimental affect on performance.

Awareness is an important first step in problem solving - whether you can solve the issue or not - understanding factors that affect your "testing performance" is key!

Carnival of Testers #25

2011-09-02T00:25:00.000+02:00

" #softwaretesting #testing "

Wow!

August was busy - partly due to CAST2011 and Agile2011 conferences triggering the creative and writing juices...

Journey and Influence
Many testers draw influences and experience from outside of the traditional software testing field.

Rob Lambert pondered the Peltzman effect and how it might apply to testing, here.
A journey so far in exploratory testing is given, here, by Albert Gareev.
John Stevenson reviewed different views on the need for product knowledge (or not) when exploratory testing.
The Bach Brother's Legion of Test Merit premiered this month, with the first recipients presented by James Bach here.
Brian Osman looked at some of the areas lacking in good testing, here.
Two interesting perspectives, here, on success and failure from Joel Montvelisky.
A crime series triggered Henrik Emilsson to start thinking about testers and criminals(!), here.
I started thinking about the chapter in Slack when I read Joe Strazzere's post on being fungible (or not), here. Good points!
The what-if heuristic from Daniel Berggren, here, shows how useful a tool it can be for a tester.
Good testing journey in search for mushrooms from Zeger Van Hese, here.
Original and very interesting ideas on creating testing perspectives from Shmuel Gershon, here.
Parimala Shankaraiah continued journal on public speaking is here. Good observations!
Oliver Erlewein writes about a new proposed testing standard on a new collaborative blog, here. Interesting new site!
Important points from Alan Page on test design for automation, here.

CASTing an eye...
Many testers jotted down their thoughts and observations, some being:

The use of Ustream allowed Claire Moss to make some notes on Paul Holland's lightning talk, here. Daniel Woodward made some interesting observations, here.
Day 1 summaries from Pete Walen and Michael Larsen, here and here.
Good rapid blogging from Markus Gärtner, here and here.
Post-conference reflections from Jon Bach, here, Elena Houser, here, Ajay Balamurugadas, here, Michael Hunter, here, Ben Kelly, here, Matt Heusser, here, and Pete Walen, here. Some important observations in them all.
Some more important point were made by Eric Jacobson, here.
Important observations from James Bach on the test competition, here.
And finally, some tips on presenting at conferences, here, from Lanette Cramer.

Until the next time...

Framing: Some Decision and Analysis Frames in Testing

2011-08-22T01:49:00.000+02:00

" #softwaretesting #testing "

What is a Frame?

The following is from Tversky and Kahneman's description of a decision frame, ref [1],:

We use the term "decision frame" to refer to the decision-maker's conception of the acts, outcomes, and contingencies associated with a particular choice.

The frame that a decision-maker adopts is controlled partly by the formulation of the problem and partly by the norms, habits, and personal characteristics o f the decision-maker

When using a decision frame to analyze a problem and come to a decision they call this framing. So, I'll refer to a frame as relating to a decision (or analysis) frame.

Factors at play

As mentioned, many different factors affect how we analyze problems, including:

Temperament/Emotions

Anger
Happy
Optimistic
Pessimistic
Tiredness
Fear (e.g. of failure)

Experience

Lessons from past situations - own experience and feedback
What has been learnt recently
Complacency due familiarity

Strategy

Your own vs someone else's
Aggressive
Military campaign - lots of detailed planning
Reactive

The factors and the weight given to them might be different for:

Stakeholder view ("Upgrade needs to be reliable", "Of the new feature set only x is live in the quarter")
Tester view ("Which risks are most important to look at now?")
Developer view ("I did a fix, can you test that first?")

The stakeholder, developer, tester and any other role in the project has a set view on priorities and aims with the project - agendas maybe - and one challenge is in trying to tie these together, or at least understand the differences and how they impact our communication. They all may have similar product goals but their interpretations to their work may be different - their influences and micro-decisions will be different meaning that transparency in communication is important.

But, there's a catch - the way we present information can affect its interpretation - depending upon the frame that a stakeholder is adopting.

Think of a frame as a filter through which someone looks at a problem - they're taking in lots of data but only the data that gets through the filter gets attention (the rest may end up in the subconscious for later or isn't absorbed), "I have my show-stopper filter on today so I don't notice the progress the team has made…"

So, being aware of the potential different types of frames that each project member might have as well as some traps associated with frame formulation is important.

Stakeholder Frames

Might include:

Emphasizing minimum time for product delivery
Emphasizing short iteration times and delivering quickly
Trying to minimize cost of product development (cost of testing?)
Emphasizing future re-use of the development environment (tendency to worship automation?)
Aiming for a reduced future maintenance cost

Tester Frames

Might include:

Emphasizing the favourite test approach
Emphasizing areas of greatest risk (to?)
Emphasizing the last successful heuristic that found a show-stopper bug
Emphasizing focus on the data configuration that found the most bugs the last time
Emphasizing conformance to a standard over a different aspect of the product
Emphasizing the backlog item that seems the most interesting
Emphasizing widespread regression as "fear of failure / breaking legacy" affects analysis
Emphasizing feature richness over stability

Note, in isolation, some of these frames may be good, but they might not necessarily be good enough.

Framing Problems in Testing

Functional Blindness or Selective Perception

Russo and Schoemaker called it Functional Blindness. This is the tendency to frame problems from your own area of work or study. Dearborn and Simon called this Selective Perception, ref [3], where they noted that managers often focus their attention on areas that they are familiar with - sales executives focussing on sales as a top priority and production executives focussing on production.

In testing this may translate into:

Testers with mainly performance test experience focussing on those areas
Recent customer support experience leading to a preference to operational and configuration aspects
A generalist spreading effort evenly in many areas

Sunk-Cost Fallacy

This is the tendency to factor in previous investments to the framing of the problem, link. A good example is James Bach's Golden Elephant Syndrome, ref [4].

In testing this may translate into:

The latest favourite tool or framework of the execs must be used as there has been so much investment in it.

Over-Confidence

As we've seen above there can be many different ways of framing the problem. It's important to be aware of this. There is a trap that testers can think they've done everything they need - their model/s was the most adequate in this situation.

Here the warning is against complacency - re-evaluate periodically and tell the story against that assessment. It may be that an issue you find during testing affects some of your initial assumptions - the approach might be good, but maybe it could be better.

(It might be that you can't change course/approach even if you wanted to, but that's good information for reporting to the stakeholder - areas for further investigation.)

Whatever your model, it's one model. Is it good enough for now? What does it lack - what product blind spots does it have?

Measurements and Numbers

Decision frames and framing sometimes uses a way of measuring whether the frame is good or useful - or whether alternatives are equal. There is a danger here when numbers and measurements get involved.

In business and everyday life there can be occasions when figures and measurements are presented as absolutes and other times when they're presented are relative figures. They can be misleading in both cases, especially when not used consistently.

Project stakeholders are probably very familiar with looking at project cost and overrun in absolute and relative terms - depending on how they want the information to shine.

So it's very easy for testers to be drawn into the numbers game - and even play it in terms of absolute or relative figures.

"This week we have covered 50% of the product"
"Our bug reports have increased 400% compared to the previous project"
"The number of tests to run is about 60% of the last project"
"5 bug reports have been implemented in this drop"
"All pre-defined tests have been run"

As you can (hopefully) see this is just data - not necessarily information that can be interpreted. So, beware of number usage traps in the problem analysis and formulation - both in those given to you and in those you send out,

Another aspect of problems with numbers and decision framing can be thought of as the certainty effect, ref [6]. This can affect how we frame problems - and even how we should communicate.

Frames and Framing Need Maintenance

Analyze and periodically check that your assumptions are correct. Sometimes the emphasis of the project changes - the problem to solve changes. Is the frame still right or correct? Are the parameters of the problem still the same, are the reference points and ways in which to measure or judge the frame - are they the same - if not, time to re-evaluate.

Working with Frames

What frames do you and your project / organization start with? (Subconcious default)
Are there alternative frames to consider? How many were investigated?
Look at what each frame includes and excludes
What is the best frame fit for the situation / project? (Do all involved agree on the 'good enough' frame?)

References

[1] The Framing of Decisions and the Psychology of Choice (Tversky & Kahneman, Science Vol 211, No. 4481)

[2] Decision Traps: The Ten Barriers to Brilliant Decision-Making and How to Overcome Them (Russo, Schoemaker, Fireside,1990)

[3] Selective Perception: A Note on the Departmental Identifications of Executives (Dearborn, Simon, Sociometry Vol 21, No 2, June 1958)

[4] James Bach "Golden Elephant" Syndrome (Weinberg, Perfect Software: And Other Illusion about Testing, Dorset House, 2008, p. 101)

[5] Calculated Risks: How to Know When Numbers Deceive You (Gigerenzer, Simon and Schuster, 1986)

[6] The Certainty Effect and Insurance

Taylorism and Testing

2011-08-14T15:38:00.001+02:00

" #softwaretesting #testing "

I've had this in draft for a while, but was triggered to "finish" it by reading Martin Jansson's recent posts on working with test technical debt and nemesis of testers.

Taylorism
When I think about Taylorism I'm referring to the application of scientific principles to management that direct division of labour, "process over person" and generally anything that is the result of a time and motion study (however that might be dressed up).

The result of this is categorising "work" into divisible units, estimating the time for each unit and the skills required to do each unit. Once you have these, then plugging them into a gantt chart is a logical management step.

Estimating
Estimating work using some estimation guide is not the problem here. The problem is when that guide becomes the truth - it becomes some type of test-related "constant". It's used as such and, more importantly, interpreted as a constant.

Problems with constants might occur if you discuss with your stakeholder items such as:

Time to write a test plan
Time to analyse a new feature
Time to review a requirement
Time to write/develop a test case
Time to execute a test case
Time to re-test a test case
Time to write a test report

Traps?
Stakeholders don't usually have time to go into all the details of the testing activity, therefore it's important as testers to not let the items that affect any estimation be considered as constants. This is highlighting to the stakeholder that the assessment of the question depends on the specific details of the project, organisation and problem at hand.

So, re-examining the above items might give some additional questions to help, below. (This is just a quick brain-dump and can be expanded a lot)

First questions:

How will the answers be used?
How much flexibility or rigidity is involved in their usage?

Time to write a test plan

Do we need to estimate this time?
What's the purpose of the plan?
Who is it for?
What level of detail needs to be there now?
What level of detail needs to be there in total?
Am I able to do this? What do I need to learn?

Time to analyse a new feature

Do we need to estimate this time?
How much do we know about this feature?

Can we test it in our current lab?
New equipment needed?
New test harnesses needed?

Am I able to do this? What do I need to learn?

Time to review a requirement

Do we need to estimate this time?
Are the requirements of some constant level of detail?
How high-level is the requirement?
Are the requirements an absolute or a guide of the customer's wishes?
How often will/can the requirements be modified?
What approach is the project taking - waterfall or something else?

Time to write/develop a test case

Do we need to estimate this time?
Do we all have the same idea about what a test case means?
Do we mean test ideas in preparation for both scripted and ET aspects of the testing?
Do we need to define everything upfront?
Do we have an ET element in the project?
Even if the project is 'scripted' can I add new tests later?
What new technology do we have to learn?

Time to execute a test case

Do we need to estimate this time?
In what circumstances will the testing be done?

Which tests will be done in early and later stages of development?
Test ideas for first mock-up? Keep or use as a base for later development?

What is the test environment and set-up like?

New aspects for this project / feature?
Interactions between new features?
Do we have a way of iterating through test environment evolution to avoid a big-bang problem?

What are the expectations on the test "case"?
Do we have support for test debugging and localisation in the product?
Can I work with the developers easily (support, pairing)?
What new ideas, terms, activities and skills do we have to learn?

Time to re-test a test case

Do we need to estimate this time?
See test execution questions above.
What has changed in the feature?
What assessment has been done on changes in the system?

Time to write a test report

Do we need to estimate this time?
What level of detail is needed?
Who are the receivers?

Are they statistics oriented? Ie, will there be problems with number counters?

Verbal, formal report, email, other? Combination of them all?

Not all of these questions would be directed at the stakeholder.

Depending on the answers to these questions will raise more questions and take the approach down a different route. So, constants can be dangerous.

Stakeholders and Constants
When stakeholders think in terms of constants then it's very easy for them to think in taylorism terms, think of testing as a non intellectually challenging activity and ultimately think of testing as a problem rather than an opportunity for the project.

Some questions that might arise from taylorism:

Why is testing taking so long?
Why did that fault not get found in testing?
Why can't it be fully tested?

Working against Taylorism in Stakeholders

The tester needs to think and ask questions, both about the product and what they're doing. Passive testers contribute to mis-understanding in stakeholders - the tester is there to help improve the understanding of the product.

The relationship of a tester to a stakeholder changes as the tester adds value to project, product and organisation. So, ask yourself the question if and how you're adding value. It's partly about building your brand, but also it might be about understanding the problems of the stakeholder.

The stakeholder frames a problem and presents that to you in a way which might be different from how the customer intended. A good starting point with some questioning is to think in terms of context-free questioning (see Michael Bolton's transcript from Gause and Weinberg, here).

Build your brand, add value to the organisation and product and ultimately the problem with Taylorism will recede.

References

Reflections on "Done": Regression & Interaction Testing

2011-08-07T03:03:00.000+02:00

" #softwaretesting #testing "

If you haven't read it then go and read Michael Bolton's post and comments regarding the usage of "done", here.

From some of the comments, discussions and opinions elsewhere there is a notion that "done" is the complete package. This was the point of Michael's post. But with each sprint, delivery or increment the product evolves - and so any definition of done will be susceptible to problems with definition.

It's very easy to have a mission that includes "existing features work as before" - but there is also a risk of missing something here….

The product evolves - interactions between features and within the system change. Not all of this can be anticipated in advance.

Interactions
So, however you might define "done" for your delivery there will (inevitably) be an area of unknowns that may* be investigated.

It's not as simple as saying this is covered by the exploratory testing (ET) part of the sprint. The ET may be for the new feature, but there is also an element that could* be devoted to learning about interactions between the feature and existing product features or components, and even understanding how the whole behaves (again**).

Of course, a good first step here is to separate out the ET missions for the new feature and the ET missions for interactions and existing features.***

Regression
Some of this might be covered in some "regression" and characteristics measurement and monitoring. But the problem with "regression" is that it doesn't necessarily have an element of learning about how some existing test suite (especially applicable to automated suites) works with the "new" (evolved) system.

An automated regression suite usually has a notion of "this is a fixed reference point" - this script should**** usually work. But the testing that is missing is usually the evaluation of "if and how" it works in the new system. This evaluation is commonly limited to looking at failures and how they should be fixed.

Some shops wrap this up as a "regression" element into the increment's mission (or even definition of done) - but wrapping things up (implying) in some other activity is exactly the original problem with "Done" - it doesn't necessarily reflect the focus of the problem in front of you at a given time.

Now, being able to reflect "exactly" the problem in front of us is something that can't be done "exactly" - that's why we need testing to help evaluate. So, dimensioning (or estimating) an activity should be wary of this.

Ultimately, a lot of this comes down to good testing - separating the assumptions (and implications) from the problem in front of you and illuminating these assumptions and implications with the stakeholder as early as possible.

Good testing starts early!

* even "should" or "must" depending on the context

** understanding how the system behaves is an iterative task.

*** Note, this is all depending on the specific context - it's not automatic to say "do this for every iteration" - each delivery must be assessed on its needs and usage.

**** "should" is one of the most over-and-misused words within software development!

How is your Map Usage?

2011-08-05T06:50:00.000+02:00

" #softwaretesting #testing #models "

Preamble

On a recent trip abroad we hired a car which had satnav. The party that we joined also had a range of different satnav products - different makers and no doubt different map versions.

On one occasion we had a convoy heading to a destination some 35km away, 3 cars (A, B & C) using their respective satnav. The cars departed for the destination in order ABC but arrived in order BAC each having taken different routes.

First Observation

Now there were a range of different parameters that could affect the route calculation:

Shortest route
Fastest route
Most economical route
Route avoid certain points
Accuracy of map
Plus other factors that are not to do with the map itself - wrong turnings, not driving according to the assumed parameters (eg speed), traffic accidents, etc, etc

After more usage of the satnav I realised some other points...

Second Observation

We tended to notice less en route to different places when using (relying on) the satnav. If we hadn't had it we would probably have noticed landmarks and features in more detail. But now we were in a 'different world' - following someone else's view of the world.

Third Observation

Reliance on the map guidance became less as familiarity of the surroundings increased. We were in the areas for a week and reliance on the map became less over time.

These observations are directly comparable to my software testing experience. The map is just a model - a certain level of detail used for a certain purpose. Change the detail or the purpose and it is a different model.

Also, the use of a map (model) can change over time. On one occasion it might be used as a main guide to get to a destination and after a period of familiarity it is used as a waypoint on course to a destination. Something to help orienteer along the way. I've touched on some of these re-use ideas before when walking in the woods.

Notice the parallels with use and purpose between maps and model usage in software testing.

Some maps are old, some new, some sightseeing, some for walking & hiking. Some use maps as a history - tracking where they have been - they can even use someone else's map and make notes about features they have seen

Points to note with your map (model) usage:

Purpose

Is it a normative or informative reference? Does it describe the contours of the landscape (main claims of the product) in a lot of detail, or is it a hand-sketch of the area in much less detail (key features to look at)?
Is it a record of progress (as in coverage of features)?
Is it a partial plan (X marks the spot - lets look for treasure and other interesting things along the way)?
Is it a mission objective?
Is it a sightseeing guide (testing tour)?
Is it a record of danger (places to avoid or see, marshes and bogs not suitable for walking - bug clusters in the product)?

Representation

One thing to note - maps (models) get old. Sometimes that's ok - some features do not change (ancient monuments, town locations, product features).
Sometimes it's not - you want the latest sightseeing information (tourist attraction) or danger areas (buggy areas of the product).
Ultimately a map (model) is a view of a landscape (or product). There might be a lot of personal views on the map (model) - what is included, to what detail, what is left out or even omitted.
Whether new or old, the use of it should be able to be dynamic - that is, your use of it is the aspect that adds value to the journey (testing session/s).

Caution for Map (Model) use in Software Testing

From the first observation, and the points above, the models (of product or problem space) should always have a known purpose - fitting the right model to the right objective - and should (ideally) be used in a dynamic way.

From the second observation this is a caution to question the assumptions connected with the model. If you're in unfamiliar territory then you may need to rely on the model for guidance, but use it dynamically. Is the information correct, what information can I add? Don't just follow - question (even internally) as you go. Even question about how the terrain would look if the map/model was wrong - to give yourself clues about what to look for.

If you rely too much on the map - whether it's someone else's or your own - then you there is a potential to lose touch with your testing objectivity/eye/mindset - something that is needed for good testing.

Still to come

From the third observation - there are aspects of time perception and the amount of information processing we make (or shortcut) and aspects of focused and stimulated attention - this is an area I'm still researching with potential implications for scripted and exploratory testing (more to come).

So, how is your map reading and usage? How is your model usage?

Carnival of Testers #24

2011-08-03T11:02:00.002+02:00

" #softwaretesting #testing "

July was a month for a wide variety of blog posts, with many tidbits….

Ambiguous Words

Michael Bolton looked at problems with using the word "Done", here.
Pete Houghton on his 2 minutes using Bing Maps and thoughts about consistency and convention.
Good points from Eric Jacobson about what is testable, here.

Experience

There has been some good blogging on the STC. Jeff Lucas gives some views on scripted automation, here.
Joel Montvelisky with some experiences on finding the right testing effort for the product and the organization. Also, here, with some points about thinking before measuring.
Still on metrics, Rob Lambert gave a good example of how using measurements as absolutes will give the wrong conclusion.
Markus Gärtner's thoughts on a Pomodoro approach to testing.
The problem of spam and moderating discussions was highlighted by Rosie Sherry, here.
Are you "doing agile"? Some thought-provoking points from Elisabeth Hendrickson, here, on checking how you're agile implementation looks. (This was re-published in July)
Good learning experiences from Pradeep Soundararajan about rapid software testing in action, here.
Part 3 of Martin Jansson's look at testing debt is worth checking out.
Catherine Powell nearly rants, but then reflects on learning, here.

Other

Lisa Crispin's review of the Clean Coder sounds like an interesting read.
CloudTest Lite got some encouraging words from Scott Barber, here, and Fred Beringer, here.

And finally….

If you haven't noticed already, the STC published the latest edition of the testing planet - well worth a look.

Until the next time...

Fake Learning: Shortcuts Don't Always Exist

2011-07-04T14:18:00.000+02:00

" #softwaretesting #testing "

Excuse the mini-rant...

I seem to have a love-hate relationship with different Q&A forums.

Why?
I like to give input on them, to share my perspective, with the hope that my take, however left-field or mundane, might help the questioner and all those taking part. I don't consider myself the guru - I'm just providing one piece in a thousand piece jigsaw - rather than have the whole picture on the front of the box!

Some people are out there asking questions in a genuinely investigative way, searching, pushing the boundaries of their own experience and forming opinions on what they find. That is fantastic, I love the energy (I won't say passion :) ), the willingness to go on a mental adventure, get out of your own mental comfort zone and see what you find and how you might use it.

For examples of thought-provoking questions look at the questions from Phil, here, or Shmuel, here, or Shrini, here.

There are no right or wrong answers - but every answer might help you learn something new.

Comfort Zone
But, on the other side of the coin there is the search for information without the searching or learning. This is definitely a case of not pushing ones own mental comfort zone.

Examples of staying within ones own mental zone - questions of the form:

What is the (insert one of: "best, most effective") (insert one of: "tool, process, method, etc") for (insert a general non-specific context or situation)?

These types of question want to bypass the learning journey. It's a pay-per-view approach - without the payment!

That may be ok if all you want to do is compile a list (the 21 capital coach journey in one week), but you must ask yourself how that information might be used - whether to get a flavour of something or to claim to "know" about something are two different purposes.

It's sort of the difference between learning by rote and information gathering (with hypothesis formation) -> ooh, scripted learning vs exploratory learning, or testing with your eyes wide open vs eyes wide shut!

Fake Learning?
The essence of these types of questions - for me - is an attempt to shortcut learning - to shortcut the experimentation, digesting the information, hypothesis (ideas) forming and making new discoveries (at least to yourself). Shortcutting this (and purporting to have gone through it) is something I think of as Fake Learning.

Several people highlighted the problem recently:

Myself on twitter, here.
Phil Kirkham on twitter, here.
Michael Bolton on twitter, here.
Rosie Sherry in her blog, here.

Depressed?
No, not really. Am I (and maybe others) going through that "hate" part of the cycle right now. Maybe yes.

But it doesn't stop my own searching and willingness to help others along the way - so I'm still on the lookout for interesting and thought-provoking questions - that doesn't mean I always have the time to respond and get involved myself!

But if I'm asked directly for help, I'll always try and help. It fits in with my testing motto too!

Mini-rant over. :)

Beware of Fake Learning!

Carnival of Testers #23

2011-07-03T04:21:00.002+02:00

" #softwaretesting #testing "

Workshops
The workshop on testing concept continued it's travel around the world.

DEWT1 took place, with a write-ups from Ray Oei, here; Zeger Van Hese, here; Jean-Paul Varwijk, here (interesting first blog), here; Jeroen Rosink, here; Huib Schoots, here.
Later in the month a workshop on testing took place in New Zealand. KWST was born! Good reading from Brian Osman, here, and Aaron Hodder, here.

Podcasts
This month saw 2 notable events from the podcast world.

Trish Khoo announced the launch the testcasters site (with Bruce McLeod), here. They sound like they have a lot of fun with it!
Also, TWIST had it's 50th edition. Michael Larsen writes about that and more, here.

Searching for THE ONE

There are recurring questions that crop up where searches are made for THE definition, guarantee, way or tool. This triggered some interesting perspectives.

Markus Gärtner examined a claim about "full automation" via model based testing, here.
Sometimes testers (and non-testers) get hung-up on the A-word (assurance), not to say anything about the Q-word! James Lyndsay produced a good perspective on Test Assurance as applied to (exploratory) testing, here.
A search for a good, better or best testing tour gave rise to this post from Cem Kaner.

Other

A good follow-up on his testing debt article from Martin Jansson, here.
Vulcan death grip in testing? Adam Brown makes a case for it, here.
A reminder of thinking traps related to test automation was given by Shrini Kulkami, here.
A plethora of recent article links was provided by Matt Heusser, here.
A piece from BJ Rollison examining levels of testing, here, was interesting for the Beizer quotes and how they might not be an adequate model.
An interesting piece from Markus Gärtner on ideas around Programmer-Tester separation, here.
The importance of understanding WHAT you're testing was an experience from Adam Knight, here.
Although the deadline has passed now for the EuroStar lab rats, some experiences from Henrik Emilsson and Martin Jansson were given via the EuroStar blog, here.

Until the next time...

Fault Localisation and Search Satisfaction

2011-06-17T00:53:00.000+02:00

" #softwaretesting #testing "

A recent meet-up with Christin Wiedemann, Oscar Cosmo and Daniel Berggren to talk testing was a very enjoyable and thought-provoking session. It reminded me of a problem that I'd recently being involved with, where I'd observed the problem of Search Satisfaction.

Background
A tricky performance problem existed and it was getting more and more "stakeholder" attention. The problem description (bug report) was quite detailed around the network for the part of the system under test (SUT) in the picture. The bug report described a fairly major problem in the SUT and the investigation had been ongoing for some time when I eventually came into the loop.

The ongoing investigation and fault localisation was made more difficult by the fact that several different problems were being observed in the whole network simultaneously (multiple problems in each network element) and so rooting out the real problem was quite tricky.

Several rounds of extracting different log and trace information to localise the fault and create patch builds had been made. Then there was some form of inertia - little or no progress.

For some reason I was involved in the loop.

Search Satisfaction
When reading "How Doctors Think", ref [1], I encountered the notion of search satisfaction. This is where a diagnosis is made based on the observed data (or really a subset of the complete symptoms), no further testing or searching is done. The search is satisfied and the diagnosis made.

When I read the bug report for the problem above, and all the associated notes, I found lots of evidence pointing to the SUT. There was a lot of information about faults in the SUT and actions that had been taken to correct them and re-test. However, there was something missing - for me. The main problem/issue didn't seem to be related to the faults that had been corrected.

The description I read in the bug report would justify some of the associated faults being corrected (in an effort to localise the main problem) but not really the most serious fault. For that there needed to be more information on the surrounding network elements - I suspected an interaction problem and to understand that I needed information on the behaviour in the rest of the network.

I raised these issues with the people directly involved. They took another look, now with a different perspective and after some further rounds of localisation (now involving a wider search) the issue was found. A fault in another network element which was masked by the interaction between several elements.

So, in this example the faults being worked on were driven partly by the data available and partly by what could be fixed. The actions on those faults were correct. The search for fixes was satisfied by some of the symptoms that could be seen. But did those symptoms explain the severity of the main problem? Sometimes that's an important question - I can observe symptom X, Y & Z, but do they explain the main problem?

Search Satisfaction - How?
Search satisfaction is not just about fulfilling a search and not wanting to continue search. We learn to see patterns, get comfortable with patterns and recognize patterns. This has been demonstrated in different studies.

In one case, ref [2], it was found that Americans and Chinese subjects focus on different aspects of a picture. These habits of perception are thought to be influenced by the environment and society that they live in - people learn what they're used to and that is the default habit.

Another study, ref [3], showed an experiment where kittens had learned to perceive their environments in different ways (by being deprived of horizontal or vertical perception) and then had problems when they needed to use the perception they'd been deprived of.

Another favourite perspective of mine - which I use when thinking about open-ended tests - is the example in this picture from hubble, here. Read the comment. Sometimes if you think you won't see anything, you might not see anything - so sometimes you need to keep looking!

So, whether seeing a pattern in what one observes when testing, those patterns can be something that has been learnt. It fits the observation. But is it the only explanation, or is there a different observation (perspective) that might contradict the hypothesis?

Observation
In the example that I observed there were two factors that kept the search "satisfied" and constrained. These were a fixed focus and constant stakeholder attention.

Too focussed:
De-Focus and Re-Focus techniques can help avoid this problem. Sometimes you have to step back and observe what's going on around the periphery to then make some different sense of the detail. This is sometimes easier said than done. Sometimes it needs a fresh pair of eyes to trigger this.

Stakeholder "attention":
This can actually induce search satisfaction. Someone wants an answer (or diagnosis) and you have an answer staring right at you. This is more difficult to avoid, but a first step to avoiding this is realizing that you might be operating with limited data.

Lessons from Science
Don't stick to confirmatory approaches. Think about disconfirmatory evidence - how could my theory be proved wrong or incomplete? Do I have enough information? Can I think of a way in which it might not be a sufficient (or good enough) theory?

References

How Doctors Think (Groopman, Houghton Mifflin 2007)
Cultural variation in eye movements during scene perception (Chua, Boland and Nisbett, PNAS 2005)
Development of the brain depends on the visual environment (Blakemore and Cooper, Nature 1970)

Carnival of Testers #22

2011-06-03T13:07:00.000+02:00

" #softwaretesting #testing "

Diverse is the word for the writing in May...

Learning & Insights

Pete Walen highlighted parallels between deduction and test investigation (a la House), here.
Some relevant learnings for testers from Ralph van Roosmalen, here.
Albert Gareev made a triple post of his interview with Michael Bolton, good reading here, here and here.
THE happy tester, Sigge Birgisson, wrote about his optimistic and positive approach to testing, here.
Binary disease, and some of the issues it causes testers, was the subject of a post by Rikard Edgren.
Some interesting perspectives on memetics in testing were posted by Peter Haworth-Langford.
Rob Lambert was prolific in May, with this post on sharing information and communicating just one of many examples.
I loved the dashboards and representation ideas in Trish Khoo's post, here!
Christin Wiedemann illustrated that sometimes if it tastes "off" maybe it is "off", here.

New Pens

New writing this month from David Greenlees, with a piece on the value of conferences, both to yourself and your employer.
Claire Moss, started her first blog with an account of her exploratory approach to STAREAST, here.

Conferences

Continuing with STAREAST, here is a good overview of Lisa Crispin's experience and learnings on the STC.
WAT2 got some good coverage from Markus Gärtner, here, Alan Page, here, and Marlena Compton, here.
Markus Gärtner also provided some insights from the Problem Solving Leadership course, here.

This is just a small sample of some of the good writing this past month.

Until the next time...

Problem Analysis - Mind Maps & Thinking

2011-05-17T06:31:00.000+02:00

Credit to Peter Haworth-Langford, Christin Wiedemann and Oscar Cosmo, with whom I was chatting back in December...

The topic of mind maps was discussed (in various usages) and I realised that maybe I had a usage that I need to write about and digest...

I frequently get asked to do a root cause analysis type of activity. Usually it's connected with faults reported against a product, but other times it might be to analyse a situation or process. In all cases I usually have a main question in my head, "what is the real problem?"

Below is an example of using these approaches for a root cause analysis of a fault. The analysis is used as a learning opportunity - to see if there is anything that needs to be changed or improved, including project and team structures and aspects of how they work.

Example
A typical root cause analysis (RCA) in my shop might look like the following. This is one example and there are many other variants.

The example "presumes" there was some missing testing (which might reasonably have been expected to happen), but it could just as easily be missing design analysis, design coding error or any of a range of other issues.

I have two main techniques that I use to help reveal information - and so help me on my way to "the real problem". Note, I say this as though there is only one, but I'm well aware there are usually many different competing aspects.

The first is the 5-whys and the second is what I think of as the "are your lights on?" approach and I use them in distinct ways:

5-whys - extracting raw data
Lights-on - attaching meaning to the analysis

The 5-Whys
This is the forensic search for raw data - collecting the different aspects of the problem - what the problem was, its solution, aspects related to team analysis, system design, aspects related to test design and execution, aspects of team dynamics, changes in timing and priorities that might any of these parts.

Actually, I don't restrict the analysis to "5" - it can be 2, 3, or 8 - as many as are needed to get a "good enough" picture or level of granularity (part of this is an understanding of how deep we need to dig). Extending the above example, it might become:

Lights-On
This approach is heavily based on the Gause & Weinberg book, "Are your lights on?", which I recommended highly. This is an approach that helps answer:

What is the problem?
To whom is it a problem?
Whose problem is it?
Does the problem need solving?

Usually, by working through these aspects of the problem it is possible to determine the extent of the issue, if it is an issue that should be addressed, if this is something that can be done now and by who.

Sometimes the problem appears due to a change in project priority or timing, something doesn't get completed and, crucially, the customer is not aware that it shouldn't use feature X in this first drop, or that it has certain limitations.

Communication (or the absence of it) is, occasionally, a root cause in itself!

A Third Way
I said I had two main ways, well recently I've realised that I'm using a third way - I say recently, as I've really only attached (or discovered) the terminology for it: Framing.

Framing
Framing is heavily based around the work of Tversky & Kahneman, with some application by Russo.

This is the filter that is used to look at the problem. In the above example, I might look at the issue from the project perspective: what priorities existed on the project (stakeholder or customer), were there aspects outside the control or remit of the project, did any of the project aspects change and with what timing?

There might be team frames: how did the internal and external communication to and from the team work; was there a project change that didn't filter into the team; information availability, assumptions and other limitations.

Finally, there might be aspects that work on project, team or individual level: risk averse or risk taking attitudes; attitudes of uncertainty, lack of support or overconfidence; or a combination of many factors.

So, applying some framing aspects to this example might give:

The framing of the problem - and using multiple frames - helps to put the issues into perspective, the situational context - what did it mean at this particular point in time, in this particular project, in this particular team set-up, for this particular customer, etc, etc.

Full Circle
The framing aspect now fits quite well into my RCA approach, or any problem analysis approach:

Raw data is gathered: 5-whys is one approach
Situational context: Framing, ideally using a number of frames.
Meaning and decision: Lights-on

Conclusion
Mind maps help in a number of aspects here: visualisation and recording of thinking and exploration.

However, the big aspect is the thinking - using a range of tools to explore the problem, not limiting analysis to the problem artifact (a team didn't do xyz...) but adding in the situational aspects and weighing up the problem with the surrounding information to get to a better understanding of it (framing). Then the information can be used to help understand if it really is a problem that needs solving and by who (Lights-On).

References

Gause & Weinberg, "Are Your Lights On?: How to Figure Out What the Problem Really Is" (Dorset House, 1990)
Kahneman & Tversky, "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981)
Russo, "Decision Traps: The Ten Barriers to Decision-Making and How to Overcome Them" (Fireside, 1990)
5 Whys, (wikipedia)

Carnival of Testers #21

2011-05-04T00:01:00.000+02:00

" #softwaretesting #testing "

"April april, din dumma sil

Jag kan lura dig vart jag vill!"
(Swedish saying when you've been "April fooled")

Well, no April fools jokes here - just proper writing and blogging about testing..

Testers Meeting
April saw the second SWET occurrence and the ninth LEWT. Many SWET attendees wrote their reflections. More for LEWT please? Elsewhere, testers were at meetups, at conferences and testing online.

A first view of STPCon from Pete Walen came here.
Rob Lambert & the STC posted a selection of videos for the lightning talks from the Oxford meetup, here.
European Weekend Testing has been hosted on a few occasions recently by Eusebiu Blindu. He writes about it here.
SWET views from Torbjörn Ryber, Ola Hyltén, Rikard Edgren, Martin Jansson, Henrik Emilsson, Sigurdur Birgisson, Christin Wiedemann (in Swedish) and Simon Morley.
LEWT09 got an overview from Markus Deibel.

Reflections

Markus Gärtner gave a comprehensive write-up of his Management 3.0 course.
A good reflection on some options for visualizing testing was posted by Gojko Adzic.
A thought experiment on factory vs context driven testing was started by Rikard Edgren, here. Some interesting discussion in the comments too!
Writing tips for software testers was the subject of Anne-Marie Charrett's post, here.

Recognition

Entaggle has had a few posts already. Here is the first (of many) from Elisabeth Hendrickson.
Changes to Joe Strazzere's "people in testing" page were posted, here.

Newbies

John Stevenson looked at a relevant question about mentoring new testers, how he did it and some initial ideas, here.
Henrik Andersson, who hasn't blogged before (I think), made an interesting entry, here, with a piece on the need for diversity.

Until the next time ...

SWET2 - Serious Testing Talk by Serious Testers

2011-04-29T00:54:00.001+02:00

" #softwaretesting #testing #swet2 "

Place: Hönö Hotell
Date: 9-10 April 2011
Twitter hashtag: #SWET2 (an odd one with #SVET2)

Antendees (seating order): Torbjörn Ryber, Martin Jansson, Simon Morley, Henrik Emilsson, Sigge Birgisson, Ola Hyltén, Steve Öberg, Johan Jonasson, Rikard Edgren, Azin Bergman, Christin Wiedemann, Henrik Andersson, Robert Bergqvist, Saam Koororian, Fredrik Scheja

SWET - Swedish Workshop on Exploratory Testing

This was the second installment of a peer conference on exploratory testing held by Swedish testers, organised by Rikard, Henrik E and Martin. Great job!

The main focus this time was the analysis, planning and status communication aspects related to exploratory testing. I thought this was a great idea - an area that sometimes doesn't get the discussion time and an area ripe for discussion.

Presentations

Johan, Tobbe, Fredrik & Saam did a fine job with their presentations - I won't go into the details as they've been described in the references below.

Johan had the marathon session - about 15-20mins of presentation followed by nearly 4 hours of discussion. It's that discussion that really searches into the experience report and picks up on different aspects and looks at them closely.

In this session there was an exploration of aspects of new testers versus experienced testers in exploratory testing. This got a lot of exploration and I think many drew or highlighted interesting lessons here. Some notes I made (this is a mixture of my own and other's thoughts and observations):

Attitude (from project or team lead) towards testers/teams is decisive.
Attitude of testers/teams is decisive (in good testing).
Importance of the team lead that "protects" testers from the project and lets them test.
But how to handle project leaders where that shield doesn't exist?
Comfort zone warning signs in testers: Ways-of-working, templates, same-same attitude and identity.
Can be a danger that testers can use test reporting as an advert for testing, i.e. a danger that we're missing the point of testing. (This was a good point from Martin!)
Dialogue between tester/team and the project lead - no surprises.
Test planning: Trust between tester/team and project leader is important (if not vital).
Actively build up this trust.
It's the planning that is important and not the plan!
Domain expert vs Right attitude?

Tobbe was next up with an experience report on the role of a tester that goes outside the traditional and incorporates aspects of project management. Some observations:

Is there a danger in the combined tester and project leader role. Danger for double-personality complex (amongst others)?
How was the conflict between lots of testing issues and reporting to a stakeholder on the project handled?
Did the double-role affect the content and detail of the reporting?

Fredrik was third with a report on using some aspects of TMap as a starting point for their work. The story wasn't really about TMap but more how they'd used it as a starting point. Some of my notes/questions:

Test analysis & planning has a small timebox - is this optimal after a few iterations, some other basis or some pre-requisite knowledge before application?
How do you examine and question your working methods and approach for the purpose of improvement?
Having a "high quality expectation" - what does this mean and imply? (what does it leave out?)

Saam was the fourth and final presentation. Unfortunately, we only managed just over an hour of discussion after the presentation. Some of my notes and observations:

Working with distributed sites - there are different cultural aspects here. How did they impact the reporting?
The idea of "1 requirement per test" - how is that counteracted?
"Confidence levels" in reporting is a good starting point - but is there a need to normalize or baseline them for stakeholders? Education for stakeholders?
How are faults in the coverage map handled?
What was the stakeholder reaction to low coverage on parts of the map?
Map: confidence vs security vs risk vs tool for communication (and not communicated).
Map: Needs to be dynamic.
Map: How are assumptions and limitations displayed? Do they need to be?
Map: Does the map change with different assumptions and limitations?
Use the map to solve the team problems and not as a method of communication.

My Lessons

I practised thinking about how I would behave and react in the situations presented in the experience reports.

The observations and questions (above) are really some of the questions I might pose in similar situations - so I'm learning from the presented experience and testing my ideas out. And that's good practice - valuable.

It was fantastic being in a group of intelligent testers - some from SWET1 and some new aquantancies. All had many different perspectives on issues that many of us face on a daily basis. Great to compare notes and discuss topics of the day. Talking of which...

Best Practices

As usual there were plenty of late-night discussions. The topic of best practices arose. As I remember, participants in the discussion were Rikard, Ola, Henrik A and myself (maybe I missed someone?)

The discussion was on the phrase "best practice" - notice it wasn't the topic of "best practices" but rather the phrase.

Rikard said that he thought it was curious how usage of the words can induce such strong reactions. There is a state in which someone can think of an idea or topic as their best so far, and consider something as "best" [to them, so far].

Henrik talked about how problematical it was to use those two words together.

My view was that it was incomplete - giving something a label of "bp" is really avoiding dialogue. It's an incomplete phrase as there's no time or any other contextual aspect - therefore using the term is lazy and imprecise.

Lightning Talks

This was a great initiative! Five minutes open floor - talk for five minutes or less if you want to allow questions. All those not presenting got up and gave an energetic five minutes.

Other references:

Passionate is a word that sometimes gets overused when describing testers ("... I have a proven track record ... yada yada ... oh and I'm a passionate tester also...") - but in this grouping passion fits. Engaged discussion, relentless searching and ultimately improving ones thinking and reasoning. All in their spare time. Now that's passion!

Cool! Roll on SWET3...

My Slides from Iqnite Nordic 2010

2011-04-26T06:00:00.005+02:00

" #softwaretesting #testing "

Long overdue, but I finally got around to uploading the slides from my presentation at Iqnite Nordic 2010.

As ever, the slides only give part of the story - you miss out on the presentation ramble, posturing, double-takes and gesturing - but maybe that's a good thing. It's a little academic in parts and there are some parts I've modified when giving this since, but there is still a good message in there.

Anyway, enjoy:

Test reporting to non-testers 2010

Oh, and of course feedback is welcome!

The Certainty Effect and Insurance

2011-04-15T01:08:00.001+02:00

" #softwaretesting #testing "

At #swet2 I gave a lightning talk on an aspect of Framing (more on this in future posts) and thought I'd jot down some of the details.

On the way to the peer conference I'd read Tversky and Kahneman's "The Framing of Decisions and the Psychology of Choice" (Science v211, 1981), and was struck by the description of what they described as the certainty effect. Or really the potential application in testing.

The Certainty Effect
This was labelled by Tversky and Kahneman (in a 1979 article in Ecometrica on Prospect Theory) on a paradox observed by French economist Maurice Allais in 1953. Expected utility theory predicts that a choice is made dependent on it's expected probability. The example from the 1979 article is:

Problem 1:

Choose between

Choice A

A return of 2500 with a probability of .33

A return of 2400 with a probability of .66

A return of 0 with a probability of .01

Choice B

A certain (guaranteed) return of 2400

In a sample of 72 respondents 18% chose A and 82% chose B.

This is in line with expected utility theory. It's also known as a risk averse approach. Now in the second problem the certainty was removed:

Problem 2:

Choose between

Choice C

A return of 2500 with a probability of .33

A return of 0 with a probability of .67

Choice D

A return of 2400 with a probability of .34

A return of 0 with a probability of .66

Now in this case, of the 72 respondents 83% chose C and 17% chose D. This is not in line with expected utility theory and is what is labelled as the certainty effect. That is, there was a disproportionate weighting on an outcome when it was certain - as this weighting wasn't reflected when the outcome was not certain.

This is related to how the question is asked (framed).

Insurance
A further example of this type of framing affecting choice can be seen in attitudes to risk - typically when looking at insurance there is a difference in attitudes that is displayed between risk reduction and risk elimination, for example consider:

A vaccine that is effective in half of cases.

A vaccine that is fully effective in 1 of 2 cases.

Then there is a disproportionate preference for the second formulation.

Applied to Testing?
By default, stakeholders want certainty. They tend to want risk elimination rather than risk reduction. They link this to cost (just like in different levels of insurance) and think that by working with cost they will get closer to certainty.

This is a problem for testers - or really how they are communicating with stakeholders.

Testers can't work in terms of certainty (without very tight restrictions and lots and lots of explanations and assumptions). Therefore, given the two possibilities of talking about risk elimination and risk reduction testers should talk in terms of risk reduction.

Additionally, the certainty effect tells us that typical decisions and choices can be skewed (disproportionately) when the risk or probability moves away from certainty (guarantee).

When handling the message and understanding expectations towards stakeholders consider:

Be consistent - never talk in terms of (or give the impression that you can deliver) certainty.
Be aware that when something is not certain then attitudes to risk and decision choices don't always follow expected weighting of probabilities.

Certainty, insurance and talking to stakeholders - it's not always logical.

Scripts: Scripted Tester or Scripted Executor?

2011-04-08T15:01:00.000+02:00

" #softwaretesting #testing "

I had a brief and interesting twitter exchange with Michael Bolton last week:

<p>&amp;amp;amp;lt;p&amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;lt;p&amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; If you can't see this then your browser doesn't understand IFRAME. But you can look at the link &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;a href="http://www.bettween.com/yorkyabroad/michaelbolton/Mar-30-2011/Mar-30-2011/desc"&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;here&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/a&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;.&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;br /&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/p&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;lt;/p&amp;amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;lt;/p&amp;amp;amp;gt;</p>

My Frame
First I'll give the background of where I'm coming from when I think of "scripted testing".

I started thinking about the checking vs testing distinction a while back (here, here and here). If you read them (rough and ready as they were) you'll see that my ideas about checks are wrapped into testing - i.e. you can't use checks without testing - testing being the information gathering exercise where you use the information gathered to help determine which other type of information to gather - that (to me) is one of the aspects of "good testing".

Note on Earlier Posts on Testing vs Checking
Those posts were a little rough - peer review would definitely tighten them up.It's actually interesting looking back on these posts (for me) as I can see areas where I've developed and refined some of my thinking. But in essence the check is an artifact and the test is the activity

So it was in that frame that I then triggered the question to Michael above.

Testing?

Am I turning scripts into tests? Maybe.

As Michael pointed out (which was quite a helpful clarification!) - I'm using them as an input - and as I referred to in the posts - I can't use a check without thinking about it's use beforehand and it's results afterwards - so in that sense it's very much a part of my testing.

Who Cares?
Some people - but definitely not everybody!

I've noticed a tendency to dismiss scripts or to detach them from testing. Well, as Michael also helpfully pointed out - there are aspects of scripting in most testing (whether it's a mission, goal or hunch) and that most good testing can't be constrained by a script.

And Finally?
For me:

Scripted Testing = Having a script wrapped up in testing, an input into the information gathering exercise.
Scripted Execution = The script on it's own. The activity is constrained by the script. Mmm, what can you do with that on it's own? Not much.

So, have you noticed when you practised scripted testing in the past?
Have you noticed when you've been constrained by the script? Did you let the script constrain you?

Carnival of Testers #20

2011-04-01T01:12:00.000+02:00

" #softwaretesting #testing "

"Beware the ides of March."

(Shakespeare)

Lots of fine writing again this month.

Enjoy...

Zeger Van Hese's post about James Randi and software skeptics. How apt!
Markus Gärtner's post against thought control, here.
Yet another good book review from Michael Larsen, here.
Marlena Compton's post on testing, testers, congruence and Lady Gaga (not necessarily in that order!)

Entaggle
I thought about writing a post about Entaggle, how I'd discovered it and my experience but I was beat to it in different ways....

Brian Osman was the first the make a post alerting folks to its existence.
Darren McMillan made quite an in-depth (3 part) analysis of the site, some business case and usability aspects.
Then I loved Phil Kirkham's gaming of the site. Check it out, here.
Some early experience via weekend testing was given, here, by Mohinder Khosla.

Value

Interesting post from Brad Swanson on the 7 deadly sins of agile testing, here.
The value of shortened feedback loops was highlighted nicely by Lisa Crispin with the aid of her donkeys.
A well-worked example of the value (or otherwise) of quantative reporting was written by Del Dewar, here.
Testers add value to projects in many ways - sometimes by the things they find. James Bach and Michael Bolton looked into some of these findings, here and here.
Good writing from Aaron Hodder on systems testing.

Thoughts

Tony Bruce wrote about his latest meetup organisation efforts - this time a conference! Looking very good! Take a look, here.
I enjoyed Eusebio Blindu's post on principles for testers, here.
Good thoughts on change management from Luísa Baldaia, here.

STC Nottingham Meetup
Two very good write-ups, both worth a look.

Adam Brown wrote a round-up with some videos, here.
Mohinder Khosla made another round-up with a different video, here.

"What's good about March? Well for one thing it keeps February and April Apart."

(Walt Kelly)

Until the next time...

SWET1 & Prezi

2011-03-30T11:27:00.000+02:00

" #softwaretesting #testing #SWET1 #SWET2 "

It's just over a week now until the next SWET (Swedish Workshop on Exploratory Testing) peer conference in Gothenburg.

It looks like it will be a good grouping of people, abstracts, discussion and lightning talks. I'm really looking forward to it.

I thought I'd experiment with Prezi to do an experience report from it. To get a feel of the tool I took SWET#1 as a starting point. Of course, it's a presentation - so you miss a deal without the "presentation part" - and it's quite basic, but captures some of the salient points.

My Prezi re-cap from SWET#1 is here.

Look out for the #SWET2 hashtag in 9-10 days time, with reports afterwards.

Feedback welcome!

Automation: Oh What A Lovely Burden!

2011-03-21T00:43:00.001+01:00

" #softwaretesting #testing "

Do you have a suite of automated test cases? Have you looked at them lately? Do they seem to be growing out of control? Have they needed some update? Does it seem that you occasionally see a problem and think, 'why didn't the test suite catch that?'

If so, then maybe you have a 'lovely burden'.

One of the things with automated suites is that they are not guaranteed to maintain themselves.... Another is that they do not always tell the tester (or stakeholder) exactly what they're doing (and not doing). The information (results) that they give can be sometimes interpreted for something more than what they actually are.

Automated test suites can sometimes give very good information about changes you've made in the system, a lot of times they give very good feedback, sometimes they catch a catastrophic change.

However, they can sometimes lull 'people in software development' into a false sense of security. Wait! The test suites are not evil, as such, so how can that be?

Well, in addition to automated test suites not maintaining themselves and not guaranteeing a lot of things - they are combined with people (whether testers, project, line or other stakeholders) into an idea of a 'holy suite'.

Why are they 'holy', untouchable and must be maintained (as though we form museums and living exhibits of test suites and frameworks)? Well, part of it is the "Golden elephant" problem (James Bach in Weinberg's "Perfect Software and other illusions about testing"). Another part of it is that people (testers, developers and stakeholders) can become detached from what the test suites are doing - something that has been around for a long time, might be 'left alone' until it breaks.

Oops!

Sometimes test suites are not maintained or refactored for several reasons. It may be a judgement call, sometimes it's not possible to easily see where the point of diminishing returns is reached, sometimes vanity (yes, we didn't see that they had an 'end-of-life' 5 years ago, but even so, I don't want to look like I couldn't plan 5 years ahead....) Projects usually have difficulty seeing clearly to the end of the project (at the beginning), so why should it be any different with any artifacts that are produced along the way (like test suites).

If I were not aware of a lot of the above problems I (mr stakeholder) might say that we need to plan better... But, as testers interested in contributing to working software products we should help contribute to the better understanding and use of automated test suites.

How?

Look at test suites regularly (or at least more than never) for:

Relevance
Signs for reaching the point of diminishing returns
The Test Suite "Frame"

Relevance

Is the test suite doing what it needs to do? Are there redundant test cases/scripts? Possibly - do you know where or which ones?

Are there cases (scripts) that never (or hardly ever fail)? Are there scripts that fail when there are always others that fail? This might show a pattern in (1) the system architecture - weak links are highlighted - this is good, but how do you react to it? ; (2) the test suite and the way it is configured - different tests funnel through the same route (is this intentional?)

The test suite is just one view of the software - maybe it's a real view or an artificial view (due to behaviour changes). Which is it?

Is it static - same data and behaviour model - or is it dynamic? If it's not dynamic do you (someone) inject dynamism in some way (e.g. change cases in and out, rotate cases used, ordering, data fuzzing, etc..) Do you have any refresh or re-evaluation routines for the suite?

Point of diminishing return

Think about the current cost of maintaining the test suite.

How many backlog items are there? Do the backlog items that 'need' implementing grow at a greater rate than can be supported with the current budget (time or people), is the architecture reaching it's viable limit, do you know or have thought about what the viable limit is?

Who's the test suite 'product owner', and how are the decisions about what goes in made?

It's important to understand what the automation suite is costing you now - this is an ongoing cost-benefit analysis - which is probably not done in a very transparent way. Not only should the current costs of maintenance be balanced against the benefits that the suite gives, but also more subtle items.

These more subtle items include the cost of the assumptions made about the suite - from the stakeholder perspective. How many decisions are based on inaccurate or incomplete information about what the test suite is giving? This is an area that is rarely appreciated, never mind understood or researched. Ooops!

The Test Suite Frame

Thinking about the test suite has several frames (models or filters in which people see the problem and interpret the information connected with it.) Some of these 'angles' might be:

1. What's the intention with the automated suite? Expectations?

Is it the same intention that the stakeholder has? If not, do they realize this? Do the stakeholders think it is all-singing-and-dancing - and if so, how do you bridge that expectation?
What about new stakeholders that 'inherit' a legacy suite? How do they get to know all the intricacies of the automated suites? They probably don't ask for them, so how does the tester communicate that?

2. Are there gaps to be filled? Planned or known about? (Maintenance plans)

3. The test suite will only give a limited view - do you actively counter this in some way?

4. Risks associated with the test suite - in terms of what it says and doesn't say? (How do you translate results into information further downstream?)

What assumptions are built into the test suite?
Happy path only?

These are just some of the most obvious frames.

And finally...

Ask yourself some basic questions (and just because they are basic doesn't mean they are easy to answer):

What assumptions are built into the test suite, what does it tell you, what doesn't it tell you, what expectations exist on it and how they are matched, or mitigated, how much reliance is placed on the suite, what risks exist with it and how they are monitored (evaluated)?

If you don't have the answer (or a view on these questions) then you have a potential burden.

You might think 'oh what a lovely burden' or you might think 'I'm a tester, get me out of here', or alternatively you might start thinking about which type of questions that need tackling now (soon) to ensure that the stakeholders are getting the information they need - and, importantly, understand the information that they are not getting. Then you/they can start wondering how much it will cost to get the extra (other) information and whether it's worth it.

But, ultimately you'll be working with automation in a responsible way.

Yes, sometimes it can be a 'lovely burden'...

Did you understand the question?

2011-03-17T11:06:00.000+01:00

" #softwaretesting #testing #cognition "

I just took a short quiz on "Science fiction vs science fact" (here it is, (link), go and take it - it'll take 5 mins).

Well, I stank!

But long before I got to the end of the quiz I realised that I wasn't sure about the intentions of the quizmaster - were they phrasing the questions ambiguously (maybe to trap or fool people), or was the subject matter naturally close to the edge of plausibility?

Anyway, it seemed like I was both unsure of how to evaluate the questions but also how to evaluate the questioner - so really I didn't understand the question (it's context you could say) and whether it was important to deliberate long over the questions. Of course, it was just a bit of fun (trivia) so I plowed on, but I was aware of all these questions and potential thinking traps I was falling into...

Availability and anchoring biases - Ah, I heard of this in the news recently - or did I? Then connecting that with another question... Thinking something sounds plausible and then not wanting to move too far away from that opinion.

Where there were a range of topics being discussed then it's easy to fall for the recency effect so you don't dwell on the question and realize you were tricked.

There is, of course, a whole topic on taking exams - including not dwelling too long to save time to revisit the question - but that's another story... If you follow the wikipedia link then you'll probably be able to find examples of lots of different biases in the way you take the test (or answer the question) - one of them being "reading too much into the data bias..." (almost the self-serving bias.)

Testing?

Yes, I immediately started thinking along a couple of testing-related lines.

Did I understand the question/requirement?
Did I understand the context behind the question/requirement?

Having the stakeholder on-hand is always very useful to clarify and clear out any misunderstandings (by either stakeholder or yourself). Sometimes that's not possible - as in the case of the above multiple-choice test - but usually the things that matter in testing will have a stakeholder will be available at some point in time.

If you have a stakeholder (or proxy) available then you can follow-up with a whole range of questions to get to the bottom of the problem.

A very important aspect is your own frame - what's your attitude to the problem, but also to understand that how the information is presented (or by whom) can affect your response.

For example, if you're not on amicable terms with the stakeholder you might adopt an aggressive questioning attitude and not be receptive to the information to be able to react/respond with useful follow-up questions.

Or: You're feeling very tired (or not as alert as usual) and so you miss some implication in the question (requirement) - and act on the first layer of information: Yes, we can send a man to Mars because we can build a rocket and life-support system (but how much has his/her muscles deteriorated by the time they return to Earth - and so how much recovery time is needed, permanent damage(?) etc, etc..)

Can you guarantee that this correction package will work?

Yes, I've heard that question in the past. Working out where to start tackling that question is a whole different post - but really there is a whole different bunch of questions that the questioner/stakeholder has and he/she expresses the "simple" (compressed) question to me - but really I need to get behind the question and understand what their "real" problem is - one way is to use Gause & Weinberg's "context-free questions" from "Exploring Requirements" (here's a transcription from Michael Bolton). Another way is to use some of the techniques from "Are your lights on?" (Gause & Weinberg again.)

Framing

Most of this (for me) boils down to framing and how that influences both our problem analysis, information intake, problem exploration and ultimately decision making. We all have it - mostly without realising the affect it plays. But the important aspect is to (try to) be aware of it and some of the problems that it can cause - then we have a better chance of answering the question (requirement) in the real spirit that it was asked!

By the way, it did occur to me that this could be misconstrued as another "exam-bashing" link - but that's not the intention :) If that thought occurred to you, then that maybe says something about your frame.

Are you aware of your own frames?

Oh, this was my first transcription from 750words - thanks to Alan Page for tweeting about his use - I'm brain-dumping regularly now!

Links:

Quiz:

http://www.bbc.co.uk/news/science-environment-12758575

Context-free questions:

http://www.developsense.com/blog/2010/11/context-free-questions-for-testing/

Framing:

http://en.wikipedia.org/wiki/Framing_(social_sciences)
http://en.wikipedia.org/wiki/Framing_effect_(psychology)

Carnival of Testers #19

2011-03-03T01:14:00.000+01:00

" #softwaretesting #testing "

“February is a suitable month for dying.”

(Anna Quindlen)

Well, no, I disagree - not with what was on offer to read in the past month, with blog posts from four continents...

Visual

I enjoyed the way that Ralph van Roosmalen illustrated the difference between iterative and incremental development, here.

A nice picture series from Mike Brown and utest.

A post from Rob Lambert where he points to a video and draws some interesting parallels to communication for testers to non-testers/managers.

Mind Maps

I got a little worried when I saw the title of Darren McMillan's post 'Mind Mapping 101' - thinking of George Orwell's room 101 - but luckily it wasn't scary and a very readable account of how he uses it in his testing. :)

Using mind maps as part of reporting was discussed by Albert Gareev, here.

Aaron Hodder uses mind maps as a piece in the puzzle to create a test approach, here.

Confer-ing

A very comprehensive account (as usual!) of Belgium Testing Days 2011 from Nathalie de Vries.

A good overview of the Bug DeBug conference in Chennai was posted by Dhanasekar S.

Pete Walen wrote about the possibility to get involved in the emrging topics track fro CAST2011. If you're going, and fancy a punt, then check out this post.

On the topic of CAS2011 you may have read James Bach's recent post about the nature of context-driven testing. If not then here's the link.

Testers working together

There is no best tester and it's almost wrong to think in those terms, was the topic of a post by Martin Jansson.

Some good advice from Esther Derby on team construction - the 0th trap.

Some clever wording in a job description and the thinking behind it was a two-part posting from Thomas Ponnet, with the second part here.

The importance of the thinking and collaboration, and not the tools, in ATDD is highlighted in this post from Elisabeth Hendrickson.

Miscellany

The super bowl and bug hunting made an unlikely combination for Ben Simo's write-up, here.

Dorothy Graham responded to a suggestion that certification is evil, with some background on the original thinking behind one of the certifications. An interesting read in three parts, finishing here.

The word 'kakonomics' graced the testing blogosphere, courtesy of James Christie. I like the word and must resist over-using it...

Have you read 'Perfect Software..'? Some are re-reading it. Here's what KC got out of re-reading it.

Good analysis of a challenge and his response, with an embedded challenge, from Peter Haworth-Langford, here.

A parable from Pradeep Soundararajan on the value a tester can add to the product - or in this case, costs that they can save.

Tim Western asked a question challenge, and then posted his analysis and thoughts around the answers, here.

A reminder from Pete Houghton about sticking to the happy path - even if you don't realise that's what you're doing.

There was a round-up of different aspects of bias that has been written about by testers, posted by Del Dewar. Good read.

“If February give much snow

A fine summer it doth foreshow”

(Proverb)

Until the next time...

Am I a Context-Driven Tester?

2011-02-28T19:18:00.000+01:00

" #softwaretesting #testing "

I'm writing this as a brain-dump that happened during a lunchtime walk...

James Bach made a very interesting post this weekend, here, and there has been a deal of discussion about it - both on blog and off.

Bundled into the comments religion started popping up - religion is a powerful metaphor - it distils some of the intensity around the distinctions. That kick-started my right-hemispere into overdrive...

Declaration of intent?
I've never come out and said I'm a context-driven tester - I've never felt the need - I don't (or try not to) preach (I had enough of that at school), and I'm not a missionary for the church of xxxx testing.

So, what do I do?
Well, I try to do my testing job in the best way I can, with the learnings and leaning that I have. If you've looked at some of my reading list, here, you'll understand that only a part of my learning is geared towards "traditional software development". Yes, the human aspect is very important - understanding how behaviour, bias, thinking traps, thought processes and group dynamics affect the product under development - starting day 1.

I conciously apply elements of systems thinking to the problems I work with (it's very evident when I work with someone from the analytic or factory school) - but I must treat the problem on its merits, how else can I give it my best input.

Does this make me a CDT?
Maybe, but not on its own.

But even when I work with people with strong analytic/factory tendencies my aim is to leave them with the impression of the value my input is bringing.

Does this make me a missionary/franchiser?
Maybe. But in a "nice machiavellian" way.

But, at the end of the day, I'm in it for me - I'm responsible for my learning - and it's that approach that I hope comes across to people I work with. Yes, I look at problems differently from some. Sometimes they want to look at the problems "my way". Then I just think, "result"!

Either way, context is very important to the problems I deal with. My toolbox is forever growing - as their is no best toolbox - and a big part of my work is geared towards tester-to-non-tester communication. So,

CDT? Maybe.
CDT atheist? Maybe.
Good tester? Hopefully.
CDT community wannabe? Yes.
Big toolbox? Definitely!

Have you thought about testing from the school, religion, cult or community perspective?

My Conference Thinking 2011

2011-02-27T16:24:00.000+01:00

" #softwaretesting #agiletd #esconfs #iqnite "

My conference radar so far...

Agile Testing Days 2011

My submission to the conference was accepted, which I'm very excited about. It's an experience report using exploratory testing, with semi-scripted input, and bridges a gap from traditional to more agile (or exploratory) practices. The conference itself looks very interesting and I'm looking forward to CONFER-ing a lot!

EuroStar 2011

Unfortunately I missed the deadline for this conference. I didn't want to re-use a presentation that I've done elsewhere (or will do elsewhere) and had recently being formulating two new presentations - but I didn't have the finishing touches ready :( I took the good advice of Nathalie and Rikard on submissions (here), and didn't want to rush anything. They'll be presented somewhere, sometime. I don't know if I'll make it there as a visitor - need to juggle the bugdet for confs and training this year - but who knows...

Internal

There's an internal conference in Linköping (Sweden) that I'm waiting to see if my proposal has been accepted. It's one that I was running a lot internally in the autumn and is an experience report (using 3 different report examples) to show how numbers can fool the report reader. There are elements to do with cognitive bias in there and it's one I enjoy giving. I'm currently re-vamping it - maybe putting in to prezi form - to put it online at a later date.

Next Generation Testing

I saw a tweet from Darren McMillan that he's going to present at this Unicom (not unicorn!) conference. Good for him! I was invited to present there in Autumn 2009, but wasn't able to make it. Hopefully, I'll be invited back some day - as I know there's some clever peeps that usually attend!

Iqnite

I enjoyed presenting at Iqnite Nordic 2010 - they were very well organised, with interesting regular and keynote speakers. Just waiting to see if there will be a Nordic 2011...

CAST 2011

By what I've read for this conference, this year, it's going to be a don't-miss event for anyone who approaches their work in a context-driven way. For me - it falls bang in the middle of holiday season and we've already committed to a bunch of travelling. But, I hardly ever say never - so I'm still keeping my fingers crossed!

Swedish Workshop on Exploratory Testing #2 (SWET2)

This peer conference is happening in April and I'm really looking forward to this it. I know there'll be a lot of good discussions with clever and eager testers. Just polishing off my submission for this one.

So, that's how it looks for me so far....

Aspects of Motivation in Software Testing

2011-02-20T17:20:00.000+01:00

" #softwaretesting #testing "

I'm currently looking at some areas of why some managers and stakeholders are so interested in test case counting, amongst other things. At the same time I had the animation (below) pointed out to me (it's based on a Dan Pink talk - you can find him on TED).

If you replace the motivation examples (in the animation) with a drive towards test case counting then you see a lot of similarities to software testing. If you think of the test case counting as the mechanical work - or the non-cognitive skill demanding work then you might recognise:

Measuring work based on test case counting -> worse performance.
Measuring and rewarding work based on mechanical skill (little/no thought implied) -> button pressing testing -> worse performance.

Dan goes on to suggest a way to get away from this type of de-motivation and towards a more motivated form of work, using Autonomy, Mastery and Challenge - aspects that are important where cognitive skills are necessary in ones daily work. He gives examples where this has worked in various communities.

If you have these in your daily work then I guess you're quite happy about it. If not, does this bother you?

Enjoy the animation!

[Dan Pink's talk at RSA]

Time to think about what's motivating you in your daily work...

Carnival of Testers #18

2011-02-02T13:48:00.000+01:00

" #softwaretesting #testing "

Carnival number 18, maybe it's coming of age, maybe still a little rebellious - ok, enough with the analogies.. January was a very reflective month - lots of look backs at 2010, some looking forward, new ideas, topics and illustrations - all making for varied and interesting reading:

Looking back

Darren McMillan was one of the first of the new year to look back, here.
Another interesting journey was described was Trish Khoo in her 2010 reflection.
A packed 2010 for Lynn McKee was described, here. I get tired just reading it...
The cartoon tester (Andy Glover) celebrated a first anniversary of the cartoon blog.

Looking Forward

Geir Gulbrandsen was looking to the year ahead with some ideas about his intended learning areas. Have a look.

Looking through the looking glass?

A fun reflection on activities with software testing in general was given by Pradeep Soundararajan, here.

Looking good!

The Dutch Exploratory Workshop on Testing (DEWT - is it the workshop or testing that is exploratory?) was born!
I enjoyed reading about Eric Jacobson's work on a first lightning round for testers - something many groups could easily adopt.
Martin Jansson made some interesting points about how testers can help with technical debt.
The SPIES mnemonic was explained by Nancy Kelln, here. There's also a pointer to Lynn McKee's listing of testing mnemonics, here.

Looking biographical!

Rob Lambert & the STC launched a new e-book on the Diary of a Test Manager. Rob says it's fictional and not about him!

Looking like an interesting meetup!

An STC-sponsored tester meetup is happening in February in Nottingham - Adam Brown wrote about it, here.

Looking...

What started out as a tweet from Michael Bolton turned into a series of posts, the last one here.

Looking clearer!

Seeing systems problems and taking a wider view was the topic of a post from Esther Derby. Worth reading.
A nice illustration from Joe Strazzere about issues with test estimation and guesstimation.
What does thinking outside the box mean? Some reflections from Lynn Mckee related to this.

Until the next time...

Framing, Threads and Books

2011-01-16T21:24:00.001+01:00

" #softwaretesting #testing #learning "

Inspired by various blogs reflecting on the past year and especially one from Rob Lambert mapping his reading and Geir Gulbrandsen comtemplating his upcoming reading I started thinking about some of the influences on my reading choices.

I then realised that there is a connection between my various reading choices, some of the ongoing study or work areas I'm involved with and discussions in the software testing community. So I started thinking of a picture of my reading as a framing exercise - to help elicit the thinking and reasoning behind my choices. This became an interesting mapping exercise - quite suited for a mind map.

The exercise would also help show up holes in my linkage (maybe I had several alternative sources and picked one - it would be interesting to show this - I haven't done that yet!) and also areas that I'd specifically excluded (this could be in the form of annotations or comments about progressing or not along a particular path - this isn't included in my current map.)

I split the groupings into books "read once" (meaning read at least once), "ongoing" and the "antilibrary" (from Taleb's The Black Swan - an unread source that demonstrates what you don't know - or in this case a subset of that.) As a first step the main sources were added as labels (whether via own research or via community recommendation or discussion - blogs, articles, test forums and twitter) as well as a linkage between books - which was read due to another book (either it was referenced or some research from that book pointed to another).

I'm going to leave my thoughts about the individual books to another time & place, as well as some coding (perhaps colour) about which ones are more frequently used and also some timescale (purely for my own benefit to help put the reasoning into context).

Other useful additions that may be made would be of magazine articles, e-books and other online literature (Rob Lambert's idea!) - as well as the aforementioned information about what was excluded and why.

The interesting aspect of this exercise was that I can see the framing behind parts of my learning journey but also that the map is a set of threads. Some of them are ongoing, some will be re-visited and no doubt extended and some will be closed down. Plus there's always room for new threads.

The way that I read books I do it in a very "thread-like" way - I have many "ongoing" at any one time and I revisit read ones. That's ok for me and charting them like this helps to record my (where I have and haven't visited).

Just like testing!

Update: I've also intentionally left out what I've learnt from each of these threads - that's a whole series of posts!

Carnival of Testers #17

2011-01-02T11:13:00.002+01:00

" #softwaretesting #testing "

December. A flood of posts - usually lots of reflections and year-end summaries, but also lots of variety and range of topics.

Confer-ence
There were some great reports, write-ups and postings concerning the EuroStar2010 conference.

Nathalie de Vries made a comprehensive round-up of the tutorial and conference days, here, here, here and here.

Nathalie also gave some tips for people considering sending in a proposal for the next conference.

A taster of the Danish Alliance was given by Zeger Van Hese, here.

Another comprehensive round-up of EuroStar conference was made by Michael Bolton, here, here and here.

Questioning

All good testers question what they're doing, observing or being presented with.

A well laid-out set of discussion points relating to an interview with Rex Black was presented by Ben Kelly. Let's hope there's a follow-up post with some results.

Pete Houghton highlighted some of the problems with the thinking connected to regression testing.

Inspiration and resources

Sources of ideas, tools and techniques in use by testers everyday...

Do you ever test to music? Some interesting insights in Christin Wiedeman's use of music in her testing, here.

I enjoyed reading Chris McMahon's story about some of his idea sources.

Rikard Edgren's take on "Exploring Requirements" was a reminder of a great book that is also useful to testers.

If you haven't seen or tried corkboardme then take a look at Tim Coulter's introduction.

Transpection. Stephen Hill made a good write-up of his experience with it, even including the transcript of the session, which is worth a read.

And finally...

Be careful what you measure, and why, was wrapped up into Jerry Weinberg's parable of the ones.

Michael Larsen's reflection on the year was an inspiring read. Nuff said!

Markus Gärtner announced a repository for testing challenges, here.

Update: 2 Jan 2011: I meant to add this, but it slipped through the gaps...

Tobbe Ryber posted a notice to say his book (Essential Software Test Design) was now available for free download. It's a worthwhile reference - even if you have the paper copy - highly recommended!

Until the next time...

Language can be confusing

2011-01-02T05:03:00.000+01:00

" #softwaretesting #testing "

I was reminded recently of the confusion that is wrought by both written and spoken language.

I read Michael Larsen's account of a weekend testing session, here. Whenever I see or hear about this type of potential for confusion I usually think of what Tor Norretranders says in The User Illusion, that communication is a process - there needs to be a synchronisation between sender and receiver for it to happen effectively.

How to combat?
This synchronisation or handshaking is a form of check with the other person, "Did you understand? Ok, repeat back what you think I just said in your own words".

Other ways to help elicit true intentions or meanings might be to use something along the lines of context-free questioning from Exploring Requirements (Gause, Weinberg). If you don't have access to the book you can read Michael Bolton's post.

Another Gause & Weinberg book, Are Your Lights On, gives another exercise - the "Mary had a little lamb" example. Here the emphasis is changed on each word as well as changing the words themselves (via dictionary or thesaurus) to get at the real or important meaning of the sentence.

Why?
As a tester the room for confusion and mis-communication is vast - whether this is restricted to a customer requirement, a stakeholder's view of an activity, result or interpretation or writing a fault/bug report in a useful way (to your team, stakeholder and developer).

Being aware of the potential for traps in mis-communication is valuable!

For anyone thinking that spoken communication is much simpler than written you might want to take a look here. Enjoy!

I haven't even mentioned confusion in translation - I'm reminded of a presentation in Swedish where the emphasis on words was misplaced so that "the following six pictures" became "the following sex pictures" - and immediately caught the attention of the audience!

Thoughts on Test Framing

2010-12-19T15:44:00.000+01:00

#testing #softwaretesting

Some thoughts on the ideas presented in Michael Bolton's post.

Firstly, I like it. A lot. I commented on the post about a day after it was published. But I knew there was some more thinking I needed to do on the topic. So here we are, over two-and-a-half months later (yes, I think slowly sometimes - sometimes one-dimensionally and sometimes in a very divergent way - meaning I create lots of strands/threads to follow-up - but I do continue thinking, usually! )

Powerful Elements
Showing your thinking is crucial to any activity that involves questioning, observation, hypothesis forming, evaluation, reporting, learning and feeding back under constraints of time, cost and equipment availability (this is testing, if you didn't recognise it already).

Showing the provenance of your test ideas is a powerful communication tool - both for yourself, team and any stakeholder. Doing this with many of your ideas will naturally show up if many of the thoughts/considerations you'd intended to include were indeed included. (What?) Ok, example time.

Pseudo-Example
A simple example here is to construct a test idea around a "happy path" of a function, the most basic and usual case for that function. This will then lead to thoughts around:

The use case itself (variants and alternatives);
Platform (how many?);
Environment (Small with many simulated elements going right through to a full customer acceptance configuration);
Application configuration (Based on what? observed or projected?);
User data (Based on what? Supposed, observed or projected?);
User behaviour (Based on what? observed or projected?);

I work in situations where these are distinct variables - not so much overlap, but in some shops some of these could be less distinct.

Evaluating and selecting which of these elements to use, include and vary is part of the test design of the test idea in conjunction with the test strategy. So, in this example we can see (hopefully) that there is a range of test ideas that could be distilled.

What More?
In the example I alluded to elements of choice/decision based on risk, priority, test design technique, strategy and availability. Sometimes this is about scoping the area for test, what to include (for now) and exclude (for now).

When I am telling the story of my testing I like to emphasise the elements that could be missed - areas I've skipped and the reasoning behind that, areas that maybe were discovered late that could have an impact on the product and for which it might be useful to recommend additional testing.

So, I want my story to be very explicit about what I didn't do. There is nothing new in this - traditional test reports were including this information over 20 years ago. But, I want to link it into my test framing in a more visible way. If I start talking about all the things I didn't do it's easy for a stakeholder to lose focus about what I did do.

So, as well as the explicit chaining of test framing showing the roots and reasoning behind the test idea I also want to include (explicitly) the assumptions and decisions I made to exclude ideas (or seeds of new ideas). In the context of the test frame this would represent the a priori information in the test frame/chain (all of the framing (so far) is a priori information).

But, there might also be scope to include elements of ideas that would affect the test frame (or seeds for new ideas) with information discovered during the testing. Then from a test frame perspective it could be very useful to include this a posteriori information.

Examples of a posteriori information would be feature interactions that were discovered (not visible before) during testing - which didn't get covered before testing stopped. There might be aspects where test ideas couldn't be 'finished' due to constraints on time, feature performance (buggy?), third-party sw issues (including tools) that couldn't be resolved in time or some other planned activity that didn't get finished (a feature element that needed simulation for example).

I usually think of these aspects that are not 'included' in the testing story as the silent evidence of the test story. Making this information available and visible is important in the testing story.

There are aspects of these assumptions and choices that are implicitly in the test framing article, but for me it's important to lift them forward. (Almost like performing a test framing activity on the test frame post.)

Maps
As a side-note: Test framing as a form of chaining / linking around around test ideas fits well into a mind map, threads and test idea reporting. But that exploration is for another time...

So... Have you been thinking about test framing?

Risk Compensation and Assumptions

2010-12-19T01:47:00.000+01:00

#softwaretesting #thinking #WorkInProgress

This is very much a thought-in-progress post...

Cognitive bias, assumptions, workings of the subconcious. They're all linked for me in one way or another. Then the other day in a twitter chat with Bob Marshall and Darren McMillan I was reminded of a potential link to Risk Homeostasis (sometimes called Risk Compensation).

Risk Homeostasis is the tendency for risk in a system to have some form of equilibrium, so that if you devote more attention to reducing risk in one part of the system you increase the risk in another part. This is a controversial theory, but there is an intuitive part that I like: devoting increased attention to certain risks inevitably implies devoting less attention to other risks. To use economic terminology there is potential for increased risk.

This also ties in with an intuitive feeling that if you are devoting less attention in one area the "risk" components there could increase without you noticing. This is almost a tautology...

I first encountered risk homeostasis in Gladwell's description of the investigation into the Space Shuttle disaster (in What the Dog Saw) and I see the similarities to testing.

Stakeholders and Risk Compensation?
A software testing project example. Taking the project/product/line manager's view about testing:-

A correction; release is coming up and a group related to release testing is performing a release test activity. That, in some minds could be seen as a "safety net" - could the PM be influenced in the level and amount of testing "needed" prior to the release test activity? I think so, and this is risk compensation in action - so-called safety measures sub-conciously allow for increased risk-taking elsewhere.

It could apply to new unit, function/feature testing needed for a "hot fix" - an "isolated" decision about what's needed for this hot fix - PM thoughts: ok, let's do some extra unit/component test as we can't fit in the functional test (which is dependant on a FW update - that can't be done at so short notice), we'll do extra code reviews and an extra end-2-end test. Seems ok?

Then comes the other deliveries in the "final build" - maybe one or more other fixes have had this "isolated decision making". Then, for whatever reason, the test analysis of the whole is missed or forgotten (PM thoughts: it is short-notice and we're doing an end-2-end test) - i.e. putting all these different parts into a context with a risk assessment of their development and "the whole caboodle", from the PM perspective there was a whole bunch of individual testing and retesting, there will be other testing on the whole - that's enough isn't it?

Assumptions
Where do assumptions come into this? Focussing on certain risks (displacing risk) results in some form of mitigation (usually). But, the sheer act of risk mitigation (in conjunction with risk compensation) implies that focus is reduced elsewhere. The danger of assumption here is that the mitigation activity is coping with risk, when in certain cases it's just 'not noticing' risk.

The act of taking action against risk (mitigation) is opening possibilities to trust (and not question) our assumptions. But how do we get fooled by assumptions. There are two forms that spring to mind:

The Black Swan. Something so unusual or unlikely that we don't consider it.
A form of the availability heuristic. We've seen this 'work' in a previous case (or last time we made a risk mitigation assessment) and so that's my 'reference' - "that's just how it is" - "all is well".

An everyday example of Risk Compensation
Topical in the northern hemisphere just now: Driving in the snow with winter tyres and traction control. I see it a lot these days (even do it myself) - when moving off, don't worry too much about grip - just let the car (traction control) and tyres find the right balance. So the driver is trusting the technology - when I used to drive with winter tyres and no traction control, there was a lot more respect about accelerating and 'feeling' the road, ie using feedback.

Maybe that's where risk compensation plays a part. It reduces the perceived importance of feedback.

Coping with Risk Compensation and Assumptions
The most important aspect in coping or dealing with this is: awareness. Any process that has a potential for cognitive bias is handled well by awareness of the process. Be aware for when feedback is not a part of the process.

Understanding those situations where short-cuts are being taken or where more emphasis is put in one area we should ask ourselves the questions:

How does the whole problem look?
Am I missing something?
Am I aware of my assumptions?
Am I using feedback from testing or other information about the product in my testing?

Have you noticed any elements of risk compensation in your daily work?

Carnival of Testers #16

2010-12-03T16:27:00.003+01:00

November was cold, much colder than normal - but there is no such thing as bad weather, just inadequate clothing (there are several test analogies there!), so whilst deciding on the right clothes I read and have been entertained by several blog posts...

First up this month was Michael Kelly with a reminder that sometimes it's necessary to "just say no!"

Ever get the feeling that explaining any test-related thinking to a non-techie is tricky and full of traps? If so, then you'll recognise something in this cartoon from Andy Glover.

On the subject of techie / non-techie, do you recognise anything in Adam Knight's "Don't call me technical" post?

So, have you recognised anything so far?

On an aspect of recognition Albert Gareev wrote about a typical trap that testers can occasionally fall into, inattentional blindness. Recognising it, and understanding it helps your learning.

Related to traps, bias and fallacies then black swans have been known to surface. Have a look at this 'humble' story from Pradeep Soundararajan and his sources of learning.

Of course, sometimes nothing will disrupt you and your testing. Then, maybe you're in the zone, as Joel Montvelisky encourages us to recognise and learn about the contributing factors.

A short note from Peter Haworth-Langford on his first year of blogging. Happy blogging birthday!

The guys at the test eye produced a two-page sheet of aspects for consideration when testing a product. It's partly based on earlier work by others, but take a look and see what you recognise and if there's anything new to you.

Communication, communication, communication. Take a look a Pete Walen's post on some communication aspects related to documentation.

A nice example of the availability heuristic in Gerald Weinberg's account of The Sauerkraut Syndrome. Recognose it?

Stuart MacDonald made a good debut on Darren McMillan's site with a tester challenge write-up.

A view on testing debt and some tips to counteract it came from Martin Jansson.

Weekend Testing landed in the Americas during November. Here are some thoughts from one of the organisers, Michael Larsen.

Bob Marshall raises some pertinent questions about the state of Agile - thinking way outside the tester's box. Recognise anything?

Some more interesting questions raised by Mark Crowther on burndown. You must reconise something here.

Maybe you recognise Shaft, but have you heard of beshaftitude? Erik Lonnrot takes a light-hearted view of metrics.

A long, but worthwhile read, from Len DiMaggio on a colleague's team fragmentation problems.

I'm sure there was something there that all would recognise, and maybe something new.

Until the next time...

Deliberated Exploration - A Walk in the Woods

2010-11-17T00:01:00.000+01:00

There is a view of exploratory testing that believes it is spontaneous (as in combustion) - one just turns up at a keyboard (or other lab/test equipment) and you "just do it"!

Nice, if you can get it.

But what if your system under test is not trivial - in terms of data set-up, configuration, simulated subsystems, user profiles, behaviour or network elements? Does this mean that you can't have an exploratory approach?

Well, of course you can.

Think of it as going on a walk/hike/trek in an area you haven't covered before. To actually get to the "start" of the hike you might use a series of public or scheduled transport (a pre-determined schedule) to get to the start point.

I emphasize the uncertainty around the "start" as you might move this depending on how you defined where the trip should start.

You might decide to start your walk when stepping off of the bus - the bus is a pre-determined script to get you to a point where you will start - but perhaps you notice something odd on the journey; maybe the bus takes a route that isn't signposted as your destination - is it a pre-determined route or a diversion, will the journey take longer ... The questions start.

Using a script to get to your starting point doesn't mean that you can't ask questions - remember, this is testing with your eyes wide open as opposed to eyes wide shut. You finally get to your start point - where the trek can begin.

Just like an exploratory approach - there might be a whole series of set-up and configuration to get the system into a state ready to start working with your test ideas. It might be that you need to construct a simulation of a network element to start testing in the desired area. All of this may take some time to achieve - a whole series of hoops and hurdles.

In some of the systems that I work with this is exactly what we do - it's about a certain design (simulation, config and test ideas) - getting to the starting point where we can start working with our test ideas.

Sometimes the test ideas yield more investigative tracks than others - but that is exactly where we "intend" the exploration/testing to start. But like any (good) tester we notice things on the way, we raise questions about our simulation assumptions, about the problem/test space in our target area - and we learn about the product (along the way and in our target areas) - we learn about our test ideas, we learn about our test ideas, if we're missing something vital in our simulation data, configuration or how the whole might relate to a real end-user.

That was our walk in the woods, and the scripted element is very much a part of the trek/exploration. The scripted part is setting the conditions for the exploration - an enabler - whether I am questioning the set-up as we go is a matter of preference, time, priorities and feedback from previous testing.

Do you observe or get curious on the way to your walk in the woods?

Carnival of Testers #15

2010-11-03T02:45:00.001+01:00

A lot of conferences this month - a real feast - almost an Octoberfest. Or was it a case of watching repeats on TV?

If you're a festival fan, a watcher of repeat TV or on the look-out for something new there was something to be had this month.

Repeat Questions?

Pradeep Soundararajan wrote a piece about labels in testing, specifically the domain label. Read the post and the comments, here.
A question about measuring tester efficiency prompted Shmuel Gershon to not follow the numbers game and show just how difficult that is to answer.
Dave Whalen his view on the QA vs QC in testing topic, here.
Testers who code or not was a topic of a James Christie post and he even managed to get a football reference in!

Agile Testing Days

Markus Gärtner was doing some sort of blogging marathon producing reports from many of the talks he visited: Lisa Crispin's talk on agile defect management, Janet Gregory's talk on learning for testers and Rob Lambert's talk on structures killing creativity.
Zeger Van Hese made some write-ups, this post on the day 3 talks could make you wish you'd been there.
Rob Lambert was giving his alternative angle on conference with a series of video snippets, here, here and here. And to show he hadn't forgotten to blog he also did a write-up, here.

STARWest

Jon Bach wrote a piece about an impromptu flash mob exploration session out in the corridor. Ideas for a new class or lightning talk?

STPCon

A fun presentation of some aspects of combinatorial testing was presented by Justin Hunter.
A sense of variety at STPCon was had from reading Matt Heusser's roundup, here.

SWET
The first Swedish Workshop on Exploratory Testing was held this month. It generated a certain amount of buzz and energy, with several people setting down their thoughts:

A view from Simon Morley, here; from James Bach, here; a semi-transcription from Rikard Edgren, Henrik Emilsson and Martin Jansson; Ann Flismark's view, here; Christin Wiedemann's view, here; Oscar Cosmo's reflections (in Swedish), and here (in English).

Challenges
Challenges are popular and this month saw a couple of interesting ones.

A challenge on deconstruction and contextual inference came from Zeger Van Hese.
This prompted Thomas Ponnet to walk through his answer and thinking.
Darren McMillan wrote about an impossible(?) challenge, here.

Miscellany

A review of Naomi Karten's book on presentation skills was the subject of a Lisa Crispin post. I'm tempted to purchase...
A good reminder on the importance of interpretation was given by Zeger Van Hese, here.
If you're wondering about how to think about test estimation in your project, you'll be set on the road to clearer thinking by reading Michael Bolton's post on test estimation in projects, here. It's the fifth installment of a group of worthy reading.
An angle on pro-active (or is it pre-emptive?) testing was given by Darren McMillan.
An intriguing idea of Test Proposals was presented by Martin Jansson, here.
It's been fascinating reading about James Bach's journey across Europe - this one from a stop in Romania. I just can't get the theme tune for Rawhide outta my head now...

Until the next time...

Swedish Workshop on Exploratory Testing - #SWET1

2010-10-17T23:45:00.000+02:00

Place: Högberga Gård
Date: 16-17 October 2010
Twitter hashtag: #SWET1 (a couple of early tweets with #SWET)

Attendees: Michael Albrecht; Henrik Andersson; James Bach; Anders Claesson; Oscar Cosmo; Rikard Edgren; Henrik Emilsson; Ann Flismark; Johan Hoberg; Martin Jansson; Johan Jonasson; Petter Mattsson; Simon Morley; Torbjörn Ryber; Christin Wiedemann

These are just some very quick reflections on this weekend's activity (more to come after further reflection) - SWET - for which I've seen two meanings (SWedish Exploratory Testers and Swedish Workshop on Exploratory Testing).

This was an inaugural peer conference on Exploratory Testing in Sweden.

Peer Conference
A first time attending a peer conference for me - I'd seen the rules beforehand, but didn't really know what to expect - either in terms of process or result. The great idea with a peer conference is that someone presents an experience report and then there is a moderated question and answer session.

Open Season
James described that part of the rules were designed to stop disruption from fellow peers (maybe himself being a prime culprit in the past.) The idea of having open season was both challenging and interesting - not just for the presenters but also their peers (James mentioned that reputations could be made or broken depending on what went on there - no pressure there then ;-) )

The open season was much longer than any of the presentations - I think the first session had an experience report of 20-30 minutes followed by 2-3 hours of questioning! Cool. Or? Think about some of the talks you may have attended at a conference - would the presenters stand up to 2-3 hours of questioning?

This was not just banal questioning either - this was serious searching and reflection from interested and passionate testers!

Learning, Buzz and New Faces
The peer conference is a great model for learning about and exploring people's experiences, opinions and views. This didn't just happen during the presentation and open season - it continued in the coffee breaks, meal breaks, during the after-session activities - I was still talking testing past 02:30. Everywhere you looked people were discussing testing and sharing opinions. A great vibe and buzz from the event!

I was able to put faces to names I recognised from blogs and online forums and got to know a lot of new testing colleagues - for all of them I say: respect!

There was some offline discussion about blogs and I think the blogosphere will see at least two new blogs (hopefully soon) - and that's something we can all look forward to - there are some great stories, thoughts, ideas and questions just waiting to get out there!

So, as you can maybe guess, I loved it. Special thanks to Michael for a lot of hard work with the arrangements and to Michael, Tobbe & Henrik for moderating duties.

Until SWET2! I can't wait.

View from the window on Sunday morning

BTW, Henrik E & Martin will be running the test lab at Eurostar this year. After meeting them I think the test lab is going to be pretty cool!

Carnival of Testers #14

2010-10-01T00:05:00.000+02:00

September, apples falling off of trees, picking apples, apple pie, lots of new apple sorts in the local shops, sometimes you make an interesting, tasty discovery. Autumn's here (for me), leaves on trees changing colour - lots of different shades and nuances - just like some of the blog posts this month....

Old Regulars

The case for slack was an interesting topic from Markus Gärtner.
Games, strategy and kids all rolled into one in Anne-Marie Charrett's post.
A interesting perspective on estimations was provided by Lisa Crispin.

Taste Sensation

If you've missed Elisabeth Hendrikson's piece on Agile wakeup call (backlash vs wakeup call), then go read it now.
Zeger Van Hese made my day, no, week,with his cautionary message about delivering the message. It wasn't Zeger but the content that was delivered AT FULL VOLUME.
And talking about things being broken (the delivery style in the previous case), Michele Smith highlighted this talk about things being broken - yes, I could see plenty of relations to my daily work. Can you?
Marlena Compton came out of the testing closet this month. Have you?

Different Tastes?

Test Framing was a new "term" to appear this month, with this post from Michael Bolton. BTW, Michael wrote a piece urging people to hire Ben Simo - but it doesn't need any plugs now :)
A dip into the sub-concious and concious distinction was observed in Blink Comparator Testing by Simon Morley.
Joe Strazzere made some good points about willingness to learn in testers, here.
A fish baking story was the way Shrini Kulkarni asked a good question.

Colour Explosion

A colourful post from a colourful tester. Was Dave Whalen testing with gumbo or gusto, or both?
Gerald Weinberg pondered questions around SW projects hitting the wall.
uTest posted an interview with James Bach which in itself made interesting reading, but if you haven't seen it then I'd encourage you to go and read the comments too!

Seasonal Favourites
Yes, there seems to be several conferences or meet-ups happening right now.

Alan Page and his views on the different types of presentations. Cows? Didn't think he'd seen any of my work.
Adam Goucher on the track vs keynote distinction. I can recognise elements of what they're both saying.
A good round-up of SIGiST was provided by Stephen Hill.

New Discoveries
Nice to see some new apples (faces) this month.

Darren McMillan used Jing to record an issue with Jing. I wasn't familiar with Jing but I am now, thanks.
Off to a blogging bang with a very readable self-analytical piece from Lena Houser.

Did you notice I avoided bad, rotton and half-eaten apple metaphors? Oh, the temptation...

Until next time.

Iqnite Nordic 2010 - Day 1

2010-09-30T09:38:00.000+02:00

The first day of the nordic Iqnite conference was a varied affair.

The morning kicked-off with a keynote from Mary Poppendieck with a talk looking at how "requirements" usually get it wrong - i.e. they are commonly not what the customer wants. I think I'd seen this message before but it was presented in quite a different way - it was looked at from the perspective of a new start-up, where there often aren't so many (if any) customers - and so the businesses that succeed are the ones solving a problem in a better way than the available competition. Examples used were the google search engine and eBay, amongst others.

The message that I took away was that the requirement handling is often too formalised and kept as too much of an internal activity - and that loses contact and feedback with the customers. An alternative is to develop a requirement capture partnership between customer and supplier.

The talk was finished with an illustration of the push vs pull production method that I've read about in a previous Poppendieck book, Implementing Lean Software Development.

My talk, Test Reporting to Non-Testers, was next up in that room. I had an interesting intro from the track chair (Pål Hauge) who used a Norwegian metaphor for my intro - as I was coming on after Mary (and so using some of her momentum?) - I was doing a "Bjørn Einar Romøren" (?? famous ski jumper) - however, I realised afterwards that a Brit being associated with ski jumping (Eddie the Eagle?) is maybe not the best association :-) I was glad that my cold-ridden voice lasted (it packed in by the end of the day). My message about not relying only on the numbers, with some example test reports, was understood, I think!

I attended other talks about requirement capture and team member make-up. The team member make-up alluded to a concept of "social capital" and emphasized the value of trust. I agree with the trust angle, whether in a team or as an individual and have pointed this out before in my encouragement for testers to build their brand.

I enjoyed meeting some interesting people and chatting about problems in daily work, some points about agile, scrum and kanban which I'll explore another time.

The speaking-part of the day was rounded off with a keynote from Gunnar "Gurra" Krantz, a swede who's competed four times in the round-the-world yacht race - his talk on team and process dynamics was very interesting. Not having any walls around the toliet (or even a curtain) in order to save on weight! Plus, it was placed next to the galley so they could always lend a hand!

Yes, software is a different world!

(I won't make the second day - customer meetings call :-( )

Lessons in Clarity: "Is this OK with you?"

2010-09-20T00:18:00.000+02:00

Sources of confusion, mis-communication and unclear expectations lie all around - more than you suspect if you look carefully. Communication and clarity is something I'm very interested in - although I like to hope all testers are. Well, a recent email exchange made me laugh at the lack of clarity - to me - or maybe the joke was on me!(?)

I (person A) sent an email to person B with a CC to persons C and D. The email contained two distinct pieces of information, item 1 and item 2.

There was a fairly quick reply - the reply went from person B to person C, with CC to persons A (me) and D. The content of the reply was very simple: "Is this OK?"

Mmm, did this mean all the information in the first email or only one piece of it (item 1 or 2)? Who knows? I didn't, but I was keen to see the reply (if any). Was it only unclear to me, all, or just some of us?

The next morning I saw the awaited reply from C: "Item 1 is OK with me." These emails are breaking some record in brevity - simplistic - and also reducing the information content, and not necessarilly reducing the confusion-potential content. This reply raised other questions about item 2 (for me):

This is not OK.
No opinion on it.
Oblivious - the question from B was misunderstood.
This is OK - then the question from B was known/understood (somehow) to apply to item 1.

And?
Who knows what this last reply meant - who cares (I hear you say) - but for me this was another example in potential sources of confusion.

One way I try to avoid this:

If I can't speak to the person and clarify the situation then I state my assumptions up-front, then they can reply with a clarification or correction - either way we reduce the scope for confusion and misunderstanding.

Note on brevity: It's OK if those involved are on the same wavelength. A great example of this was when Victor Hugo sent a telegram to his publisher after a recent new publication:

VH to Publisher: ?

Publisher to VH: !

But, if in doubt, spell it out...

New "What's it all about?"

2010-09-19T23:22:00.000+02:00

Since I started this blog in April 2009 I had the word "quality" placed somewhere near the top of the page - specifically in the "What's it all about" box. I've just made a change from:

The Tester's Headache is sometimes about issues connected to the balancing act of executing a test project on Time, on Budget and with the right Quality. Other times it's a reflection on general testing issues of the day.

to:

The Tester's Headache is sometimes about issues connected to the balancing act of executing a test project on Time, on Budget and trying to meet the right expectations. Other times it's a reflection on general testing issues of the day.

I've been doing a bit of sporadic writing/thinking about this recently, the occasional tweet and the odd comment on selected blog posts. There will be more to come soon (I hope - all to be revealed soon) on my thoughts around the word 'quality' in the testing-specific domain.

I'll still occasionally use the #qa hashtag on twitter - hey, I can't start a revolution/re-think from the outside can I? Or can I?

Thoughts, comments? Provocative? Thought-provoking or sleep-invoking?

Blink Comparator Testing vs Blink Testing

2010-09-17T00:48:00.000+02:00

I saw an article on the STC the other day about a competition to go to the ExpoQA in Madrid.

I was immediately interested as I backpacked around Spain years ago and Madrid is a great place to spend time, but I digress...

I scanned the programme (top-down), got to the bottom and started to scan bottom-up and immediately saw an inconsistency. Then I started wondering why/how I'd found it. Was it a particular type of observation, test even?

I realised that when I started scanning upwards I had some extra information and that was why it popped out. I twittered a challenge to see if anyone else spotted anything - at the time I added the hashtag #BlinkTest. After further reflection I wasn't so sure that was correct - I started thinking about the blink comparator technique that astronomers used once upon a time.

It seemed like a blink test (that I've seen described and demonstrated) - but after some thought it seemed closer to a blink comparator test. Was there a distinction, what was lying behind why I'd spotted the inconsistency? Was one a form or subset of the other?

Could Blink Testing be a general form of information intake (and subsequent decoding) and Blink Comparator Testing being a special form where the pattern, legend or syntax is specified and the scanning is aimed at spotting an inconsistency. Maybe. Hmm...

Blink vs Blink Comparator?

I'm going to think of a Blink Comparator Test as being one that takes a particular type of priming (concious) - here's a pattern (or patterns) and this is used to compare against the observation - maybe for the absence or presence of the pattern/s.

I'll think of a Blink Test as being an observation without priming (subconcious) - although there will always be some priming (from your experience), but that's a different philosophical discussion - and it's the subconcious that waves the flag to investigate an anomaly.

Of course, both can be used at the same time.

Why distinguish?

It's about examining why I spot an inconsistency and trying to understand and explain (at least to myself) what process is going on. So, why the interest in understanding the difference? For me, that's what helps get to the bottom of the usefulness and potential application, and indeed recognising the technique 'out in the wild'.

This started out as looking at an anomaly (maybe even a typo) in an article, and now I have an addition to my toolkit - I probably already had it, but now I'm aware of it - and that means I have a better chance of putting it to good use. I can see uses in documentation, test observations and script design (to aid observation/result analysis). Cool!

Oh, the inconsistancy I spotted was the use of (Sp)* in the time schedule, which wasn't in the legend. Simple stuff really producing all that thinking...

Carnival of Testers #13

2010-09-02T00:28:00.001+02:00

Number 13 edition of the carnival. Lucky, unlucky, is it all going to fall apart in a dis-jointed mess? Let's see...

No Breakages?

The August selection started off with a bang. Trish Khoo reflected on a mindset of breaking stuff as just being one of many tester mindsets.

(Look!) No Hands?

Anyone else out there that has helium hands? If so, or wondering what that is, then read Michael Larsen's view, here.

No Nonsense!

CAST got some reporting from Anne-Marie Charrett guest posting on the STC. Read the report on the first day, here.

Not Scary enough?

Do you have scary test plans, test ideas or are just a bit scary yourself? Catherine Powell ponders the value to the customer of a scary test plan.

Balls?

It's a good while since I juggled balls successfully. Juggling daily work is also filled with problems, as highlighted by Joe Strazzere.

No Testing?

Ajay Balamurugadas wondered about how or if he was testing when he wasn't testing. Confused, intrigued? Then go read the story, here.

No Competition?

Competition time again. Pradeep Soundararajan wrote about some of the competitions that have been occuring, with some plus points and some room for improvement.

Interviews!

A three-part interview of Ben Simo was made by uTest. Worth reading!
The Software Testing Club started publishing a series of short interviews, featuring Lisa Crispin, Anne-Marie Charrett, Parimala Shankaraiah and Simon Morley.

No Comparison?

Tim Western described his thoughts about testing being an inter-disciplinary skill.
A comparison between BBST and RST was made, here, by Geir Gulbrandsen.
Thread-Based Test Management was coined by Jon Bach, here, and James Bach, here. Go read them, think about it, question and think some more.

No Punctuation? No Problem?

Well, made it to the end. This month's carnival was brought to you by the punctuation marks ! and ? Finally, a reminder of how punctuation can trip up testers is here.

Until next time ...

Strung up by the Syntax and Semantics

2010-08-23T06:45:00.002+02:00

Ouch!! Sounds unpleasant doesn't it?

I've just re-read Eat Shoots and Leaves; originally read it a few years ago, then forgot about it after leaving my newly finished copy on an airport coach in some foreign land.

So, it was a joy to re-discover this book and read again over the summer.

What's all the fuss about? It's just punctuation!
That's the thing! A lot of testing communication - whether the requirement specification of the product or the reporting of the results and findings usually happens in written format, and it's the written format that has potential pitfalls for misunderstanding and confusion.

So, when you receive a document/description/mail is it always unambiguous. Is it ambiguous on purpose (ie unknown behaviour not documented) or by accident? Is your own writing similarly unambiguous?

True, it's possible for the spoken word to be ambiguous, but then we usually have the luxury or putting in a question to clarify our understanding.

So, the moral is: ambiguous? Start digging - is it accidentally unclear or is there a source of potential problems in your testing - the 'swampy area' on your testing map!

Superfluous hair remover
I'm getting into that age range where more hair is developing in "strange" places - ears - and less on the top of my head. I'd love to hear an evolutionary explanation for hairy ears - that were not needed until now... (Oh, I'm not being exclusive either: any creationist can chip in with the design explanation also. Agnostics: press the "don't know" button now.)

So when I read the phrase (superfluous hair remover) it immediately struck me on two levels - the age-related one that I just alluded to and the tester in me that wonders "what does that mean?" or "how do I interpret that?".

Do you see what I mean?

Is it talking about a hair remover device too many (a device that is superfluous) or is it talking about a remover of hair when you have more hair than is wanted (superfluous hair)?

What's the solution to this written problem? Well it's a hyphen. A "superfluous hair-remover" might be something I'd see advertised in a "Unwanted items" newspaper column. A "superfluous hair remover" or "superfluous-hair remover" might be something seen in a "For Sale" column or store. Note, there's still room for ambiguity though!

Lesson?
Don't always assume that the writer of a product description, specification or requirement outline or specification is writing exactly what they mean.

What, I hear you say, why wouldn't they write what they mean. Well, that comes down to ambiguity in the way someone (maybe not a technical writing specialist) describes the product - both the wording and the punctuation.

If it looks fishy/suspect - there's a good chance there is a need for testing investigation (and/or a bunch of questions).

Also if you're writing reports - whether test progress or bug reports - make sure that the written word is saying what you think it is (or at least your interpretation - true your interpretation may be skewed ...).

My sympathies to anyone who is writing or speaking in their non-native language. I make the odd slip in Swedish - although not as bad as a colleague who whilst giving a presentation got the emphasis wrong, so what should have been "you can see from the following six pictures" came across as "you can see from the following sex pictures" - that's one way to catch the attention of the audience!

Watch those hyphens and pauses!

Good Testing & Sapient Reflections

2010-08-18T22:09:00.000+02:00

It was a good while since I read James Bach's posts about sapient testing, here, plus some of the previous ones (here and here). I even contributed some observations on it in the testing vs checking discussion.

I have an understanding (my interpretation) of what is meant by sapient testing and it's something I can tune in with completely. In my book there is no room for a human tester to be doing non-sapient testing. Agree?

So, do I call myself a sapient tester?

No.

I work with a lot of testers and non-testers. Sapient testing wouldn't be any big problem for the testers I work with to understand and accept. The non-testers might be a different story - for the past 3 years in my current role I've been working on introducing intelligent test selection (initially applied to regression testing) within a part of the organisation.

I've made a big deal about thinking - about the testing we're doing, what we're doing and when, what we're not doing and trying to get a grasp on what this means - for me I've been contrasting this with the abscence of these aspects. I haven't called it non-intelligent testing, I've just made a point of not calling it intelligent testing.

At the same time I've also started introducing the phrase "good testing" and implying that if we're not thinking about what we're testing (and why, what it means, what we're not covering, what the results say/don't say etc, etc) then we're not doing "good testing".

Of course, there's scope for people to say "I think about my testing" when they might not - I observe, question and discuss, and together we come to a consensus of whether they're thinking about their testing: what assumptions are involved, what are the observations saying, what are the observations not saying, should the test approach be changed, what else did they learn...

By using the phrase "good testing" I'm also priming the audience - especially the non-testing part - they want the testing to be good and now they're learning that this implies certain aspects to be considered. Eventually I hope they automatically connect it with the opposite of "just" testing.

So, changing an organisation (or even a part of it) takes time - just like changing direction on an oil tanker - there's a long lead-time. That's one of the reasons why I don't use sapient or sapience in my daily vocabulary - I'm using the "keep it simple" approach towards non-testers.

Pavlov or Machiavelli?
If that means using "good testing", "intelligent testing" or "thinking tester" to produce a type of Pavlov reaction from non-testers then I'm happy with that. Does that make my approach Machiavellian?

So, although I might be sapient in my approach and identify with a lot of the attributes of being a sapient tester, I do not call myself sapient. Does this make me a closet sapient tester?

Are you a closet sapient tester? Have you come out? Are you Machiavellian or otherwise nicely-manipulatively inclined?

Are you a 'good tester' or do you just do 'good testing'? Do you know or care?

Problems and Context Driving

2010-08-18T01:39:00.000+02:00

I got involved in a small twitter exchange about problems, bugs and perception of the problems the other day with Andy Glover and Markus Gärtner.

Problem Perception
There was a view expressed that a user's perception of a problem is a problem.

I'd agree with this in most "normal" cases. But then there was a knock at the door and Mr. D. Advocate lent me his hat.

So putting the devil's advocate hat on I thought about how this view might not be enough. Or borrowing De Bono's phrase, "good, but not enough".

Cars
My way of looking at this for a counter-example was to think of driving a car. If I press a button, or depress a lever, expecting a certain response or action and something totally different occurs is this a bug? It might be, depending on what the button/lever was and the response.

If I'd pressed the button marked AC and the radio came on - I might think, "there's a problem here".

If I'd pressed a lever for the windscreen wipers and the indicator blinkers started then I might double-check the markings on the lever.

Blink Testing
BTW, Anne-Marie Charrett labelled this lever mix-up as an alternative form of "blink" testing!

Further Ramblings
I then started having flashbacks to my own encounters with cars in different countries and how I'd understood the issues:

Being stuck in an underground carpark at SFO, engine running, facing a wall, unable to reverse as I couldn't release the parking brakes - there were two and one was hidden. Manual to the rescue. Perception issue about where I'd thought the parking brake release would be.

Driving to San Jose (same car) and a heavy rain shower starting - how the heck do I switch the wipers on? Pull over and get the manual out. Perception problem about where the lever normally is.

Opening a taxi car door in Japan - taboo. Soon followed by the next of trying to tip the driver. Applying customs of one culture to another. My problem.

Driving someone round a roundabout (left hand side of road) when they'd only ever experienced driving on the right hand side. Were the muffled screams a "problem" or just a perception issue?

Trying to park in Greece - not on the pavement, unlike everyone else. Problem with local customs using my perception of a norm.

Sitting in the back of a taxi on the way to the office outside Athens going round a mountain pass whilst the driver is reading a broadsheet. Is my anxiousness a cultural thing?

Being surprised by overtaking customs in Greece. Flashing lights before the maneuvour starts. Irritation and confusion. My problem, not being familiar with the local customs.

Driving round the motorways of France and Italy - minimal gaps between cars - my problem, perception problem?

Driving in Sweden - coming face-to-face with a moose at speed. My problem. Obviously the moose has right of way!

Image by Steffe via Flickr

Problem vs Perception?

Problems can have fixed interpretations (this is an agreed issue) and areas of vagueness. Is this my perception or interpretation?

As testers I think we try to root out and understand what our perceptions are and then understand whether they are reasonable, on track or in need of further investigation.

Some of this might be working against a well documented expectation or at other times not knowing what to expect - I deal with both extremes.

One way I handle this is to keep an open (and skeptical) mind and then work out what my hypothesis (interpretation according to the observation) is and compare that with the product owner's.

Sometimes there's no clear-cut right or wrong interpretation.

But it helps to keep an open mind with perception (and borrow Mr. D.A.'s hat now and then).

Have you had any problems with perception lately?

The Testing Planet has a site!

2010-08-17T14:06:00.000+02:00

I read a post from Rob Lambert the other day about The Testing Planet, about the next submission deadline, a short Q&A and how it's open for all types of contributions.

What I didn't really notice was that the testing planet has it's own site now. I didn't really see this announced - although I've been on holiday so very possible that I missed it. I think I saw a pointer too in a tweet from the @testingclub.

If you haven't seen it yet go and check out the new site for The Testing Planet. There you can look through both editions (the first being the software testing club mag from February), click through the contents list (for the 2nd edition) and even click through the different tags in both editions.

Very nice looking site and layout!

Have you seen it yet? If not I'd recommend a look.

Creative Thinking Challenge Follow-Up

2010-08-17T02:52:00.000+02:00

During the summer I read a fairly interesting book, Think! Before It's Too Late, by Edward De Bono. Brief review of the book below.

In one section on creative thinking in 'Knowledge and Information' - just after being quite negative about multiple choice exams, something I've reflected on before, (here) and (here) - he poses a couple of challenges. One of these challenges was the basis for the Creative Thinking Challenge I posed.

Go read the comments to the above challenge to see the different approaches - I give a brief view on the approaches later in this post.

My Approach
My approach was to adapt the "Gauss method" and apply it to even numbers. After experimenting a little I was able to see the pattern for even numbers. So, my approach to the problem...

First listing the two rows, one in ascending and the other in descending order:-

2 4 6 ... 298 300

300 298 296 ... 4 2

finding the pair sums (300 + 2, 298 + 4, etc) all summing to 302, noting that there are 150 such pairs and that the sum will be double the required amount, so I must divide by 2. Giving

(302/2) * 150 = 151 * 150

As for the mental arithmetic, well I took 15*15 to equal 225, then added a couple of zeros (one for each one I'd removed from 150*150 - ie divided by 100 before, so I must multiply by 100 afterwards), giving 22500, then I must add the final 150 (to get to 151*150) giving 22650.

(we all know our 15-times table right?

I, for some reason, know it's 225, but doing it long-hand in your head (?) would be 15*10 = 150 + 10*5 = 200 + 5*5 = 225)

This method is the so-called Gauss 'trick' or method.

The Gauss Method?
I saw a suggestion (on twitter I think) that this was the famous Gauss problem, a problem that he is supposed to have solved as a school child.

I found a reference that apparently shows the first known solution to this type of problem was presented by Alcuin of York; a translation from the Latin of the problem and solution are (here) - it's the one about the doves on ladders!

The Challenge Answers
Firstly, many thanks to those that took up the challenge and were willing to submit their answers! It was really interesting to see the different trains of thoughts. There were some different and inventive solutions - and that's what it's all about.

The first effort came from Trevor Wolter. I think he took the approach that I would've taken if I hadn't known the 'trick'. He wrote out a few observations, looked for a pattern, came up with a hypothesis and presented it. Right answer!

The next entry from anonymous either knew about the formula for arithmetic progressions or looked it up. He/she then presented the modified version. I assume they tried it out. There is potential for mis-interpretation in the formula though - the 'n' value must be the highest even value in the series. If I'd said add up the even numbers from 1 to 301, there would be some pre-filtering needed.

Next up was Timothy Western, who presented a simple logic case and then did the actual calculation with the aid of excel. Short and sweet - right answer!

Abe Heward gave a close rendition of the "Alcuin of York" method. Creative thinking and right answer!

It looked like the arithmetic progression angle was the basis for Stefan Kläner's answer. Right answer!

The "Alcuin" approach was in use by Ruud Cox - well demonstrated steps and the right answer!

The arithmetic progression approach was used by utahkay. Right answer!

Finally, Parimala Shankaraiah gave a detailed walk-through of the initial observations, spotting patterns, following a hunch about arithmetic progressions, looking that up, modifying the formula, trying that out and matching with the observation to present the final hypothesis. Good analysis, research and explanation. Right answer!

Summary
What's interesting about these answers is that there are different methods of approaching and solving the problem - there are no right or wrong approaches. I found this problem intriguing purely from the "creative thinking" or angle - this is something that can be practised, to add as another tool in the testing toolbox. Something I'm going to explore more.

Another good point about showing your thinking is that it's a natural step for writing bug reports. So I reckon a lot of the answerers can write bug reports with enough detail.

Book Review
The book is part complaint on the current state of thinking, part re-hash of previous work and part self-promotion. A fairly ok read but I do have quite a few niggles with it. The book has had mixed reviews - although I read it quite avidly, which is slightly unusual for me!

There is a repeating statement through the work: Good but not enough. After a while it grates a bit. I understand his point and perhaps the continued repetition is the "drilling it home" method, but I'm not always receptive to those types of approaches.

The book gives short summaries of his previous work on Lateral Thinking, Six Thinking Hats and Six Value Medals. The summaries are fairly superficial - in theory not enough for someone to pick up and use in depth, although not impossible to grasp and use at some basic level.

Another niggle I have with the book is the lack of references or bibliography. De Bono states that this is because all his ideas are his own - however he uses Gödel's theorem to state a point about perception without referencing the theorem. Some references and bibliography would be nice, otherwise it just comes across as opinions without the back-up.

There are some reasonable points made, but when they're not backed up by references to the source material the reader has reached a dead-end in this book (if he wants to dig deeper into the source material).

I haven't read any previous De Bono work - I'll probably delve into the Lateral Thinking work at some point, but for now I have some pointers.

More on some creative thinking reflections connected to this book in another post.

Getting inspired about creative thinking for testing?

Carnival of Testers #12

2010-08-01T02:29:00.000+02:00

In the northern hemisphere it's been a hot July. That's both weather and blog output!

The month started out with introspection (or was it just a talking to yourself fad?)

Rikard Edgren was asking himself questions in the corridor whilst Jon Bach was having a team meeting with himself. Two very insightful posts worth reading!

In this time of humid weather and lightning strikes it was good to see some advice on lightning talks from Selena Delesie.

Michael Larson wrote about the "Law of Raspberry Jam" and "Army of One" and what it he means to him.

Abe Heward was asking good questions related to survivorship bias of bugs and quiet evidence.

Most bloggers hit a writer's block once in a while. Rob Lambert describes how he regained his mojo, here.

Stephen Hill drew some interesting parallels to software testing after visiting an English courthouse.

In case you missed it the Software Testing Club announced the launch of the Testing Planet.

Making the release call for software, the problems and parallels with mysticism, were pondered by Curtis Stuehrenberg.

If you'd tried Ben Kelly's puzzle then you can read his reflections and what he learnt here.

Trish Khoo drew an interesting analogy between blog writing and shipping software, here.

Test case counting was a popular subject this month. James Christie wrote about his objections, several others pitched in and John Stevenson did a worthy round-up and addition to the subject.

More on the metrics angle was explored by James Bach, here.

Last, but not least this month was Markus Gärtner. He's been exploring aspects related to testing and quality with his latest post here.

Until next time...

Yet more Monty Python & Software Testing

2010-07-27T01:05:00.000+02:00

I drift in and out of Monty Python analogies for Software Testing now and then. Here's a previous reference <link>.

Whilst falling asleep the other night I remembered a Phil Kirkham post about Monty Python and my comment on it, so I thought I'd indulge myself and do a little bit of Monty Python - Software Testing analogising(?) - just for therapy :)

Here's the "Life of Brian" based comment I made:

Coordinator: Crucifixion?

Prisoner: Yes.

Coordinator: Good. Out of the door, line on the left, one cross each.

[Next prisoner]

Coordinator: Crucifixion?

Mr. Cheeky: Er, no, freedom actually.

Coordinator: What?

Mr. Cheeky: Yeah, they said I hadn't done anything and I could go and live on an island somewhere.

Coordinator: Oh I say, that's very nice. Well, off you go then.

Mr. Cheeky: No, I'm just pulling your leg, it's crucifixion really.

Coordinator: [laughing] Oh yes, very good. Well...

Mr. Cheeky: Yes I know, out of the door, one cross each, line on the left.

Or, as applied to testing...

Coordinator: Scripted test?

Tester: Yes

Coord: Good. Other there on the shelf, one scripted test case each.

[Next Tester]

Coordinator: Scripted test?

Tester: Er, no, an exploratory approach please.

Coord: What?

Tester: Yes, they said I could come and do some testing with my eyes open.

Coord: Oh, I say, that's sounds very nice. Well, off you go then.

Tester: No, I'm just joking my PM gets scared if we don't follow the script.

Coord: Oh, well in that case...

Tester: Yes I know, over there, one scripted test case each.

Life of Brian is a rich source:

Spectator I: I think it was "Blessed are the cheesemakers".

Mrs. Gregory: Aha, what's so special about the cheesemakers?

Gregory: Well, obviously it's not meant to be taken literally; it refers to any manufacturers of dairy products.

Or, as applied to testing...

Tester I: I think it was "Blessed are the certified".

Tester II: What's so special about the certified?

Tester III: Well, obviously it's not meant to be taken literally; it refers to any arbitrary label or categorisation.

There's lots of potential in the Meaning of Life too:

Three Project Managers (PM) and a management consultant (MC) discuss the state of affairs:

PM#1: Ah! Morning Perkins.

PM#2: Morning.

PM#1: What's all the trouble then?

PM#2: Test reports in disarray. During the night.

PM#1: Hm. Not nice numbers eh?

PM#2: Yes.

PM#1: How's it feel?

PM#2: Stings a bit.

PM#1: Mmm. Well it would, wouldn't it. That's quite a lot of extra information you've got there you know.

PM#1: Yes, real beauty isn't it?

All: Yes.

PM#1: Any idea how it happened?

PM#2: None at all. Complete mystery to me. Woke up just now... one piece of detailed analysis too many.

PM#1: Hallo Doc.

MC: Morning. I came as fast as I could. Is something up?

PM#1: Yes, during the night old Perkins (PM#2) had his test progress reports disrupted.

MC: Any headache, bowels all right? Well, let's have a look at this test report of yours then. [Looks at sheet] Yes... yes... yes... yes... yes... yes... well, this is nothing to worry about.

PM#2: Oh good.

MC: There's a lot of it about, probably a virus, keep warm, plenty of rest, and if you're reporting progress remember to stick to statistics.

PM#2: Oh right ho.

MC: Be as right as rain in a couple of days.

PM#2: Thanks for the reassurance, doc.

MC: Jolly good. Well, must be off.

PM#2: So it'll just sort itself out then, will it?

MC: Er... I think I'd better come clean with you about this... it's... um it's not a virus, I'm afraid. You see, a virus is what we doctors call very disruptive. So it could not possibly have made a positive impact on the quality of these reports. What we're looking for here is I think, and this is no more than an educated guess, I'd like to make that clear, is some multi-cellular life form the genu *bonus extertus*. What we management consultants, in fact, call a good tester.

All: A good tester...!!

PM#3: A good tester - on this project?

PM#1: Hm...

PM#3: A good tester on this project...?

PM#1: Ah... well he's probably escaped from a zoo.

And remember don't be complacent in testing:

A tester has been asked for his certification in testing:

Tester: "I didn't expect a kind of Spanish Inquisition."

Certification advocate: "Nobody expects the Spanish Inquisition! Our chief weapon is surprise...surprise and fear...fear and surprise.... Our two weapons are fear and surprise...and ruthless efficiency.... Our *three* weapons are fear, surprise, and ruthless efficiency...and an almost fanatical devotion to the certification syllabus.... Our *four*...no... *Amongst* our weapons.... Amongst our weaponry...are such elements as fear, surprise.... I'll come in again."

I was working on the dead parrot and test tool vendors, but it got messy - so I'll stop there!

Test Case Counting Reflections

2010-07-22T23:49:00.000+02:00

James Christie wrote an interesting piece on some issues with test case counting, here. I started writing some comments but realised it was turning into a post in itself. So, here it is...

I recognise the type of scenario that James writes about - of a project manager, stakeholder or even fellow tester being obsessed with test case statistics.

I'm even writing about this from the other side - why the non-tester (manager / stakeholder) might be interested in tc counting... But I'll leave that to another post.

I think it's a case of the test case being the least common denominator - it's an easy measure to ask for - to get a handle on "what's happening". The key for the tester, though, is to convey the value (lack of, or limited) of such a number/measure (although the first step for the tester is to understand the problems and limitations with tc counting..)

What are the figures saying, but also what pieces of the picture are they not showing?

I purposely tell my teams not to talk to me in terms of tc numbers - this is quite challenging at first - but once they understand the information I'm interested in (the feature, aspects of coverage, risk areas, limitations with the model/environment, built-in assumptions and other aspects of 'silent evidence') then it actually generates a lot more creative and searching thinking (I believe.)

How might a conversation with a stakeholder on test case statistics go? Let's do a thought example and see how it might go and problems it may hide/show.

Stake Holder: "How many TC's have been executed, and how many successfully?"

Tester: "Why?"

SH: "So I can gauge progress..."

T: "What if I said all but 1 TC had been executed successfully?"

SH: "Sounds good. What about the 1 remaining one?"

T: "That /could/ be deemed a show-stopper - a fault in installation that could corrupt data"

SH: "Ok, have we got anyone looking at it?"

T: "Yep, 2 of the best guys are working on it now."

SH: "Good"

T: "But..."

T: "It might cast a question mark over a bunch of the 'successful' TC's that were executed with this potential configuration fault"

SH: "Mmmm, what's the likelihood of that?"

T: That's something we're investigating now. We've identified some key uses cases we'd like to re-test, but it really depends on the fix and extent of the code change.

SH: Ok, thanks.

T: During our testing we also noticed some annomalies or strange behaviour that we think should be investigated further. This would mean some extra testing.

SH: Ok, that we can discuss with the other technical leads in the project.

The stakeholder has now got a (more) rounded picture of the problem/s with the product - he's also getting feedback that it's (potentially) not just as simple as fixing a bug so that the last remaining TC will work. Concerns have been raised about the tests already executed as well as the need for more investigation (read maybe more TC's) - all this without focussing on test case numbers

Not all examples will work like this, of course - but maybe it's a case of not talking about test cases, or talking about test cases and saying this is only a fraction of the story.

There is a whole different angle about getting the stakeholders to understand the limitations about units that are called test cases. One of James Bach's presentations comes to mind, here.

Got any strategies for telling the whole story rather than just the numbers?

Creative Thinking Challenge

2010-07-22T19:31:00.000+02:00

I've been reading about a form of alternative education (here) recently, and been thinking about the need for creativity (research) in my daily work. I've also been reading about approaches to thinking, and creative thinking.

I saw an example of creative thinking recently to add quickly the numbers 1 to 10, or 1 to 100. The method was quick but not necessarily intuitive - which is part of the point.

I started thinking about this to see if I could adapt the method to add the even numbers between say 1 and 300. I did the mental arithmetic fairly quickly and then checked my working with pen and paper afterwards.

There have been some interesting tester challenges doing the rounds recently, and I thought that a thinking challenge was in order.

So, the challenge is to add the even numbers (2, 4, 6,...) from 1 to 300.

Bonus points if you can do it in less than 30 seconds! Less than 10 seconds if using a calculator. (I'm not counting alternative thinking time here - that I'll leave to you to work on.)

If you want to post a comment with the answer here then you'll need to show your thinking!
(I'll delay comment publication a few days in case there's any latecomers that wants to try it - without being tempted by the comments.)

There will be a follow-up post - where I identify the book and some other things I've learnt from it.

Presenting at Iqnite Nordic 2010

2010-07-16T03:30:00.000+02:00

The Iqnite Nordic 2010 programme was just released, here.

I'm happy to be presenting "Test Reporting to Non-Testers".

It'll be primarilly an experience report with some real data and problems they have caused in their presentation and interpretation.

The presentation will build on themes around assumptions (in the report author and receiver), statistics (their use and misuse) and reading too much into the reported information.

By looking at problems that the examples have caused I'll be able to suggest some alternative approaches for testers when communicating with non-testers.

Communication with non-testers is a big interest of mine - not only do testers have to deal with modelling and assumptions of the product and stakeholders, they also have to handle the communication channels towards non-testers. This has lots areas for pitfalls (a whole different type of modelling and assumption awareness!)

Unfortunately, the mind-map I made when drafting this is huge - it needs to be printed on A3 to be readable. The result is that I'll only be touching on some areas in the presentation - but I think they're some of the key ones and worthwhile.

So there's room for a part II of the presentation!

Testing Refresher from Sanderson of Oundle

2010-07-15T04:09:00.000+02:00

Today I was "philosophising" with my 5 yr old daughter - wondering about why she thought the sun was so big if I could make a circle of it with my hand (she had a good answer) - and then we started discussing which of the moon and sun was bigger and reasons why one might "appear" bigger than the other. This generated a great number of questions and hypotheses.

This activity reminded me of "Sanderson of Oundle", I'd recently read an article and a booklet on the subject. Just the title sounds intriguing doesn't it?

Who?
F.W. Sanderson was headmaster of Oundle public school from 1892-1922 - and the story of his approach to teaching sounded both revolutionary and visionary and generated interest from pupils and parents alike.

After I read the article I searched and managed to track down a copy of a booklet that the school produced on the seventieth anniversary of Sanderson's death. It's only 24 pages but gives a great insight into a great enabler.

Sanderson said:

"We shall see what changes should come over shcools. They must be built in a large and spacious manner, the classrooms being replaced by halls or galleries, in which the children can move in the midst of abundance, and do and make research: not confined to a classroom."

"The methods will change from learning in classrooms to researching in the galleries; from learning things of the past to searching into the future; competition giving place to co-operative work."

He had two main methods:

Let there be no work which is not in some sense creative.

To him, classrooms were just tool-sharpening rooms and the real work was done in the laboratories, library, museum, art room or power station. The creative work took part in research outside of the classroom.

Let all work be co-operative rather than competitive.

He held creative work (through research) to be higher than examined work and he believed the best way to achieve this was by co-operation. His opinion was that all had areas in which they can excel and it's for the teacher to help the pupil to find those areas - not to constrain them to a norm. BTW, he didn't believe in bad students - just students that the teacher hadn't found the right angle or area of interest yet.

Why the interest to me?
I've recently "regressed" into the scientific approach - or maybe I'm a born-again scientist (mathematician actually)? I'm on a little bit of an evidence-based refresher... So I found the work that he was doing (and the way he was doing it) to be so enlightening - and with a lot of common sense.

Just take his two main methods:

Creative work:
This is so true for me as a software tester - I don't do things just because they're there or that's the way it's always been done. I try to evaluate and understand what I'm doing, the selections I'm making, the assumptions that are there (both obvious and less obvious) to be able to give a full account of my actions. That gives me the piece of mind that I'm trying to do a good job.

Take an automated test case (or test suite) - I wouldn't run it without considering the reasons for running it, what information I'm getting out of it, what it's not telling me and even considering if this selection is still relevant. If I can cover those bases - and be able to discuss them with colleagues - then I'm making a creative input into the decision around that test.

Co-operative vs Competition:
Co-operation in project environments revolves very much around communication. That's not just writing a report on time or emailing someone to say that all plans have changed. It starts before that - framing the expectations around how the communication is going to work - a sort of trust and confidence-building exercise. If X knows I'll treat all information in a professional way and be a good ball-plank for him then I'm much more likely to know about any potential changes/problems sooner.

Research and being creative is very much key to most work connected with software development and testing - and co-operation is a natural part of this.

Competition in project environments can have several connotations. Most of the "competition" whether it be one-up-manship or the CYA syndrome is usually connected with non-cooperation. Non-cooperation in the sense of a non-functioning or not-open two-way dialogue does not help the project.

Takeaways
I loved reading about Sanderson of Oundle - I wish I'd gone to a school like that and with a teacher like that.

It was also a reminder for me about the importance and relevance of creativity and cooperation in my daily work. Without creativity there is no thinking (and vice-versa) and if I'm not being creative (or thinking) about my testing then I'm not doing good testing.

Discussing testing with my colleagues (cooperation) is a basic need for good testing.

Got any Oundle-like opinions?

EuroSTAR BlogSTAR Candidate Roll-Call(?)

2010-07-14T03:17:00.000+02:00

Is the popularity/talent contest format crossing over to the testing world now? Maybe, and maybe that's a good thing. It has the potential to generate interest and involvement, publicity for the conference and a way-in and impetus for testers whose company budgets can't stretch to the conference.

EuroSTAR's search for a BlogStar sounds interesting - and worth taking a look.

Is this the next X-factor or "The Testing Blogger's got Talent"? It's got the popular vote factor - and that's a really interesting angle.

There is the potential for a great showcase! There's a lot of great bloggers - some who didn't make it to Eurostar via the VideoStar competition that I'd be really interested in seeing blogging. Note, I haven't re-freshed myself on the video entries (and maybe my memory is a bit hazy) - but I voted for the 3 mentioned below.

So here's my short wishlist for entries.

Possible Line-Up?

Rob Lambert - always got something topical to say. His video entry was slick - something about roadsigns? :)

Pradeep Soundararajan - his "joker" video gave me willies! His writing has many learning observations.

There are other Indian bloggers that could easily enter and generate a lot of interest - Parimala Shankaraiah and Ajay Balamurugadas to name but two!

Then there's the group that are regularly involved in WTANZ - Marlena Compton, Trish Khoo, Keis & Oliver Erlewein - all generating good work - with opinions aplenty!

Anne-Marie Charrett - her "don't vote for me" video was fun (or was it "yes but, no but"?) There's always a lot of intelligence in her posts.

I saw a tweet that Adam Goucher might think about entering - cool!

Peter H-L always has an interesting take on topics - I liked his suggestion about a group entry - but not sure the prize budget would stretch to that!

Markus Gärtner - he's already going but it could be fun to see an entry from him - although his blog output would probably be more than anyone else ;-) But always worth reading!

This isn't an exhaustive list of interesting test bloggers - just the ones not in my current inattentional blindness blind-spot!

And From Leftfield

The search has been promoted as looking for: "Not for the faint hearted", "with an opinion", "not being afraid if everyone doesn't like what you've got to say": Well prompted by the national origins of the eurostar team I'd say that calls for a "Father Jack" blog! Oooh, tempting...

Or maybe we should follow Peter's train of thought with a team entry - who wants to be Dougal, Ted and Mrs Doyle????

(In case folks are getting confused - these are Father Ted references.)

There's actually been many an absurd observation in "FT" that you could draw a testing angle on - "near, far away..."

Final Thoughts & Question

Interesting competition and a clever way to get people involved. Some folks might wonder about the opportunity cost involved in generating posts on another site - but just think of it as it being another sounding board for all those posts you have in prep! I know, it can be an effort though...

I do have a question about the voting though - I wonder how that will be done - and do hope it's not a simplistic poll. I mean if I get my friends and family (that are not testers) to vote for me does that really show my blogging as being more interesting to testers?

I'd be inetersted in seeing other factors in the voting: the most-read blog post, most read blogger, most comments, most retweets, etc, etc, Just thinking...

I'm tempted to enter - but have other things to sort out first. So, for me, the jury's out just now.

If I enter and somehow reach the conference I promise not to come to the gala in the persona of Father Jack! Maybe ;-)

Carnival of Testers #11

2010-07-04T01:26:00.001+02:00

June was a busy month - not only were there a range of good blog posts but the World Cup in South Africa kicked off. Many people heard about the vuvuzela for the first time - and a host of football cliches and analogies (not new for many!) I'll try to maintain part of the spirit with the odd footy cliche or analogy.

Testing is a funny old game. Enjoy!

Markus Gärtner kicked-off the month with a good write-up from XP2010 of a Lean presentation by Mary Poppendieck.

She shoots, she scores! Parimala Shankaraiah with a good retrospective, here.

Some hard work in the middle of the pitch! Jon Bach and Lisa Crispin (amongst others) tackled the subject of Diversity in Agile and Women in Agile.

A dazzling run down the wing with a nice piece on Jon Bach's observations around approaching a combination lock problem.

A long throw-in from Trish Khoo continuing the gender diversity in IT, here.

Alan Page shows it's a game of two halves with his recount on the wide wide world of testing sources.

There are no easy games from James Bach and his views on the need for challenge (and why he does it) in the testing culture.

If you missed it the first time then the replay has plenty of highlights from James Christie's call for testers to challenge the culture.

A great advert for the game from John Stevenson and his account of a training session in India.

Andréas Prins was strong on paper with his post on his learning and sources around test profession-related learning.

With a few minutes of the game left Maura van der Linden gave a great account on her experience with transpection.

A strong challenge from Abe Heward and his assertion that good testers are not robots.

A late corner from Michele Smith with a testing question on a "bug free" claim.

Gatecrashing the top 4 was Simon Morley and his thinking about showing his thinking post.

Zeger Van Hese's ran his socks off with his perspective on collateral product features turns up.

Into extra-time and illuminating perspectives and observations on a WTANZ session from Keis. What a lovely finish!

Until next time, enjoy!

Showing your thinking or thought-process

2010-06-25T06:05:00.000+02:00

A chunk of this post is based on a comment I posted on James Bach's post on challenging colleagues, here. You've probably read it, but if not I'd recommend it - the post not my comment :)

The Challenge
Challenging colleagues is an interesting topic. I personally enjoy being challenged - although maybe not a constant deluge so that I don't get time to think (but that's probably just a symptom of a different problem.)

How I respond to challenges is very much a part of me - responding to challenges is intrinsically a personal thing - everyone does it differently. Some people might not want to respond, but that just means it takes time for the response to emerge, develop or evolve - not everyone (myself included) can always respond quickly. For me, "let me think about that and come back to you" is sometimes a necessary "breather".

Maybe somebody responds in brutal Anglo-Saxon ;-) But that's probably a symptom of something else or maybe a undiplomatic way of saying "I don't want to respond to that right now."

But, I think being challenged and taking up challenges is a necessary part of being a thinking tester.

Analysis
Suppose you're trying to describe a problem to a colleague (or even just discussing a topic) and the "why" and "what do you mean by that" questions come up. It forces you to go deeper or re-assess your approach (either you haven't got to the right level of detail/background yet or your approach is making an assumption about how the other person thinks/responds...)

This is good - it forces you to think, analyse what your own pre-conceptions are and understand more detail of what you're trying to say. This self-evaluation is key. At least I think it's a key part for any thinking tester.

For any thinking tester this is necessary to help their own learning and understanding about a topic - those are the tools that will help them ask better questions and even look for assumptions that others are making (whether it's about a product feature, claim or bug.)

So, challenges per se are a "must" or a /good/ thing. Of course, how people respond to them is a more personal thing - sometimes it's instant and sometimes they need their own space, go away and think about it. Time to digest, ruminate, cogitate and then respond.

But it's the thinking that's important - and even showing your thinking.

Showing Your Thinking
During my "maths days" then "showing your working" was actually quite important - it was an insight into your assumptions and map-making to come to a solution, idea, hypothesis, conjecture etc. It was helpful to the lecturers - as they could highlight traps, wrong-turns and shaky assumptions. Their job was to help you think through the problem and even bounce ideas back and forth. So showing your working was just a way for the lecturer to get up to speed (if need be) to your thinking - and that's a good thing most of the time.

I try to re-use that idea - not only because it can help others understand my point or question - but I might re-visit the idea sometime later and then it's a useful reminder about what my own assumptions were "at the time" - they might not hold true longer or been forgotten (very possible!)

Plus it helps to stop things getting lost in translation - even when you have the same language! You might speak the same language but a different dialect or sub-language. Do you feel me?

I enjoy reading testing blog posts and comments. Sometimes that's the trigger needed to kick-start my own thinking around a topic - or usually a certain angle of it. Some of my best creative and instant thinking comes whilst answering questions and even posting comments on other people's blog - time to go fish for those comments...

So, I definitely think challenges are a good thing. But, responding to challenges and "showing your thinking" are even better!

Do you show your thinking?

Carnival of Testers #10

2010-05-31T22:50:00.000+02:00

Another month, another wave of testing blog posts. What can I say? Variety is spice of the testing blog life! And it's edition number 10!

Challenges

For people familiar (or not) with smurfs then have a look at the challenge by Markus Gärtner. The write-up is worth a look too.
Lannette Creamer went all trekky with a testing challenge, here. It generated a fair bit of interest - but I think I was the only one to use the microwave oven heuristic.... An insightful summary and reflection popped up too!

Bites

Curiosity: Lynn McKee wrote a good summary of Gil Broza's interview with Michael Bolton. There's also a link to the audio.
Comprehensive: As a first blog post Felipe Knorr Kuhn started with driving Selenium from Java, here.
Animal Welfare: Frog eating was the eye-catching hook from Anuj Magazine about getting tasks done - even the unpleasant ones.
Trans-what? A transcript and analysis of a transpection between Michael Bolton and James Bach on expected results and inputs makes worthwhile reading.
Bug Rebellion: Andy Glover gave a cartoon view on bugs being reproducible - or was it a case of the audience not agreeing with the message at a conference? Mmm, you decide...
Model Count: How many models do you use in testing? Don't know? Interested in finding out more? Then take a look at Rikard Edgren's take, here.
Automation for bug finding or for testing or something completely different? Dorothy Graham presents her view for testing, here.
Automated Exploratory Testing? Although not strictly a blog post Rob Lambert started an interesting discussion (different viewpoints) on the STC, here.

On Writing

Marlena Compton wrote about the positive contribution made to testing by writing.
The writing about testing peer conference got a rounded summary from Chris McMahon.
A forthcoming "Ask The Tester" article by Matt Heusser was given the heads-up, here. It's your chance to send in questions so take a look!

The C-word

Keyboards have been getting warm recently. There's been a range of views and discussion circulating around certification.

I will say nothing more than to direct you to some of the contributor's posts - some with a lot of discussion in the comments: James Bach, Trish Khoo, Dave Whalen, Cem Kaner, Nathalie de Vries, Simon Morley & Stuart Taylor.

Until next time, happy reading!

Tester Certification - my take...

2010-05-18T23:46:00.000+02:00

Recently, there's been a bit of a commotion, kerfuffle, a minor hullaballoo - Well, when I say recently, it's been going on a little while - ebb and flow, coming and going, just like a pleasant memory, bad smell, nightmare, whatever - take your pick!

I started this post over a year ago - but after a post from James Bach, here, I was prompted to finish this post. So, here's my take.

Before
Ambivalence!
Yes, that was my feeling towards certification before I encountered the ISTQB.... I'd heard of it, even been asked about it in interviews - I didn't have it and saw no reason for one. It hadn't affected my employability (as far as I know.)

It was about 2 years ago when my shop embarked on a programme of certification - I don't know the reasoning behind it - it was policy and I had more important things to think about. So folk started getting booked on an ISTQB foundation level course with an exam at the end.

The syllabus seemed interesting at first glance - another take on items and approaches I'd already worked with - I'm always open to new avenues of learning. Maybe a standardized terminology so that all could understand terms - whether new tester or old in the tooth. That's gotta be good, right? That was a hope/wish for me after a skim of the syllabus.

During
The actual course was interesting enough - but it was clear that it was geared to passing a multiple choice exam.

There were aspects of the course that didn't quite sit right for me - definitions and terminology - I'd been a tester for a good while at this point and now I was being told to unthink some ideas - even though I could argue the case for using my definition - that wasn't the point.

The point was to pass a multiple choice exam - then you could think how you wanted.

Hmm, ok, paradise lost! Is the idea of standard terminology is just boiling down to an exam?

The idea of standard terminology maybe wasn't such a utopia - with a relatively easy exam these terms as well as the certificate would just be bandied around - just another keyword/buzzword to put on a CV.

After
To me that doesn't do a tester any good to only use standard terminology. If they think that testing is about using the right terminology instead of thinking what they're talking about - and even expressing it in alternate ways - then they're missing the point. That's sad!

The tester has to be able to use the local terminology and definitions if needed.

I've worked against a certain amount of "test management"-speak in the past. I think it's great that testers can get up and articulate their ideas to both their peers and managers. For this I think a certain amount of terminology can get in the way. And that's a shame!

When I talk to non-testers (managers) I go to some lengths to emphasize the difference between testing and good testing - and for me good testing starts with yourself - expressing your ideas and making yourself heard and understood - sometimes putting ideas in simple language. And that's important!

Ambivalence!
Yes, that was my feeling after the coure.

Now
There's still a fair bit of ambivalence for me. I haven't benefitted from the certification.

I used to think the course could be a useful intro for a new tester. Now, I think there are better ways - routes for self-learning, self-practice and mentoring. These are much more effective - and ultimately gratifying.

I don't want to reside in a swell of buzzwords - I think it's time to talk about what you value as a tester.

Got any good or bad experiences of certification?

Testing Style: "Eyes Wide Shut" vs "Eyes Wide Open"

2010-05-02T17:01:00.000+02:00

#softwaretesting #qa #testing ;

What's your approach to testing?

What first impressions do you get from "eyes wide shut" and "eyes wide open" as applied to testing?

Let me do a quick mind-dump, word association thing - the order below is exactly how it turned out - I then went back to add a bit of detail between the bullet-points. You'll have to excuse the odd Monty Python moment...

Eyes Wide Shut

Parrot

I suspect I was primed by the recent video interview with James Bach about bucanneer testing and parrots.

Rote

Following word-for-word - no deviation.

Following and willing to be led. Whilst this may be good and necessary in some test activities, I think this is in the minority of cases.

Sheep

Probably primed by Rob Lambert's recent agricultural reference (actually a serious post!)

Blind

Not looking around.

Sensory deprived

Not using all information inputs.

No questions

Just follow the guide/script/instructions even if it doesn't make sense.

Lemming

Can lead to the black swan event of all "eyes wide shut" testers.

Same-old same-old

No change, never change, comfort zone, safe. Perfect candidate for automation.

Eyes Wide Open

Searching

Using the script/test idea as a guide - an incomplete map.

Alert
Observant
All senses in use

These three are very similar - when the tester is switched on and ready to react, think ahead and even pre-empt.

In the zone

When all senses are engaged and the tester is questioning the product it's similar to the sporting equivalent of being "in the zone".

Feedback

The tester is questioning to give good feedback. Nobody expects the Spanish Inquisition. Questioning is our main tool, questioning and feedback, ohh...

Exploratory

Map-making in real time!

Investigative

The best of journalistic and scientific investigation - following-up on previous experience, hints and clues with the information in front of you.

Ideas

The activity generates new ideas - either for further testing, how to solve test problems, feedback on the product or how to improve the product.

Note, I don't consider this as scripted vs non-scripted. I consider it more "good testing" vs "not-good testing" (I place no other emphasis on it than that...)

There are some cases where scripted testing is necessary (even a legal requirement) and others where it's needed as a guide.

My take on buccaneer testing and parrots - they are different species and unless the parrot is supposed to do everything-by-rote then it's the "eyes wide shut" case. I've got another take that I'll expand on in another post.

Of course, if the parrot is a Norwegian Blue then it definitely ain't dead and it's actually very good testing! :-)

Do your own word association and see what you come up with!

Carnival of Testers #9

2010-05-01T12:11:00.001+02:00

April was an active month blog-wise on a number of fronts. There was some conference activity, some planned and some spontaneous. Spontaneity was also seen on the testing front with the much-tweeted #parkcalc.

A real variety from new announcements, some detailed analysis and other thought- and discussion-provoking posts - exactly how it should be!

Enjoy!

Charting

Anne-Marie Charrett wrote a piece on combining mindmaps and mnemonics. She even makes her mindmap available if you want to download it. Anne-Marie also compiled a list of the testing tips from the first Tag Tuesday #dttip.
Reinder Otter posted a slideshow on Self-Organisation. If you're into self-organizing teams then take a look.

Confering
ACCU2010 was hit by the flight restrictions over Europe from the Islandic volcano - I won't even try and spell it (the one beginning with E and followed by lots of vowels.) Some people made, some didn't and some had to make alternative arrangements.

Peter Haworth-Langford wrote about his session with James Bach.
The Friday got a good summary from Thomas Ponnet's viewpoint, here.
Lisa Crispin's flight didn't make it over but she wrote about the other opportunities that she took. There's always a silver lining to every volcanic cloud!
Possibly the post that generated the most interest from the ACCU arena was Robert Martin's take on software testing and professionalism. This not only generated lots of comments, tweets but quite a few blog posts.

ST&P announced their dates and call for participation at STP Con 2010.

Matt Heusser presented the case for attending as an opportunity, here. Will you answer the call? I'm tempted!

The UK Test Managers' Forum also got represented in the blogosphere.

Stephen Hill's rounded summary of the day was posted, here.

The end of April saw Star East 2010.

Yvette Francino produced a write-up on the conference with a video interview with James Bach, here. Have you ever been fired by all the right people?

Community
A new weekend testing started in April - we saw the first WTANZ session. Very cool!

The first blog announcement came from Marlena Compton.
Write-ups of the first session were provided by Oliver Erlewein, here, and Keis, here.

Clouds
Volcanic Ash from Iceland prompted a few blog posts in April.

Zeger Van Hese drew parallels between volcanic activity and systems thinking.
Santosh Shukla reflected on the opportunities of being stuck in London due to the ash cloud. Another silver lining fellow!
The open space Open Volcano 10 was initiated due to folks being stranded. Rob Lambert put down his reflections on the day.

Clouds of the volcanic kind were not the only topics for consideration. Cloud computing got a few mentions in April - more on that in a follow-up!

Calculations
The ParkCalc subject generated a lot of activity - blog posts, tweets and a follow-up debrief. Great ad hoc testing opportunism!

Matt Heusser got the ball rolling by drawing attention to it as a testing challenge, here. A surefire way to grab attention!
Selena Delesie posted the first response, with lots of good observations.
Other takes came in from Keis, here, and the European Weektesters, here.
Jon Bach ran the de-brief and Matt did the write-up.

There were many more interesting posts that I didn't write about now. They haven't been forgotten and will caught in another post looking at longer term trends...

Until next time happy reading!

Assumption: Mother of all Flip-Ups!

2010-04-28T23:19:00.000+02:00

#qa #softwaretesting #testing

One of my current interests is to do with communication between testers and non-testers.

I recently heard some gems from non-testers, but there's always two sides to the story...

Assumption #1: "It's a known problem!"
Admittedly, I think I've probably said this in the past. However, my "rose-tinted memory" remembers me putting this into some context - it was in this situation, or it's known by X in your team.

But that's the problem with assumptions, it's such an easy trap, an easy place to fall over. So after a while an assumption is almost a known starting point - "oh, we've seen that so many times by now everybody must know about that."

But this is a great illusion. Just because one or two people have seen this (maybe they're in some fault report triage group) doesn't mean that everyone else has seen it.

When I dug into this particular "known problem", it was hidden in a email and referenced in an old report - neither were discoverable. So I make the distinction between a problem that is "known to some group of people" and a problem that "is discoverable by everyone".

If I now use the term "known problem" then I'm careful to attach which group of people I think it's known to. If I don't attach such a group (or context) then I assume it's discoverable...

Assumption #2: Troubleshooting - Best when the designer and tester are working side-by-side
The implication here was that they should be located near each other, that not talking face-to-face would not work.

I think this is a bias problem - either someone has experienced "one" way in which this can work. Alternatively, they've seen problems when not communicating face-to-face and assume that's the reason.

It takes maybe one or two additional routines when working remotely from a partner, but there's no reason why it can't happen.

I remember one time being located on a customer site 9 time zones from the designer that was helping me troubleshoot. We sorted our routines and structure out - we optimized the short time we were able to talk each day and the pieces fell into place. So I have no reason to think that it can't work - sure, it's more difficult, but certainly not impossible.

Assumption #3: Troubleshooting - Best when the designer can drive the activity (ie do the test) themselves
This assumption "seems" to take the view that the tester is a robot, that the feedback is exactly what the designer requests.

Maybe some testers work like this but the majority that are worth their salt or want to learn are constantly adding to the equation - making the some of the two (designer + tester) greater than the individual parts!

This assumption may be another bias problem - maybe a bad example that stuck in someone's head.

Overcoming this type of bias is a bit tricker - especially when acting as a third party. It's usually up to the tester to build their brand, but I'm still working on how to approach this problem as a third party!

If you've any assumptions that you've heard recently then drop a comment...

Remember: Assumption is the mother of all fudge-ups! (or something like that!)