Indicators, Testing & Wine Tasting

The other day I was discussing a problem with a developer. Essentially, it was about how to judge feedback from a continuous integration system next to the other information that the testers in the team were producing. The reason I'd started asking questions was that with the myriad of information available there was a risk (as I saw it) to ignore some information - or cherry-pick the information to use...

Indicators

This led me to describe how the different pieces of information contributed to the picture of the software being developed, and as such they were indicators.

Some people confuse test results for absolute truth. "It worked", "it passed", "it x'ed" - all past tenses.... Getting from a past result to a future prediction of performance isn't easy - unless you're demonstrating on the customers equipment or some identical configuration.

Test results can be important markers or references - especially if a customer thinks of one from a demonstration - then there is an implicit expectation made (assuming the customer was happy with the demonstration) - that particular test case could be thought of as part of an acceptance criteria.

A test result can't be understood in a meaningful way if it is taken out of context and many results from automated suites or continuous feedback systems are inevitably context-reduced. That doesn't mean they're not useable, but for me that means they are indicators of current performance and future likelihood of behaviour. Also, although automated set of tests can be very powerful - they also leave out information (or people interpretting the results leave out information) - the so called silent evidence of testing, ref [1].

How we report and discuss these results adds the context and makes them context-specific. Getting from a test result (or results) to "it works" or "it meets customer expectations" is not as simple a task as a stakeholder (or developer) might wish. More on context-free reporting in another post...

Thought experiment - illustration

Of all the possible tests that could be executed on this system, I have a set (no matter how comprehensive) that I think of good-enough. Now suppose one (1) test failed - and only one test. Would you:

• Report the problem (or sit with a developer to localize the problem),
• Wonder what other potential problems might be connected that haven't been observed,
• Wonder if the information from the failed test is sufficient (extra testing or extra tests),
• Wonder what information this failed test is saying in context of other/previous testing done (or ongoing),
• Investigate the significance of the test that failed - maybe even see if there was any connection to recent changes in the system (under test), environment, test framework or other parameterization.

Note #1, hopefully you chose all of the above!
Note #2, this problem expands if you ever have a subset of tests (say an automated set of tests used as a sanity check) and 1 test (or even X tests) in your sample (subset) fail.

The example here is to show that a result doesn't stand on its own - without some situational context. Sometimes the investigation around the problem is about adding that situational context so that you (or someone else) can make a judgement about it.

Flaky Feedback?

One of the problems for the team was a mistrust with the feedback they were getting from the continuous integration system. There had been a glitch in the previous week with the environment which had thrown up some warning signs.

But,

• Warning signs are exactly that - they're not judgements - they are "items for attention"and always need more analysis.
• An environmental issue is good feedback - oops, will this work in production, rather than just on our machines?
• Why only mistrust bad results or warning signs? Doesn't the same logic apply to "good" results or "lack of warning signs"?

Mistrust can be an excuse to not analyse or when overloaded it's easy to say we'll down-prioritize analysis. Mistrust can also result from a system that becomes too flaky or unreliable - in which case you need to consider (1) can I use that system, (2) what should I use to get the information I need, (3) do I need external help?

Another question for these team set-ups (especially as/where continuous integration is being introduced) - is the team itself ready for this feedback? I'll explore this in another post.

Wine Tasting

After I commented on the different sources of information, how they all added to the big picture (the aspect of adding context to a result) and that our knowledge of that "big picture" was constantly changing the remark came back, "this is more like wine tasting than software engineering.."

In a way I couldn't agree more. There are a lot of similarities between "good" software testing and an expert wine taster - they both require skill, constant learning and practice. Descriptions are both objective and subjective - knowing how to distinguish and give the right tone to which is tricky. Both are providing a service. Neither the software tester nor the wine taster is the stakeholder (end user or their sole representative).

But this raises an interesting question - in a world of multitudes of information (and nuances in that information):

• Is the team ready for this? The sheer amount of information and strategies in tackling or prioritizing the information. Topic for a different post...
• Is this a simplistic view of testing (and even software engineering)? Testing is not a true/false, go/no-go or back/white result, it is not a criteria or quality gate (in itself).

Software Engineering?

I think we can be misled by this term. When one thinks of engineering it might be in terms of designing and constructing machines, bridges or buildings. The word 'engineering' has primed us to think of associations to engineering problems, many of which have an element of precision in them - often detailed blueprints and plans. But not always....

I occasionally watch a program, Grand Designs, that follows people building their own homes - whether via architects and sub-contractors or totally alone. A common factor of all these builds are that they are unique (even where a blueprint exists) as usually some problem occurs on the way, (1) a specific material can't be obtained in time so a replacement needs to be sought, (2) money runs out during the project so elements are cut or reduced, (3) the customer changes their mind about something (changing requirements), (4) some authority/bureaucracy is slower than hoped for, delaying or changing the project  - so very little is completely predictable or goes to plan. A common factor: where there is human involvement/interaction then plans change!

Software - it's conception, development, use and maintenance is inherently a "knowledge-based" activity. The testing part of it is inevitably entwined with eliciting and making visible assumptions about it's use and purpose as well as giving the risks associated with the information uncovered, investigated or not touched. So, I'd like people to get away from the idea (frame) that software engineering solely as a blueprint, planned, right/wrong or black/white activity.

Using the term "software engineering" is fine - but put it in context: "software engineering is software development with social interactions (that may have a unpredictable tendency to change)".

And finally...

• Don't assume bad results are not useful or useable.
• Don't assume good results tell the whole story.
• Product Risk tends to increase where analysis of results and their context doesn't happen.

References

[1] The Tester's Headache: Silent Evidence in Testing

What's Your Testing Motto?

Do you have a motto in your testing work?

What's a motto?
Do you have a general approach to your work? Maybe it's an attitude or general starting position. Maybe it's something that sums up your team approach, your problem approach, the approach to test issues - it could be a separate approach for each or a single approach that works on many different levels.

When I first started as a trainee function tester (that was the job title) one of my first team leaders said to me, "Anyone will help you, but you will have to do the work."

Personal Motto
I liked that, adopted it and personalised it. I reframed it for my use as, "I'll help with anything, but it's you who needs to do the work."

This fits into the teamwork approach grouping of mottos, or phrases that sum up my approach.

On the face of it this could sound negative. I never interpretted it that way when first hearing it and have never meant it that way when using it. To me it is an enabler for teamwork. If used within a team it means that everyone supports everybody else within the team and also that everybody contributes to the team. So, from that perspective it's very inclusive.

I don't remember the last time I used the phrase (maybe 2-3 years ago), but everybody that works with me (including managers) knows about it, buys into it and I occasionally hear it used in front of me - so it's an idea that is easy to adopt and spread.

People like and see the value in it, whether it's called a motto, a phrase or an attitude.

Other Mottos?
Can I find a phrase that sums up my attitude/approach to problem solving and test approaches? Mmm, let's see:

Problem solving: I'm a divergent thinker and emergent learner, so I think my attitude has got to be broad coverage, initially shallow and follow-up on key areas.

Test approach: This is a combined top-down and bottom-up approach, trying to understand the big picture as well as digging into the details (a typical answer from a divergent thinker and emergent learner!)

Note, it doesn't mean that these approaches always work for me - but they are starting points.

Why bother?
You could ask the question, "why bother trying to sum up my testing approach?" Maybe you have a list of things that you could categorize that you use, something taking a page or chapter to discuss.

Well think of the example of twitter. Sometimes when trying to get a message across (in 140 characters) you need to re-think what you want to say, cut out the noise and try to distill the message.

It doesn't always work - sometimes you remove part of the message/meaning as well as the noise. But, try it as an exercise - what do you do, why and can you describe it? It's a very powerful exercise.

Do you have any mottos?

Can your testing approach or attitude be summed up in a few words? In your own words, or somebody elses...

Expert-itis, Diagnosis and Treatment!

#softwaretesting

I started threading a few thoughts together before my holiday – more as a cautionary tale. However, after the recent outbreak of plagues I thought I’d give it a medical slant….

Who knows, I might start a whole thread of testing pests, diagnosis and treatment!

Description

This is a condition affecting testers and test leaders, commonly in projects/iterations experiencing stress in timescales, build releases or test execution completion. The inducers of these affects come from outside the test team/organization.

This is the occurence of test experts that are not practicing testers that want to dictate/direct the testing effort.

Diagnosis

A non-tester (typically PM or developer) expresses their opinions (sometimes forcefully) about how the test execution (and even design and follow-up) should progress.

Occasionally, this is done in an oppressive manner meaning that they are dictating the testing effort – micro-managing (instead of the test leader).

Treatment

PM’s have the “most” right to say how something should be done. They’re responsible for the completion of the project/phase, including all parts within it.

Do not antagonize or dismiss opinions. Take the opinions on board – within the scope of your remit. Developers have very valuable input into aspects of the testing effort – but you’re the one responsible for the test effort - your team may have a joint dev/test responsibility in some areas and separated in others.

If you can’t reconcile differences with a developer over how an aspect of the testing should progress then discuss with the team leader, test leader (if there is one) and PM, if needed.

If you can’t reconcile differences with the PM then you need to discuss divisions of responsibility – where the test responsibility ends – include the line boss, if needed, to get things clarified.

If you’re unfortunate enough to end up in this situation then think out your game plan first – motivation, problems to your work, problems this causes for the team and project, suggested way of working (emphasis on the team and project benefits) – before discussing with PM or line responsible.

If you can, bring in a real test expert to help put over your case. A real test expert, in this context, being a tester in your organisation that all sides will listen to.

Outlook

Where the cause is an "old-style" PM (not rating the tester very highly) then the outlook can be quite poor. Perseverance is the key here, but usually this type of PM is not going to change very quickly.

Occurences of testing expertitis have reduced in recent years, partly due (but not exclusive) to:

Testers have a weighter role in development projects – they are not the second class citizens that they might have been 10 years ago.

Many organizations are more “en-lightened” these days – open to many different development and testing approaches. This isn't just a nicety; the projects have to be efficient - meaning everyone is working from the same storyboard.

Many projects/iterations work in more cohesive units meaning there is less of the blame game.

Other Comments

Although I haven’t seen this for a few years – the ingredients for it to occur are not so diverse. A PM from the old-school /pre-enlightenment – they still exist – is probably the main catalyst for this type of event.

NB. Comments, observations and input from non-testers are valuable and should always be appreciated – it’s where the dynamic turns into a directive (that doesn’t come from or with the ok of the team/test leader) that it can be a problem.

Do you suffer from testing expertitis? Do you have any other testing affliction?

Are you a divergent thinker?

#softwaretesting #teamwork

Almost sounds unpleasant doesn't it? Something that was frowned upon in Victorian times? Well, I am a divergent thinker, and more...

Problem solving and learning fascinates me - especially the interaction with teams. The how and why people learn in different ways and how they approach different problems. Luckily, sitting in a software test engineering environment, I get to think about this, put it into practice and continue to learn and develop daily.

Whenever I sit down with someone to look at a problem - whether it's brainstorming, bouncing some ideas around or to help someone out - I'm intrigued by the different approaches and paths people take to tackle a problem.

Problem solving is like going on a journey - lots of different ways to do it (paths to take) - hopefully arriving at the same destination (the problem in question is solved.) This has a very close analogy to the way people think and learn - lots of different ways to do it, to hopefully achieve the same result.

So, what am I?

Many years ago I was involved in a self-managed learning program. Out of that my instructor and myself came to the conclusion that I was a divergent thinker. Okayyyy I thought...

Is that all?

No, we also concluded that I was an emergent (constructive) learner. Not bad, but what does all this mean?

Testing relevance?

Some key attributes are:

Emergent Learners

- Building knowledge from the bottom-up

- Diving-in to learn from experience rather than from a book

- Evolutionary - adapting to change and being self-organising

- More DIY-learning...

Divergent Thinkers

- Brain-storming

- Creativity

- Looking at the problem from several angles -> "Big picture" & "out of the box

Sounds great to have these qualities in a test team.

Of course, there are drawbacks with these. Emergent learners may have holes in their knowledge as they haven't gone via the classroom - but I think the ability to adapt and learn makes-up for this.

This doesn't mean that one type of learning/approach is better than another - as with any team, it needs a healthy mix of all types - then everyone is backing each other up... That includes mixing high-flyers and less-able members - it's the team that counts, not the team hero.

Similarly, I wouldn't want a team of only brainstormers (or even brain showers!) - all that whiteboard and post-it activity - they'd never do anything!

In practice, is it true? Has it worked?

In early 2005 when I joined an organisation as a system integrator - with part of the role being to establish a system integration process and organisation - I was the only tester not to be sent on a training course for the HW/platform we were using. My manager's explanation: "we're time stretched and you're the sort of person that will learn more without the course..."

Well, I didn't completely agree - it was like a bit of tightrope walking without a safety net - but I pulled on my waders (or was it diving suit) and jumped in at the deep end.

Horrible mix of metaphors!

True enough, I adapted and evolved both a strategy and process - this attitude fitted nicely as the org was an incremental development project in start-up, so there was plenty of change, modification and adaption.

Does this mean these traits are more suited to incremental development? Yes, they're good to have but again I think it's the healthy mix of competences that makes the project work. Everyone has elements of all learning styles - just that some are more practiced than others...

Is that all then Ted?

Recognising and understanding the differences is what's important. As a team leader it's very handy to know (or guesstimate) these styles in different team members because adapting the presentation to the individual wins every time! Also tasks can be shuffled around to match strengths, or even exercise/improve weaker areas.

As an aside, if you're interested in mind-mapping (will appeal more to the divergent thinkers) there's a lot of good software tools out there. One I've been playing with recently is bubbl. Quite nice.

Don't confuse emergent learning with emergence - this is another term I'm interested in - more to do with how teams can self-organise...

Blue-sky thinking with Social Media (part 1)

The spread of communication in test teams has long been essential in the smooth functioning of the activity. This may be to distribute activities, test campaigns, troubleshooting, collating progress reports, latest known problems, etc, etc.

With the move towards increased use of social media there are many opportunities for improved effectiveness in the team structures and dynamics.

Information exchange via informal and more formal networks in test groups has always existed.

• Formal: Due to the project/team/unit organisation forming a semi-rigid reporting/information chain.
• Informal: For example whiteboards, emails and wiki pages.

With the increased use of social media there are many opportunities that test engineers (whether in teams, intra-team or management) can grasp to aid communication exchange.

Increased effectiveness will probably come from a hybrid use of the formal and informal channels.

In this team working context the social media could be wikis, webpages, some collaborative software, amongst others. Many of these have existed for some time but are quite often just used as a dumping ground for information - libraries, records, how-tos etc.

So, with all these different ways of communicating what do we do? Let's do a bit of blue-sky thinking...

Dynamism?

A more fluid/dynamic team set-up? Something more evolutionary? This is an idea that will probably scare the socks off project managers used to more rigid team set-ups.

Suppose team A & B are working on two different areas. During the course of their execution they hit a problem (affecting both teams). Traditionally, there are several ways this can be approached - a sub-team (from both teams) is set-up to trouble-shoot, one team focusses on the problem whilst the other works around it or the troubleshooting is handed over to someone/team external to A or B.

Well, suppose the trouble-shooting group formed dynamically. This happens either via IM, wiki post, blog, email etc - certain team members (spanning the two teams) decide that the problem should be worked on jointly.

The idea here is that the network exists (the pool of test engineers) and forms a team dynamically (eg two people discover they're stopped by the same problem and join forces, even calling in external help) - this joining of forces aids the problem solving (in most cases)

This type of team dynamic probably needs to be ok'd/coordinated - but as the teams/network gets more and more used to it then the team formation should become more "natural" or "self-selecting".

This dynamism is meant to only create temporary teams for as long as they are needed!

Another example of dynamic team formation is a group forming from a given network to do some brainstorming on improving a process - the interested parties initiate the activity, setting up the group dynamically and kick-starting the work.

Tools for all of this - to help facilitate it?

I think some of best tools are the collaborative SW tools (project web pages with directive, status, discussion boards and team calendars), linked in with blogs (eg for thoughts/opinions on what went good/bad in a project), wikis & webpages (resources for how-tos) and email (ad-hoc questions/requests or inspiration).

When I say evolutionary I don't mean that the "weak" are left behind. This activity has to be inclusive - adoption and usage rates will always vary, and there is no place for exclusion.

Dynamism continued?

What about fluid team structures dictated by the problems in front of them? This is about moving towards collective intelligence! More of that later.

Other benefits & problems with social media and team dynamics in a later post.