Communication, Paradigms and Interactional Expertise

Over several years I have been looking at communication involving testing, here and here are examples from 2010,  and it never ceases to amuse (or concern) me how little we communicate well; then come the knots we tie ourselves up in to cope with this lack of information and content (and indeed communication, discussion and relation to context).

Occasionally the problem is brevity - information (context) is omitted. Other times it is a different paradigm - a different way of thinking about contexts. Sometimes it is a framing problem - not realising the background argument (or concern) someone has.

I have constant contact with senior managers and I know that they are always interested in content and context - what does it (some piece of information) mean in our product's and company's situation? A stakeholder or manager wants to know how the information you have found relates to their business context.

The last thing I want from a manager or stakeholder is an appreciation of "no added value" due to brevity, lack of content or context or because I don't engage their language or understanding of the situation. Something alone the lines of the Billy Madison clip:

I will look at some issues with stripped out information (brevity) in a following post. In this one let's take a look at the differing views and paradigms that can make us stumble or not notice how communication is failing.

Paradigms & Different Views
Paradigms - these are where different people hold different world views of something, for example what testing is, it's place in product development and what it can, can't and should do. We have probably met people with very different views and appreciation of the work we each do and where it fits into the bigger scheme of things.

There are a few ways to approach this problem. One is to educate people into your way of thinking. This might be the optimal goal, but it isn't always realistic - especially with senior managers. It's not always possible if someone doesn't understand that there might be a different way to look at the topic.

Another approach is to find common ground - a trading zone for language.

Consider the case with three actors: How do three people communicate, one with an ISTQB-view of "words", another with a non-ISTQB view of "words" and a stakeholder (business leader) with his own understanding / view of words? Do you convert everyone to one view, or do you look for an alternative?

To talk to a business manager about testing do you insist he must know all the terminology you wish to use, or do you look for an alternative?

In 2007 Collins, Evans & Gorman [1] explored the idea of trading zones and interactional expertise. On trading zones:

"Peter Galison introduced the term `trading zone’ to the social studies of science. His purpose was to resolve the problem of incommensurability between Kuhnian paradigms: How do scientists communicate if paradigms are incommensurable?"

I.e. how do the three actors communicate with each other in a reasonable way when they each operate in different views of the world (paradigms)? Hence the concept of a trading zone for language.

"Two groups can agree on rules of exchange even if they ascribe utterly different signifcance to the objects being exchanged; they may even disagree on the meaning of the exchange process itself. Nonetheless, the trading partners can hammer out a local coordination, despite vast global differences."

And one important form of these trading zones is interactional expertise.

"Interactional expertise is the product of a successful linguistic socialization. Although expressed as language alone, it cannot be too heavily stressed, interactional expertise is tacit knowledge-laden and context specific. "

Interactional Expertise
Applying Collins’ view [2] of interactional expertise from the sociologist-scientist relation to a tester/developer-stakeholder relation [italics and strikethrough are my application]

“… where interactional expertise is being acquired, there will be a progression from “interview” to “discussion” to “conversation” as more and more of the science [business/stakeholder context] is understood.” 

“ Above all, with interactional expertise, conversation about technical matters has a normal lively tone and neither party is bored. As things develop the day may arrive when, in response to a technical query, a respondent [stakeholder] will reply “I had not thought about that,” and pause before providing an answer to the sociologist’s [developer/tester’s] technical question. When this stage is reached, respondents will start to be happy to talk about the practice of their science [business context] and even give studied consideration to critical comments. Eventually respondents [stakeholders] will become interested in what the analyst [developer/tester] knows about the field because he or she will be able to convey the scientific [business context] thoughts and activities of others in a useful way. ”

“Where there is no developing interactional expertise […] the conversations never become interesting to either party, the analyst [developer/tester] can never transmit information, take a devil’s advocate position or, crucially, distinguish between “pat” answers and real conversational interchange, nor between jokes and irony on the one hand and serious responses on the other. Worse still, though a field might be riven with controversy […] the analyst cannot understand what the protagonists are disagreeing about, nor how deep the disagreements run, nor, with any certainty, who disagrees with whom! ”

Note, this relationship goes both ways - stakeholder <-> developer/tester. Over time the stakeholders / business leaders worth their salt will develop their interactional expertise to talk with the product developers / testers.

Importance of these concepts in your daily work
Some important points to highlight:

• Acknowledging (or being aware) that language trading zones exist between different people or stakeholders then a natural way to cope with this is to advocate a greater need for interactional expertise.

• Plainly: Testers need greater interactional expertise in dealing with stakeholders and business leaders. This means understanding their concerns, how they think about problems, listening for signals in which pieces of information they react to (and don't) - why are some pieces interesting. To a certain extent it means translating information into their language. What do the risks in your testing mean to them and the customers? How might they word or translate it?

• What type of information do they need to translate and why. It's not usually just about - can a product version release in the next quarter, sometimes it's also about understanding their risk picture - and what other contingencies they might need to prepare for.

• In fact, if you can talk with and listen to your business leaders more and more - understand how they think and talk and what their concerns are - you will get familiar with the trading zone language. For example, why do they emphasise certain ideas and concepts - and how does my work relate to that?

• Tip: Ask your stakeholders or business leaders if they are using language or concepts you don't understand.

• Corollary#1: Business leaders naturally need to listen to their product development organisation - for information about how they talk, which information they emphasise. In general, business leaders are more adept (used to) doing this - as they need to communicate cross-business.

• Corollary#2: There is no one right way to talk about a subject if two people are located in different paradigms (or even namespaces) - without acknowledging the need of language trading zones and even understanding which context the other person is grounded.

Potentially Related Posts
Communication Patterns
The Conway Heuristic

References

[1] Collins, Harry, Evans, Robert and Gorman, Michael. 2007; Studies in History and Philosophy of Science Part A -- Trading Zones and Interactional Expertise

[2] Collins. 2007. Rethinking Expertise (Chap 1, “Origins of Interactional Expertise”)

Some Software Development & Testing Challenges 2016

So it's 2016 and I have been reflecting on some of the challenges I see for Software Development, with emphasis on Software Testing.

Continuous Integration

Everyone knows what this is right? The concept has been around a while* - everyone has been there and done that, if you read the hype. But who is innovating?

A lot has been written about CI and its place in support of testing... Or has it?

Some Challenges

1. Massive parallel script execution against the same target drives a re-think on test framework design, modelling and creation - impacting data modelling and needs for flexibility in frameworks and harnesses
• This is a move away from single isolated and independent "tests" on a stateful application. It will trigger a change in test script approaches. Where is the work on that? Pointers gratefully received...
• I have seen some research on "multi-session testing of stateful services", but more is needed.
2. CI script execution and studies showing the effectiveness of dynamic test script selection strategies for team or testing support
• I see that as a rule-driven approach to setting a series of checks on commits, e.g. (1) which checks cover my updated code-base (execute and result), (2) which whitespots in my codebase are there now (report)
• Where are the studies or experience reports, where is the work?
3. There are socio-technical challenges with CI use and implementation. Technology is the easy part, the soci-technical part comes in when organisational issues and preferences distort the technology choices. This might range from "we have always done it this way" to "the language or framework of choice here is X so everyone must adapt to that".
• CI is a development approach, and is distinct from testing. It's like an extension to compiler checks**. Thinking around selecting and adding to those "compiler checks" needs to be dynamic. Experience reports, empirical studies for this?
• There is a danger that "testing" is driven into a TDD/Acceptance Test-only mode.
• I would like to see more research on organisational and soci-technical challenges around software development...
4. Are people really going all-in on cloud and virtualization technologies to solve their CI related bottlenecks? Mmmm...

Software Testing

Some Challenges

1. Detachment from Software Development
• This can be seen in various forms
1. Distillation down to process on "testing" only - the ISO 29119 papers are a classic example of this. This is the "reductionist" approach to a wicked organisational problem - not very sophisticated and with a high risk of solving the wrong problem.
2. Other examples are some/many software testing only books - yes, it can be good to highlight the testing and testers role and special challenges there, but until you start from software development as a whole (systems thinking approach) then there is a high risk that you are making a local optimisation. Another reductionist approach, liable to solving the wrong problem.
• So, software testing focus -> good; software testing interplay and interaction and CONNECTION to the creative process -> better.
2. Mis-understanding of the software testing challenge - how to link a creative process (software creation and positive and confirmatory tests and checks) to a challenging process (testing for issues, highlighting problems, testing for risks)
• Many organisations focus on confirmatory tests - a proxy for customer Acceptance Tests - as an MVP (minimum viable process), i.e. a proxy "get out of gaol card". See Myers [2] example of testing in an XP approach is an example here.
• Myers [2] first wrote about the psychology of software testing. However, Martin et al [4] make the case for reframing this as an organisational approach/problem. Rooksby et al [5] observe the cooperative nature of software testing.
• More studies on satisficing the organisational needs please!!
3. Lack of soci-technical studies and research into software testing and its part in software development. Rooksby & Martin et al [4] & [5] performed ethnographic studies of software testing to highlight its cooperative and satisficing nature. This called for further research
• Sommerville et al [6]:
• "An over-simplification that has hindered software engineering for many years is that it is possible to consider technical software-intensive systems that are intended to support work in an organization in isolation. The so-called ‘system requirements’ are the interface between the technical system and the wider socio-technical system yet we know that requirements are inevitably incomplete, incorrect and out of date."

The sooner we stop treating software development, and especially testing, in reductionist approaches, consider the socio-technical aspects - especially for large and complex systems - the better. And, today, most systems are inevitably complex.

Got any pointers to recent advances? I'm all ears...

References

[1] Li, Chou, 2009, IEEE; A Combinatorial Approach to Multi-session Testing of Stateful Web Services

[2] Myers, 2011, 3rd ed; The Art of Software Testing

[3] Rethinking Experiments in a Socio-Technical Perspective: The Case of Software Engineering

[4] Martin, Rooksby, Rouncefield, Sommerville, 2007, IEEE; ‘Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company

[5] Rooksby, Rouncefield, Sommerville, , Journal of CSCW; Testing in the Wild: The Social and Organisational Dimensions of Real World Practice

[6] Sommerville, Cliff, Calinescu, Keen, Kelly, Kwiatkowska, McDermid, Paige, 2011, Communications of the ACM; Large Scale Complex Systems Engineering

*I led the architecture work on a multi-stage CI system in ¨2010

**yes, a big simplification.

Standards, Straw Man Arguments and Superstition

I listened to a webinar in November 2013, ref [1], about the testing standard, "ISO 29119".

At the time I made some notes on the recording, but there was one item that particularly struck me as slightly intimidatory. However I happily assigned it to my mental trash bin and just remembered the meta-labels, "Straw man, scare tactic".

I had forgotten about the recording... until today when I saw this on twitter from James Christie:

james_christie @michaelbolton What happens when the press asks your CEO, "did you follow standards?" when there's been a fiasco? 2014-05-28 15:33

Note: These were not James' words.

Gut feeling: Straw man argument!

Why?

1. Why would the press be asking about a software testing standard when there is no evidence of the testing standard "working" or preventing similar "fiasco's"? It seems equally plausible for the press to ask if the omens in the CEO's tea leaves were followed!

2. If you have a CEO, that's worth his salt₁, and there has been a fiasco (presumably a software catastrophe), then I'd be very surprised if the thing that's highest on his mind is some internal witch-hunt.

He's probably gonna:

1. Put people on fixing the issue
2. Put people/teams on liaising with the customer (if it's one big and, understandably, angry one)
3. Put people on understanding what went wrong
4. Understand if the company did what they thought was right / good enough

I'd be surprised if a company that used technically competent people would have the/a root cause being that a template (or group of templates / standard) was not followed - especially if there is no empirical study (evidence) to demonstrate how the standard (template) might avert such a fiasco.₃

If a catastrophic failure has been found to be caused₂ by not following a template then I would love to hear about it.

I've been in those discussions with responsible managers and execs - and (maybe this is my good fortune of dealing with competent managers)₄ they're more interested in that the technical influential/responsible people do what is right / good enough for the needs of the business, rather than be concerned about being fully or partially compliant to a particular template.₅

Yes, I'm not naive, I know some people, projects and company cultures follow some template as an ass-covering exercise. I've been there.₁

So... The straw man argument is constructing a need to explain whether or not a particular template is used, when the need for that template is not established.

Reasoned argument?

Ok, so it was time for me to revisit the webinar, where at timestamp ~33-34mins, one hears:

"If you are still doubtful about using these standards, then I have a question for you.  

Can you afford not to? 

Imagine you're responsible for the testing of an important application. It could even be business critical, or safety critical, and something goes noticeably wrong with the application in use.  

Even if really good testing could’ve missed the bug, how easy will you find it to explain to the business your testing missed the bug and no, your test processes do not comply with international testing standards; that they are the ones we've used for years and no, you don't have them fully documented and justified.  

So, can you afford not to use them?"

First impression: You should follow a template.... just in case.

Second impression: Slightly sinister tone. 

Ok, let's take a closer look. Closer inspection:

1. The first two sentences are plain rhetoric, trying to establish doubt in the listener - doubt due to not following a standard. This is "rhetoric" because there's no argument/justification about why a standard would help you avoid this situation.

2. "if really good testing could’ve missed the bug" - good testing can miss bugs. This is not necessarily anything to do with a process. Good processes can miss bugs.

3. "how easy will you find it to explain to the business your testing missed the bug and no, your test processes do not comply with international testing standards" - this statement is intending to sow doubt. But the international testing standard has no evidence of being useful, appropriate or a proven track record.

I could go on and pick it apart, displaying the parts which are negative evaluations, argument precursors etc, but you get the picture. It says very little more than an innuendo of repercussions if you don't use the standard.

In Summary

• If someone tries to scare you, or generate doubt and worry, ask to see the evidence, the track record, the money. Look out for straw man arguments, misleading and off-the-point comments and allusions to superstition or unjustified belief.
• Successful CEO's don't usually look for scape goats in their staff, don't usually follow checklists without a good business reason. 
• For me the jury is still partially out on the new software testing standard - I will publish some thoughts on its content at a later stage. But, just because it's been labelled a standard does not make it comparable to other ISO or IEEE standards where interoperability or conformance to a standardised result is usually the goal - here the goal seemed to be to create a standard (job done! Is that the business case? WTF! )

Notes
1. If you have a CEO (or responsible manager) that doesn't trust the technical competence of the organisation then either (1) change the organisation, or (2) change organisation. And Yes! I have done both in the past!

2. Note, if a template is intended to replace competence then this should be declared also. (This puts an additional burden on the template, guidelines and usage of the template.)

3. The existence of a standard, template or checklist does not stop mistakes in following such a checklist

4. Competent managers (in my experience) usually do not chase paper trails to a template, and usually trust their staff. I'll give some tips on how to deal with managers that are tempted by reliance on checklists over competent staff in a separate post.

5. Businesses tend to adopt (or use) the standards that make business sense, rather than adopting a standard purely because there is a standard.

References
[1] Eurostar webinar: Webinar 75: ISO 29119 - The New Set of International Standards on Software Testing

On Test Results and Decisions about Test Results

A Test Result != Result of a Test

An Observation != Result of an Observation

It’s not the test result that matters, but the decision about the test result!!

Pass-Fail-Dooesn’t Start-Inconclusive-Can’t Execute

How you come to these results (and there are more) is interesting. But from a testing viewpoint what you do, what actions you take based on such results is very interesting.

“If a tree falls in a forest and no one hears it, does it make a noise?”

Or

“If a test result is obtained and not used or considered, was it worth it?”

Used?

Did it confirm an expectation?

Did it contribute to a body of evidence that a feature, product or system is behaving within bounds?

Did it help understand risks with the system?

If the answer is no, then it might be time to take a good long hard look at what you’re doing…

Body of Evidence / Evidence

All test observations and test results are not equal. They contribute to the picture of a product in different ways. But that picture is not necessarily a paint-by-numbers book. It’s not something that you can necessarily think I’ve ticked all the boxes, I’m finished.

Note: In many cases, Testing is not painting by numbers! 

Unless you’re a doctor finding no pulse and rigor mortis has set in!

It’s about making sense of the observations.

The 1% Problem Soundbite

Suppose 1% of your tests fail. Suppose you’ve seen a problem that 1% of your customers will experience.

The 50% Problem Soundbite

Suppose 50% of your tests fail. Suppose you’ve seen a problem that 50% of your customers will experience.

Based on this information is it possible to say anything about the product?

No. But you might have something that says, “need more information”.

These are what I think of as context-free reports.

From those soundbites, you don’t know anything about the nature of the problems, product, market that the product might be used in, circumstances for usage, etc, etc.

Suppose the 1% problem is a corner case - not allowing installation in a geographical location if some other feature is active - that might affect how the product is launched in that market. Suppose it’s something “cosmetic” that might annoy 1%, but not prevent the product being used. These two different types of observations (results) might produce totally different results.

The 50% case - is this a break in legacy, some feature that customers are using and needs to be operated differently; or is it a new feature interacting differently with existing features (e.g. flagged by some automated scripts) that hasn’t been launched yet and might be only for trial/“friendly users”. Again these observations (of essentially first order feedback, after Weinberg, ref [3]) might have totally different results.

Decisions and Supporting Data

1. Are you a tester that decides when testing is finished and a product can ship?

2. Are you a tester that tries to give your boss, project manager, stakeholder a good account of the testing done on a product? Might you even give some insight and feedback about what that testing might mean?

Ok, assuming you’re in the second category…

What does your test observation tell you?

What do your test observations tell you?

I’m reminded of a model I’ve helped use in the past about understanding test results, ref [1]. But now I’m looking at the flip-side, not what a “pass” means but what a “not-pass” might mean. A simple version of a test result / observation might look like:

Note, this is a typical interpretation when a test result is deemed to be of the form OK / NOK (including inconclusive, don’t know etc.) The implication of this is that when the desired test result (OK, pass) is obtained then that “test case” is done, i.e. it is static and is unchanged by environment or circumstance. This might be typical in conformance testing, where responses within a desired range is expected, usually when the test subject has a safety component (more on this in another post).

But, if the results are not black-and-white or can be open to interpretation (as in the 1% problem soundbite) then a different model might be useful.

This model emphasises the project, product and testing elements to consider - these may change over time (within the same project), between product versions and between testing cycles (or sprints). I’ve drawn the line between product owner and product decision as a thick black line to highlight that this is the deciding input.

More on testing and silent evidence can be found in ref [2].

A larger version of this picture can be found here.

References

[1] Testing Chain of Inference: A Model http://testers-headache.blogspot.com/2012/08/testing-chain-of-inference-model.html

[2] Silent Evidence in Testing http://testers-headache.blogspot.com/2012/03/silent-evidence-in-testing.html‎

[3] Quality Software Management: Vol 2 (Weinberg; 1993, Dorset House)

Simplistic Views, Complex Subtexts and Whaa?

Recently I read an opinion piece, ref [1], from a Technical Evangelist (more on that label later). My first impression of it was a muddled message. Which point was the piece trying to make?

So, I made a quick close analysis of the text and posted some questions, which basically amounted to, “do you have data/evidence to back this up” and “can you clarify what you mean”, ref [2]. I’ll be happy if I get a response.

Close Analysis

Close analysis (after ref [3]) is a form of critical analysis where the text is dissected into argument markers, assuring and guarding terms, discounting and evaluative terms and rhetorical devices. The aim is to analyse a text sympathetically - i.e. to try and make the person’s argument as good as possible - to get to the real meaning of what they are trying to say.

This is my usual first step where I need to pay close attention to text. In the case above I found it unclear and bordering on incoherent hand-waving. Sometimes where the people are responsive then there might be a discussion that starts with, “can I ask some questions?” (ref context-free questions, ref [4])

Yes, it’s time to stop, think and ask questions - what is the message, where is the supporting analysis or evidence.

I will probably be doing this more and more. I’m currently making a close inspection of Taylor’s Principles of Scientific Management. I’ve also recently had reason to look at Program Test Methods and I’ll be revisiting that to provide a commentary of it. I started looking at these because they either get referenced in several places or are the basis for further work. I may end up revisiting more of my software testing library and other literature (as I have done with “regression testing” in wikipedia and the ISTQB glossary, ref [5]).  

Some useful Heuristics for Potential Spotting Problematical Articles

Evidence or Data?

If a claim is made is there a reference to some analysis that you can read or how that conclusion was discovered?

Beware of data being retrofitted (cherry picked) to support a claim.

Do the claims appear to be anecdotal or have a single source?

Meaning

Is the meaning clear?

Does the conclusion follow from the argument? 

 - These two are the main parts of close analysis.

Why was the piece written? Was it an opinion or were they trying to influence opinion?

How was the reaction to the piece? Peer review, open comments, discussion or other commentary?

Labels

The labels people use are important. Evangelist has many connotations (to me) of preaching someone else’s thoughts, words and ideas. Ok, so if you don’t think for yourself should I go to the source?

BTW, the same (to me) applies when I hear someone describe themselves as a “passionate <whatever>” - I wonder does that mean they’re enthusiastic, irrational emotional or irascible.

Labels like expert and authority should be distinguished whether it is self-appointed or peer-recognised.

And finally….

I know this won’t stop articles and books where “I only have a few minor comments”, but it’s important to distinguish a simplistic or nonsensical view from a complex subtext or just plain BS.

If you don’t understand it or where a claim is made without evidence, call it out!

References

[1] LinkedIn “The Hard Truth about Software Testing” http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing

[2] Comment: LinkedIn “The Hard Truth about Software Testing” http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing?goback=%2Empd2_*1_*1_*1_*1_*1_*1_20140411172902*546939713*5the*5hard*5truth*5about*5software*5testing&anchorTime=1397426609731&deepLinkCommentId=5861232018901209088&trk=mp-detais-tw-cmnt 

[3] Understanding Arguments, 8th edition, chapter 4 [Sinnott-Armstrong, Fogelin; 2009, Wadworth]

[4] Exploring Requirements: Quality Before Design (Gause, Weinberg; 1989, Dorset House)

[5] The Tester’s Headache: Another Regression Test Trap http://testers-headache.blogspot.se/2013/03/another-regression-test-trap.html 

Examining the Context-Driven Principles

At Let's Test 2013 James Bach had a keynote about the Context Driven Testing principles, some of their origin, usage and implications of them. An interesting piece of trivia that James revealed was that the basis of the principles were written down very quickly, with some refinement afterwards.

He discussed a little about the implicit meanings behind the 7 principles, but the slides were skipped through so I didn't see the details. However, by this point at least one question occurred to me.

My question in the open season went along the lines, "The principles were mentioned as a litmus test to gauge if someone was context driven. However, adopting a scientific approach (ie one of searching for information and not settling on an assumption), there may well be more than 7. How do we add number 8?"

An underlying question was (for me):

1. If the principles were written so (comparatively) quickly why haven't they been challenged from testers claiming to follow or adhere to them?
2. Indeed, isn't this a good exercise for a tester - context-driven or otherwise?

Therefore, I thought I would take a look at them - as a (comparatively) quick exercise. James did mention that Markus Gärtner had extended the principles as an exercise previously. I remember seeing a discussion on the software-testing yahoo list some time ago and checked it out - and sure enough I'd discussed the possible extensions there, and thought it was a useful topic to revisit.

The 7 published principles, ref [1], 

1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

My Approach
In assessing the principles I used two approaches, (1) Critical/Logical consistency - do the propositions support conclusions or implications?; (2) Are the principles testable (falsifiable)?

Assessment
#1 [Unchanged] I tried to extend this one, but I think it is rhetorically good - succinct and a good start to the principles. This can not be tested but is inferred by induction.

#2 [Remove] I could think of  removing/replacing this one. It is a derivation of #1. The essential content here is that "Practices evolve over time" - I thought of making this my #2, partly to remove the distraction of "best practices" (that would be a derivation of these principles), and partly to emphasize the evolution of practices. However, I finally settled on having no #2.

#3 [Rephrase] The principle is an assertion. For this one I want to emphasize that the project, the processes and people form a system and that they change over time. The importance here is to illustrate that the system is not static, it is complex because of people (and their emotions) and as such, working with such a system is not a trivial activity. Therefore I would re-phrase.

#4 [Remove] This is a derivation / implication of #3, an application of the principles.

#5 [Unchanged] Again, succinct and difficult to refute.

#6 [Unchanged] An assertion that where good software testing happens there is a high intellectual demand.

#7 [Remove] This is unnecessarily wordy and sticks out as too different from the previous principles. It also drifts into certainty and seems like an untestable principle, and so I would remove it.

Addition
#8 Here I want to tie the system mentioned in the rephrased #3 and good software testing.

Result

1. The value of any practice depends on its context.
2. -
3. Projects and people form part of the system that work together, where products and practices evolve over time
4. -
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. -
8. Good software testing encapsulates the system of project, people and practices that work on building the product.

So, I ended up with 5 principles. The application of these would then produce variants and clarifications. For instance, the best practice item is derived from #1. Understanding systems of people, emotions and time is derived from these and many, many more.

I stopped there, but if I spent more time could probably refine them somewhat. But they are good enough for this exercise. And as James suggested - if you started from scratch, you might well not end up with the original 7 principles.

So, how would your interpretation look?


Reference
[1] http://context-driven-testing.com/

Checking and Information Value

It's now over three years ago that I wrote my last post on "testing and checking" - and I wrote three in total. I consider them a little 'rough' but I knew they contained some gems so revisited them recently - refs [1], [2] & [3].

James and Michael wrote a blog recently, clarifying and redefining what they meant by testing and checking, ref [4].

After reading it - and thinking about it - I felt that some emphasis was missing - for me. This doesn't talk directly to the definitions but more the application of them.

Note, James & Michael seem to apply this refined terminology within the frame of the Rapid Software Testing methodology, ref [11], but I think others will use it outside of that also.

Some History re-cap
After reading the original post, ref [5], I started wondering, "why?", what was the motivation behind making the distinction. I didn't experience the problem being described and couldn't see how the distinction might help me. As part of that I started wondering about how checks and checking was being framed and the extent to which it was being wrapped in testing.

My original questions:

• Why make a check when no testing is involved?
• How or why would a check be made without testing?

- ie why make a check and not analyse the result; why make a check without any thinking/analysis (testing) beforehand?

Note, the definition didn't say that a check wouldn't be evaluated, but it could be misinterpreted that a check without testing was useful (although I didn't think that was the intention).

So, at the time, I couldn't detach checking from testing - I couldn't see a situation where a check would be made without testing. That was my perspective and how I would advocate checking.

But I'm also very aware that there will be people/organisations that do some form of checking without any testing... How might that happen?

• "We always run this batch of scripts. When they pass then the SW is ready."

Note, I believe the definitions are trying to enable more accurate observation and discussion about elements that were testing or not.


Potential Trap

A check (or checking) is used in isolation without any test "thinking" (whether as analysis beforehand or afterwards.) This case is a form of the trap mentioned in ref [6].

On to Information Value
So, what was I missing? My early posts were looking for reasons / needs for the distinction - and I accepted that the distinction was useful (although, I admit, not immediately as I didn't see the utility in them in the beginning - that came later.) My remaining question was around emphasizing the importance of testing and the selection of a check as being rooted in testing.

During the last year I started thinking more and more about emphasizing information value of testing. This is partly trying to establish congruence between customer needs and testing goals, ref [7] & [8], partly about introducing test framing, ref [9], and framing checking from a testing perspective, ref [10].

So, where does this bring me?

Checking in isolation
Talking about checking in isolation has value when identifying the activity - whether as something that is planned or an observation in a process or activity.

• "On the upcoming release we'll reuse some checks from the last release - they will give feedback on basic data configuration - whether the data configured pre-upgrade is still visible after upgrade." (plan)
• "You're reporting a fault due to an automated script flagging an issue?" (activity) "Ok, has the result and activity been double-checked - that the flagging is correct?" (testing)
• "There is a bunch of automated scripts that run on every code change - and they never issue any problem indication. Is that ok, or is it time to review their content and purpose?" (process) 

Information Value of Checks and Checking
When considering whether to use a check or checking there is an implicit activity of comparing what you'd like to know about a product, what you can achieve (and how) and what each instance of a check might give. At least, I claim (with only anecdotal proof) that this is the case in many good testers. For example:

Wanted Information about product

Note, this example excludes people not making an evaluation about a check or whether or not to do any checking or what the check outcome says...

Part of this evaluation (sometimes estimation) is to understand the value of making a check - something cheap and easy might be preferable, but something expensive that returns a large information value might also be worth the effort. The key thing is the decision to make the check - because it will provide some information.

Example: In the past I've advocated 169hr tests (checks) - the result might be good/no good (pass/fail), but the value of it might be part regulatory PLUS a lot of additional data that can be mined/explored.

Why it might be useful to check or perform checking is then attempting to work out the value of the information it might give. This is information about the product "under test".

Information value can be estimated beforehand and/or assessed afterwards.

Checking as part of Testing
When information value (or at least its intention) connected to a check or checking is captured then the check or checking becomes part of testing.

A model of testing, checking and information value could be expressed as:

Testing ≥ Checking + Information value

In mathematical terms it could be written:

Testing ⊇ Checking + Information value

This means every instance of "Checking + Information value" is an element of Testing but also that "Testing" does not equal "Checking + Information value".

Checks (with no associated information value) ∉ Testing

Meaning checks without associated information value (either decision to make a check or analysis of a check outcome) are not part of testing.

Conclusion
Whether I think of checks, checking and testing inside the Rapid Software Testing methodology or not I think the terminology is useful. It is useful to identify artifacts of checking in the wild - whether in observed processes, activities or intended actions. It is useful to identify and categorize as any field scientist would categorize their observations.

In talking to people about processes and their actions it is very useful to describe and illustrate when a process was being followed, maybe even identifying assumptions that were not being stated and how they might confuse or muddy communication about what was really happening. It might be useful to highlight the extra work needed to make the activity into good testing.

In my worldview of testing there are no checks or checking in isolation, there is either an intention or idea beforehand about the usefulness of such a check or an analysis afterwards about the usefulness of the check or other assessment about the information it yields. Often, in the cases of good checking (implying good testing) both the beforehand intention/idea/hypothesis and the resulting assessment will both happen. That is my ambition where checks are used.

In the cases where checks are used without pre-determined intention or resultant analysis - and this does happen - my ambition is to raise the bar. In my worldview this is not testing. I want to raise the bar partly by awareness - noticing (and getting others to notice) this happens, partly by contrasting this to how it might look if we connect some information value to those checks. In so doing, understanding the real value (and maybe cost) of good testing. Note, there is no automatic raised cost by adding information value to checks to get testing, but rather a reduction in the overheads of "fake testing" (an activity which was maybe thought to be testing but wasn't).

I wish to raise the bar of checking by emphasizing the associated thinking attached to checking to identify information value (positive or negative), and so raise the bar of testing.

Thoughts, comments?

References
[1] The Tester's Headache: To test or not to test, just checking... http://testers-headache.blogspot.com/2009/08/to-test-or-not-to-test-just-checking.html
[2] The Tester's Headache: Sapient checking? http://testers-headache.blogspot.com/2009/09/sapient-checking.html
[3] The Tester's Headache: More notes on testing & checking http://testers-headache.blogspot.com/2009/09/more-notes-on-testing-checking.html
[4] Satisfice: Testing and Checking Refined http://www.satisfice.com/blog/archives/856
[5] Developsense: Testing vs. Checking http://www.developsense.com/2009/08/testing-vs-checking.html
[6] The Tester's Headache: Another Regression Test Trap http://testers-headache.blogspot.com/2013/03/another-regression-test-trap.html
[7] YorkyAbroad Slideshare: Testing Lessons From The Rolling Stones http://www.slideshare.net/YorkyAbroad/testing-lessons-from-the-rolling-stones
[8] The Tester's Headache: Mapping Information Value in Testing http://testers-headache.blogspot.com/2012/12/mapping-information-value-in-testing.html
[9] Developsense: Test Framing http://www.developsense.com/resources/TestFraming.pdf
[10] The Tester's Headache: Framing: Some Decision and Analysis Frames in Testing http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html
[11] Satisfice: Comment on Testing and Checking Refined http://www.satisfice.com/blog/archives/856/comment-page-1#comment-268572

Another Regression Test Trap

I mentioned in a previous post, ref [1], a trap to be aware of when thinking about regression testing. Time for another...

There is a lot of "rubbish" written about regression testing. As with other aspects of testing it is sometimes commoditized, packaged and "controlled".

Trap

Regression Testing is the act of re-running tests - that have previously been developed, designed or executed. Period.

The trap is that it stops there...

Sometimes, there is a contextual aspect included - which is addressed by stating that analysis of changes in the product, including the processes that were used for that change should be made. But in many cases - and indeed, in several pieces of literature discussing regression testing - the extent of this analysis is about setting the regression test scope from the previously developed or designed test ideas.

Hmmm, seems reasonable? Or does it? Let's look at a couple of commonly available sources of information.

Other Sources on Regression Testing:

Wikipedia, ref [2]: "Regression testing can be used to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change."

So, selecting a minimum set of tests - systematically - is an efficient test of a system, if that set provides adequate "coverage" of a change in the system. Mmm, good luck with that. This is bordering on pseudoscience, ref [4], - it's an untestable and vague claim. Having said that, the statement is qualified by "can" - hmm, ok, not so helpful though.

ISTQB glossary, ref [3]: "regression testing: Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is
changed."

I can almost agree with this. Almost. Well, ok, not really. What's a big sticking point for me? "Ensure". My dictionary (New Oxford American to hand just now) states that ensure is either a guarantee or the action of making sure... Ok, so how can my testing guarantee or make sure that a defect has not been introduced. Why would it want to make sure that a defect hasn't been uncovered - wouldn't you really want to know about this????

Note, it's possible to find similar problematic definitions in testing text books.


And?

Ok, am I splitting hairs? Am I dissecting these definitions more than I should? Well, what level of clarity should I hold them too? Should I be satisfied that they give "surface" explanations and that I should not examine or look too deeply? The thing is, if a definition falls apart under scrutiny is it worth using? I can't use it and I don't advocate it they can't define something then the definition falls apart (or becomes ambiguous) under scrutiny is it worth using????


What's the problem?

My issues with typical regression scopes are connected to two often unstated assumptions:

1. An assumption that the regression scope that "worked" once is sufficient now - where the system has changed. (This means that the number of test ideas covered previously - whether expressed as instances (test cases) or not - are good enough for a legacy scope now.)
2. An assumption that a previous regression scope is somehow a conformance scope. (This means that an instance of a test idea - a test case - is correct now, in the new/modified system.)

Note, if these assumptions are made they should be stated. It could be adequate in some cases to assess the previous scope and determine that it is sufficient for the current needs. It could be useful to state that the previous regression scope is being treated as a conformance scope (or reference scope, or oracle) - if that's what the assessment of the current problem/system produces.

I think a post on how I'd approach a regression test problem is in order...

Tester Challenge
On a train journey returning from SWET4 to Stockholm I sat with James Bach and he tested me on "regression testing". He suggested there was one case where a previously executed test campaign could be repeated as a regression testing campaign. I gave him two.

I'll leave these as an exercise for the reader...

References
[1] Regression Test Trap #492 http://testers-headache.blogspot.com/2013/02/regression-test-trap-492.html
[2] Wikipedia: Regression testing http://en.wikipedia.org/wiki/Regression_test (phrase checked 29 April 2013)
[3] ISTQB: Standard glossary of terms used in Software Testing, version 2.2 http://www.istqb.org/downloads/finish/20/101.html
[4] Wikipedia: Pseudoscience http://en.wikipedia.org/wiki/Pseudoscience

Bonus Reference
[5] Checklist for identifying dubious technologies http://www.randi.org/images/commentary/200703/Checklist%20for%20identifying%20dubious%20technical%20processes%20and%20products.pdf

Regression Test Trap #492

I didn't call this "the" trap, as I think there are several. This is one aspect.

In June last year I had a little twitter rant/exploration (or transpective investigation, [1]) about regression testing. It was triggered when I saw a LinkedIn skill labelled as "Regression Testing". I wanted to know which skills that people considered specific to regression testing.

Now, one can take some of the LinkedIn labels with a pinch of salt, but I wanted to test the water about this one.

Note, I really enjoy these "bursts of activity" I have on twitter. There are many great minds that immediately question, probe, ask for explanation or example. I love it and am very appreciative to anyone that asks me to explain myself. It makes me stronger and clarifies my thoughts!

The twitter discussion dump, via twimemachine, in reverse order:

@Arborosa you have that problem in non-regression-specific testing also Thu Jun 28 06:54:20 +0000 2012@jamesmarcusbach Oh, the NAP is a great resource - was looking at this the other day http://t.co/Tjn3b18L Wed Jun 27 10:00:53 +0000 2012@jamesmarcusbach That's great, thanks - I need to spend more time on it - reminds me of Göranzon's work-> must dig that out again Wed Jun 27 09:49:44 +0000 2012@jamesmarcusbach True, it's one of my aids to avoid the trap of going from "application in context" to "application in general" Wed Jun 27 09:42:56 +0000 2012@jamesmarcusbach I try to distinguish between skills and application of skills, reminds me of "Dialogue Skills and Tacit Knowledge" Wed Jun 27 09:25:44 +0000 2012@jamesmarcusbach I call the workshops a feature walkthrough, see slides 20-27 in http://t.co/XnF5Ubnk - usual health warning with ppt slides Wed Jun 27 08:50:05 +0000 2012@jamesmarcusbach In my context, these skills are not specific to regression testing Wed Jun 27 08:37:44 +0000 2012@jamesmarcusbach Test idea maintenance - not specific to regression testing 4me - can be triggered by feature or test harness change/growth Wed Jun 27 08:36:42 +0000 2012@jamesmarcusbach Interviewing and analysis - I do this as part of feature interaction analysis - sometimes 1-2-1, sometimes as a workshop Wed Jun 27 08:36:00 +0000 2012@DuncNisbet yes, start from a picture of what you want to learn + where risks are - then let that point to what's "interesting" to know Tue Jun 26 22:08:01 +0000 2012@DuncNisbet I think of tests with a potential to find a problem as "interesting" - ideas to pursue/try out Tue Jun 26 22:00:29 +0000 2012@rahul_verma me too - I thought your questions were good and relevant! Tue Jun 26 19:10:03 +0000 2012@rahul_verma look forward to it Tue Jun 26 19:02:17 +0000 2012@rahul_verma yes, that's what I aim for Tue Jun 26 19:01:48 +0000 2012@rahul_verma and how many other types and application of testing were they no good at? :) Tue Jun 26 19:00:47 +0000 2012@rahul_verma I think people mix up activities, skills, application and results - and makes our job (communicating about a product) harder :( Tue Jun 26 18:57:20 +0000 2012@rahul_verma applying a skill successfully in an activity (RT) does not make that a skill specific to that activity Tue Jun 26 18:56:15 +0000 2012RT @rvansteenbergen: Want tweets about the latest news from testblogs? Follow @testingref #softwaretesting Tue Jun 26 18:45:00 +0000 2012@jamesmarcusbach I'd love to hear about the other things Tue Jun 26 18:14:31 +0000 2012@rahul_verma agree! Tue Jun 26 18:11:32 +0000 2012@rahul_verma on a certain level performance testing and regression testing do have an overlap of ambition Tue Jun 26 18:10:10 +0000 2012@rahul_verma performance testers focus on an element of a product's characteristics - to measure, monitor, feedback, recommend and help tune Tue Jun 26 18:09:02 +0000 2012@rahul_verma exact knowledge is tricky to define.... Tue Jun 26 18:03:28 +0000 2012@rahul_verma I'd say the purpose distinguishes it as regression testing, but not necessarily the skills used Tue Jun 26 18:02:42 +0000 2012@rahul_verma Are you asserting that regression testing is done by regression testers? Tue Jun 26 13:05:39 +0000 2012@rahul_verma What skillset in "regression testing" is specific to regression testing? Tue Jun 26 13:03:13 +0000 2012@rahul_verma Think skillset rather than domain knowledge Tue Jun 26 13:02:15 +0000 2012@rahul_verma Is that separate from good testing? Tue Jun 26 12:55:56 +0000 2012@thetestmile And is that separate from good testing? Tue Jun 26 12:55:34 +0000 2012@rahul_verma Are you saying you can be a good tester and not a good regression tester, or vice versa? Tue Jun 26 12:55:08 +0000 2012Now where's the "black viper testing" skillset button.... Tue Jun 26 12:42:55 +0000 2012"Regression testing" is a "skill" in linkedin. Really? Discuss... #faketestingmeasurement Tue Jun 26 12:41:39 +0000 2012

Trap

The trap is to think of the skills associated with good regression testing as being separate from good testing.

In my context and experience, I consider good testing to take into account aspects of new features that interact with legacy (existing) features or systems. This implies that "regression testing" is included. But if I think of specific skills needed for regression testing then these are just as applicable to "non-regression testing":

• Investigative skills about the domain (system)
• Discussing development and the existing system with developers/BA's/stakeholders
• Test Idea / Environment Analysis & Maintenance
• Reporting and discussing the status of the system
• etc, etc

So, is a "regression testing" skill separate or unique to regression testing? I assert not, and that it can be just as common in "non-regression testing" or any good testing.

To Be Continued

I have another trap about regression testing - coming soon.

References

[1] Satisfice: Transpective Dialogues for Learning http://www.satisfice.com/blog/archives/62

Other "traps":

[2] The Linear Filter Trap http://testers-headache.blogspot.com/2012/03/linear-filter-trap.html

[3] The Documented Process Trap http://testers-headache.blogspot.com/2012/06/documented-process-trap.html

[4] The Hindsight Trap http://testers-headache.blogspot.com/2012/03/hindsight-trap.html