ISO 29119 Questions: Part 1

Reason & Motivations?

There is an ISO software testing standard (ISO/IEC/IEEE 29119). Currently parts 1-3 are published, and parts 4-5 are in draft. I have read parts 1-3 and the draft of part 4 and am and have been forming several questions. Some questions might be regarded as problems, some as issues and others as a need for clarification.

I will use a series of 3 posts to document. 

Part 1: Looks at the reasoning and motivation for the standard - based on the information publicly available.

Part 2: Looks at some of the content.

Part 3: Looks at the output and usage of the standard and some implications.

Note, in this post I’ll only refer to publicly available versions and parts of the standard - the part I am considering here is the introduction - that is viewable in the preview part of the IEC webstore, see references.

Why?

Whenever someone comes to me with a new idea - I usually hear about all the reasons why it’s a great idea. What isn’t usually obvious is to say why the change is needed - or what problem is being solved.

29119

And so I had the same question in my head with 29119. I wanted to know the reasons and motivations behind it - especially ones that might support its usage or adoption.

According to 6.1.4 of the ISO Guide for Standards, ref [3], the introduction should state:

The introduction is a conditional preliminary element used, if required, to give specific information or commentary about the technical content of the document, and about the reasons prompting its preparation.

So, there should be reasons in the introduction of 29119. Looking at the introduction, there are three potential reasons given.

Reason 1?

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed to set of standards for software testing that can be used by any organization when performing any form of software testing [see refs 1 & 3]

Looking more closely at that:

• “internationally”: ISO implies this already - so this part is redundant
• “agreed”: the drafting of a standard requires consensus/agreement of 75% of the drafting members, so this is also redundant when the standard is published.

Note: One could ask questions about the representativeness of those drafting 29119, but that’s not a question I’m looking at here.

So “internationally-agreed” is redundant.

• “set of standards for software testing” -> seems to be a duplication of “software testing”, redundant.

• “can be used”: possible - this probably relates to how the standard can be tailored (more about this in part 2 of this post). 
• “by any organisation”: quite a bold claim, but linked with the previous point on tailoring this could be read as any company can tailor the standard.

So - “can be used by any organisation” -> could be generously interpreted as “can be tailored”. But then this is still problematical. 

Standards application and conformance are usually full compliance, partial (or tailored) compliance or non-compliance. Therefore, the fact that you can declare conformance or not goes with the territory of a standard. Therefore, this is redundant information - in terms of reasoning for a standard. 

An alternative reading could be that the standard is “useable” by any organisation. Again, this is redundant information - either the standard is adopted and conformance declared against it, or not. And why would effort be placed on a non-useable standard? Therefore, this is redundant information.

• “when performing any type of software testing”: again another bold claim. But part 1 of the standard “defines” the types of testing it is talking about.. The “performing” part is also implied as why else would you use the standard if not for software testing? So these parts are also redundant - by implication if you’re following the standard you’re following the definitions of the standard. (more about the definitions in part 2 of this post).

So, the paragraph becomes:

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed to set of standards for software testing that can be used by any organization when performing any form of software testing

Correcting the grammar for the parts that are stripped out would look like:

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define [a] … set of standards

Reason 1: Verdict?

So, when you strip out the redundant parts of the statement the purpose of the standard is to create a standard. That, in grammar, is called a tautology -> redundant information. That’s not much of a purpose - it still does not tell me why the it was produced.

Reason 2?

“This series of international standards can support testing in many different contexts.“ [see ref 1]

• “can” - I think this claim is a moot point - I suspect the reason is that, “if you claim conformance then it’s supporting testing” - so it’s not actually a reason /for/ the standard. It’s like saying, “if you follow the standard then you are standard-compliant”.

• “support testing” - well as part 1 is attempting to define testing and part 2 is attempting to define a process then in a way it could be claimed to support testing. On the other hand, it could be read that if you perform testing in accordance with parts 1-3 then the standard is supporting your testing - actually this is wrong, because it has defined what is “your testing” and potentially “what is excluded” - in this sense it’s not supporting at all. 

Reason 2: Verdict

My reading of this sentence is, “if you follow the standard then the standard will support your testing”. Unfortunately, this is a circular argument, ref [4], i.e. not really a supporting reason for the standard at all.

Reason 3?

“Together, this series of international standards aims to provide stakeholders with the ability to manage and perform software testing in any organization.”

• “stakeholders” - this is ambiguous who is meant. I’ll be generous and assume the user of the standard is meant here.

• “ability to manage … software testing in any organisation”
• “ability to … perform software testing in any organisation”

Interesting. This could be interpreted as anyone - absolutely anyone - by following this standard, can perform (and/or manage) software testing in any organisation. In fact, I’m not sure how to interpret it in a different way - there is no guidance (clarification) in the text.

Ok - I have seen (quite a few) people in my time that (i) can’t manage software testing and (ii) are pretty mediocre at software testing. In my judgement I have a hard time understanding how this standard would give the ability for “anyone” to become “testing managers” or “good testers” - except by creating a lowest common denominator. 

I can almost hear the explanation, “Find the least able person and we’ll calibrate to that person.” Really?

I have met and worked with - and I’m sure others will have similar experience - people (managers and testers) in various companies that are “going through the motions” - they tend to work /within/ a box (set of definitions or practices) and are not the ones who think or challenge assumptions (i.e. the ones who “think on their feet”). These are the people that spend more effort defining boundaries than communicating - they are plodders. If the aim of the standard is to produce a set of test managers or testers that either (a) think inside the box, (b) don’t think about or question what they’re doing, (c) plod along; then it is not something that can be associated with good (or efficient) work.

A Possible Reason 3 Verdict

The statement is telling - and probably says more than it really should.

Come on! If the aim of 29119 is to set the bar so low that anyone can look as though they are performing good work - BECAUSE they appear to be following 29119 - then that Mr Regulator, Mr Customer, Mr Potential-Stakeholder, I repeat, that is a “bad smell” - it’s a sign the company claiming conformance hasn’t got their eye on the ball! In a sense - you’d want to be extremely careful of anyone claiming conformance…

Introduction verdict?

According to 6.1.4, ref [2], I didn’t see any reason for the standard.

The skills I used to analyse the text were reasoning, logical and critical thinking - basic skills in any good testers toolkit - it comes into play (in my experience) when discussing requirements, understanding needs from stakeholders and reacting to and understanding conflicting needs in a project situation. Or even trying to understand standards. Actually, it’s difficult to do well and takes practice and concentration.

I had the impression that a number of testing experts were involved in the drafting and review of the standard. I think the resultant communication in the introduction has room for improvement - and doesn’t bode well for the rest of the documents. I’ll leave it to the reader to judge if the people involved in drafting the introduction did a good job of explaining the reasons and motivation for 29119.

Other Sources

Looking elsewhere for reasoning for the standard. Ok, let’s try some other sources.

Web #1

If I look at the “softwartestingstandards” web page, ref [5], I see this:

“By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.”

Ok, the part about “adopting the only internationally-recognised and agreed standards for software testing” is covered above - it’s the same logic - and reduces to a redundant phrase (says nothing in terms of motivation).

The next part is interesting though, “which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world”.

I’m interested that “high-quality approach to testing” is used - this is not stated in 29119, either as a motivation for using the standard or as a result of using the standard. As far as I can see there is no evidence (whether as case-study, or something else) to support this claim. Therefore, it is rhetoric - an attempt to influence without any evidence (proof).

Web #2

A Presentation given at the BCS, page 8 ref [6], states the motivation for 29119 as:

• Demand for existing 'standards’

• Conflicts in current definitions and processes

• Gaps in the current standards provision

• A Baseline for the Testing Discipline

• Current industry practice is lacking

• Buyers unclear on what is 'good test practice'

“Demand for existing 'standards’” - it’s not stated (or referenced) in the presentation where or what this demand is, therefore it’s an unsupported claim. Rhetoric.

“Conflicts in current definitions and processes” - 29119 is replacing some standard, so there is scope to believe that this replacement is reducing conflict between definitions and processes. However, the case for why this needs to happen is not made - it’s not demonstrated (in this presentation or elsewhere) what this conflict looks like or what problems it causes. Therefore, this seems to be a pretty weak argument. In terms of support in the presentation this claim is unsupported. Therefore in the scope of the presentation it is rhetoric.

“Gaps in the current standards provision” - it’s not stated (or referenced) in the presentation where or what these gaps are, or if or why they are relevant to being a motivation for a new standard. Rhetoric.

“A Baseline for the Testing Discipline” - it’s not stated (or referenced) in the presentation where or what this need is, therefore it’s an unsupported claim. Rhetoric.

“Current industry practice is lacking” - I could probably agree to this (but maybe for completely different reasons!) Without more information it’s not clear what this means - and isn’t stated (or referenced) in the presentation. Therefore, the claim is unclear and appears to lack any supporting evidence. Rhetoric.

“Buyers unclear on what is 'good test practice'” - I like the idea of presenting and distinguishing good test practices. However, the argument about buyers is not supported in the presentation (or referenced) with evidence. Therefore it’s an unsupported claim. Rhetoric.

Original Proposal for the Standard & ISO Directive #1

According to Annex C of ISO Directive 1, ref [7], a proposal for a new standard just have a justification. Specifically:

“C.3.2 The documentation justifying new work in ISO and IEC shall make a substantial case for the market relevance of the proposal.C.3.3 The documentation justifying new work in ISO and IEC shall provide solid information as a foundation for informed ISO or IEC national body voting.”

Now to documentation proposing the work for the new standard, ref [8], produced in 2007:

The market requirement section states:

“The national standards on which the new international standard is to be based are widely used both as the basis for commercial contracts and international qualification schemes. However, their national origin often results in them being ignored by potential users in some countries. At present there are a number of gaps (and overlaps) in the coverage they provide. A coherent set of international standards for the complete life cycle is required.”

The first two sentences state that some national standards are used, but not widely. The third sentence states there are gaps/overlaps between existing standards. 

The forth sentence is interesting: “A coherent set of international standards for the complete life cycle is required.” But where is the case made according to C.3.2 & C.3.3? According to point C.3.3 there is no evidence to support this point.

Ok, no help there.

In the “purpose and justification” section it does state:

“The purpose is to produce an integrated set of international standards to cover the software testingprocess throughout the development and maintenance of a software product or system.”

and

“In overall terms, the purpose of the project is to unify and integrate the currently fragmented corpus of normative literature regarding testing that is currently offered by three distinct standards-makers: BSI, IEEE, and ISO/IEC JTC 1/SC 7. The result of the project will be a consistent, unified treatment adopted by all three organizations.”

These two purposes seem complimentary, but there are problems. 

In the first says that “the software testing process” will be covered - but this seems undefined. The part 1 of 29119 outlines concepts and definitions, doesn’t it? Well there’s a problem there - that I’ll cover in part 2 of this post. But to summarise here - no parts 1-3 do not define “the software testing process” - part 1 is informative, meaning it has examples and not definitions - there is nothing to stop someone else creating their own definition. Part 2 has problems with conformance meaning that it can’t (and doesn’t) define THE software testing process. More details in part 2 of this post.

So this purpose might be real, but it’s also unrealistic. Unjustified.

The second purpose - to integrate information from different sources (BSI, IEEE & ISO) might seem reasonable - but this is more of a paper exercise than anything to do with standardising. There is no justification why this should be done or what problem is caused to the users of those individual sources. So, this is an unjustified purpose.

Summary

I wanted to understand the motivations and justification behind 29119. 

I looked in the introduction of 29119-1, where it is supposed to be stated according to ISO’s directives. I didn’t find anything that didn’t reduce to tautology and rhetoric. Now, I don’t know personally any of the members of the workgroup that produced 29119. But they are supposed to be experts in their field, and yet they produced an introduction to describe the motivation of 29119 that does not stand up to scrutiny. 

Did they all have an off-day - that lasted 6 years? Groupthink? Or what was it?

I looked elsewhere - presentations given to the BCS and the softwaretestingstandards site. There were a lot of claims without supporting evidence or references. This is also known as rhetoric.

Then I searched in the original proposal for the workgroup that would produce the standard. There, I found claims without evidence, some were unrealistic and some were almost a wish list.

After reading this I wondered about the people that reviewed the proposal. I’m sure there’s a bunch of intelligent people that looked at this and yet they seemed to have missed a lot. Was it some form of groupthink, I don’t know.

Well, I've been looking for reasoning and motivations for 29119 that will stand up to scrutiny - so far without success. In the next part I'll look at some of the content of 29119 in more detail.

References

[1] IEC/ISO/IEEE 29119-1 preview http://webstore.iec.ch/preview/info_isoiecieee29119-1{ed1.0}en.pdf

[2] ISO/IEC Directives Part 2 ISO/IEC 

Directives Part 2

[3] The Art of Software Testing Standards http://www.softwaretestpro.com/Item/5895/The-Art-of-Software-Testing-Standards

[4] Wikipedia: Circular Reasoning http://en.wikipedia.org/wiki/Circular_reasoning

[5] Webpage: ISO/IEC/IEEE 29119 Software Testing Standard http://www.softwaretestingstandard.org/index.php

[6] BCS - British Computer Society, page 8: http://www.bcs.org/upload/pdf/sreid-120913.pdf

[7] ISO/IEC Directives, Part 1 http://www.iso.org/sites/directives/directives.html

[8] ISO/IEC JTC1/SC7 3701