tag:blogger.com,1999:blog-59481415874229803382024-03-13T00:10:13.534+01:00The Tester's HeadacheLooking at issues affecting Software Development, with a special interest in TestingSimon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.comBlogger166125tag:blogger.com,1999:blog-5948141587422980338.post-1434599100341618122017-09-17T12:47:00.000+02:002017-09-17T12:47:13.694+02:00Testing as Feedback<u><b><i>Reflections and a thought experiment connected to testing</i></b></u><br />
<br />
I saw this tweet from Martin that triggered my thinking. I noticed it because the tweet was getting several responses and Martin poses an interesting question, as he usually does.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFmNX9MwEwgD7ebnAfZWuW7YMJqr3gqLe_256SHGV2k4Z5Ftsuwuk6e6ftcUGzo57pkO0uSb-eQ0rZsFMJqudD3ktZ6Pt6FaJhHEsgc-JIav-xqMDhe1ipGXSe3qHuHCj5zb7_BmKa4CVh/s1600/vds4.tiff" style="margin-left: 1em; margin-right: 1em;"><img alt="https://twitter.com/vds4/status/908270611465728001" border="0" data-original-height="177" data-original-width="514" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFmNX9MwEwgD7ebnAfZWuW7YMJqr3gqLe_256SHGV2k4Z5Ftsuwuk6e6ftcUGzo57pkO0uSb-eQ0rZsFMJqudD3ktZ6Pt6FaJhHEsgc-JIav-xqMDhe1ipGXSe3qHuHCj5zb7_BmKa4CVh/s400/vds4.tiff" title="" width="400" /></a></div>
<br />
<br />
I looked at a few of the replies. The testing community is full of insightful thinkers. These two caught my eye:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAsRMdlLJx7lzwFl0sQlIf3ikSaH8ob1FjfevDyxWWOVeXp1qtTw7JUfpGHuqjdoJXVwbJAWRx_5zEsftoaXsoazmAWCLr1WDDfPBM88B87pQe7mt5dvKBQH1GUPyKeKa3_GL4FXyFdZuV/s1600/testing-as-feedback2.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="346" data-original-width="517" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAsRMdlLJx7lzwFl0sQlIf3ikSaH8ob1FjfevDyxWWOVeXp1qtTw7JUfpGHuqjdoJXVwbJAWRx_5zEsftoaXsoazmAWCLr1WDDfPBM88B87pQe7mt5dvKBQH1GUPyKeKa3_GL4FXyFdZuV/s400/testing-as-feedback2.tiff" width="400" /></a></div>
<br />
<br />
Of course, I had to comment on this because it goes to the heart of any hypothesis based and social interaction development.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmXhbOvSpzta9gp8jRc-KLrsgWQDzzsFIUS2zNSNH2bHNr5LohxldxJtU1kS9eJoldv_Ho3rEu8iiIzzKHrg4ULsIXaH7q-dZ58uynW1yXnL0ZbD7pPMAviVziC23XQx0_TS11jHJuFojp/s1600/testing-as-feedback3.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="131" data-original-width="519" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmXhbOvSpzta9gp8jRc-KLrsgWQDzzsFIUS2zNSNH2bHNr5LohxldxJtU1kS9eJoldv_Ho3rEu8iiIzzKHrg4ULsIXaH7q-dZ58uynW1yXnL0ZbD7pPMAviVziC23XQx0_TS11jHJuFojp/s400/testing-as-feedback3.tiff" width="400" /></a></div>
<br />
<br />
Then I wondered how the replies were being formulated, and even how they were being received.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFG0hrVOjTn0HX54-U2kRyLmwf4SOb-B6CHSXHTOzIFsMkM6Wsw1_UpDmHwbniZxPW-4FdEdd5jtrYcfEu7OabbgOgFrIhRZL4v14Jyr431HkQXagWSbEIwYpURTLC8fCwR7p2FFAmnBkA/s1600/testing-as-feedback1.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="178" data-original-width="510" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFG0hrVOjTn0HX54-U2kRyLmwf4SOb-B6CHSXHTOzIFsMkM6Wsw1_UpDmHwbniZxPW-4FdEdd5jtrYcfEu7OabbgOgFrIhRZL4v14Jyr431HkQXagWSbEIwYpURTLC8fCwR7p2FFAmnBkA/s400/testing-as-feedback1.tiff" width="400" /></a></div>
<br />
Maybe I'm over-loading Martin's original intent, but I'm trying to extend his line of thinking, so bear with me.<br />
<br />
Based on Martin and Jari's questions, an approach to analyzing a situation is to consider an item (artefact), activity or set or interactions and consider what would the situation/system look like if you flipped the meaning 180 degrees; an opposite, or antonym. Then look at that resulting situation and consider do you learn anything about the original.<br />
<br />
<b>Stumbling block?</b><br />
The idea of what testing is and isn't and who does what for what purpose, is not clear. That was part of the reason for my "data question" - who is answering from which perspective and how does one know. If this was an anthropological or qualitative analysis study the original question might have been set up differently - but then it might not have gotten the same engagement or attention.<br />
Actually, I think the biggest stumbling block with the replies to the question posed is that the respondents are not indicating what they think of as testing for the "opposite of testing" answer, which is always going to be limited in twitter.<br />
<br />
As interesting as many of the replies are, it's difficult for me to use them to understand "testing" or the system of software development from the replies.<br />
<br />
That triggered my own thought experiment.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5MMQaqL5USyO_7HXXt-zgGSRijr-eJgwdfoRUG9AgCgZ2BXEYN-qWoEw0H2JBrucBPUoqRNOJTHkNYjJPUCWtGZptXdh54QvYLBxJ1KRogix9wNgWkAkPH4yi5Jr0RXKYVuN3xC-Tb856/s1600/testing-as-feedback4.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="154" data-original-width="512" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5MMQaqL5USyO_7HXXt-zgGSRijr-eJgwdfoRUG9AgCgZ2BXEYN-qWoEw0H2JBrucBPUoqRNOJTHkNYjJPUCWtGZptXdh54QvYLBxJ1KRogix9wNgWkAkPH4yi5Jr0RXKYVuN3xC-Tb856/s400/testing-as-feedback4.tiff" width="400" /></a></div>
<br />
<br />
For this experiment, I will not try to define testing directly, but I will use another element in software and product development to understand the potential impact of "no testing". So, the simplistic assumption is that the opposite of "testing" is "no testing" and that this will be observed by looking at other characteristics in a product development lifecycle, in this specific case <b>feedback</b>.<br />
<br />
<b>Approach</b><br />
A system can be analysed by altering a meaning, or purpose, for one of its components and see how the resulting new system might perform. Another way is to remove that component.<br />
<br />
Compare to a picture of a group of people - what does the picture tell you about the group of people, their interaction, their context or their potential. Now remove a person or an item. What changes about the whole system or story do you now see (search for photoshopping, green screening or photo manipulation for examples) - e.g. do you see things that were obscured by the now-removed object or person.<br />
<br />
<b>Why Feedback?</b><br />
It is difficult to look at a system of interactions that makes up product development and isolate components, activities and elements of output. For me, feedback is an interaction between people or information about a system or product - it can be internal within the system (company, teams) or external (information received or gleaned from outside the company or teams). Internal feedback - to me - has a value, care and thought attached to information or data points, maybe even discussion or talking points.<br />
<br />
Compare, "login doesn't work" to "login doesn't work under circumstances X, Y & Z" or "login performance has changed since version a.b.c". These are all potential forms of feedback - I claim that the second two have more value than the first. A team or product owner can react on all of these, but my claim is that it's easier to make decisions based on #2 & #3. <i>Note, I'm making no claims on who or what generates this type of feedback.</i><br />
<br />
Note, my experience is that <b>good testing</b> generates good feedback in systems of product development, which aid a variety of decisions and analysis - sometimes that comes from individuals and sometimes from systems that individuals have put into place to facilitate feedback and analysis.<br />
<br />
And so, riffing on testing being removed from software development and the impact on feedback:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmBhJB5V32c0Pu37ilP8Qvd29jzrpob86ogLx5lhGtsw1G6PIxbEwTygVCQUXrjk0kQCLgM4chn3lgQXJotl1IPGOHeGl__nGb-8cjo1j6Arwsa9ACoWgPqgNBbGdnIIJ_8mA7rlkzaoNL/s1600/testing-as-feedback5.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="153" data-original-width="512" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmBhJB5V32c0Pu37ilP8Qvd29jzrpob86ogLx5lhGtsw1G6PIxbEwTygVCQUXrjk0kQCLgM4chn3lgQXJotl1IPGOHeGl__nGb-8cjo1j6Arwsa9ACoWgPqgNBbGdnIIJ_8mA7rlkzaoNL/s400/testing-as-feedback5.tiff" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEio2N60BnlGzPybTb7mW7YlJ3hyphenhyphenk5xEBrGoT99wmqKsPNHByWvffRXCBMt0UVZ3qazPM4n1aVf5W8Snmq-YLofmNI0KJvO75jPKvhOM6dQ3D98ogPvKti8Y5ZxDEd-QdjABIvV7jgI6CzoZ/s1600/testing-as-feedback6.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="151" data-original-width="515" height="116" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEio2N60BnlGzPybTb7mW7YlJ3hyphenhyphenk5xEBrGoT99wmqKsPNHByWvffRXCBMt0UVZ3qazPM4n1aVf5W8Snmq-YLofmNI0KJvO75jPKvhOM6dQ3D98ogPvKti8Y5ZxDEd-QdjABIvV7jgI6CzoZ/s400/testing-as-feedback6.tiff" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRKxf8f-rJJHJ3NT9AUmKlHE3uEL3VIQo8smf42whcgEC9DBcs_WKcY7jrIHdw6RaFb9gXcA5mFc5OMO6LIPFWlLbwOhyIFanYgH0EBXrw6X-msPHsQlN2NmeVek0zOa8BX6zM3SXke9oR/s1600/testing-as-feedback7.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="126" data-original-width="510" height="98" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRKxf8f-rJJHJ3NT9AUmKlHE3uEL3VIQo8smf42whcgEC9DBcs_WKcY7jrIHdw6RaFb9gXcA5mFc5OMO6LIPFWlLbwOhyIFanYgH0EBXrw6X-msPHsQlN2NmeVek0zOa8BX6zM3SXke9oR/s400/testing-as-feedback7.tiff" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjExBRN0yPENv_ZjxBbYNA0l_nzQjqo3u8OWb-itubCHt57sZ6BIcjyLouOn3-fq-kBle7vshss00oBrOn_IGI6YsyHoJ4h4FQB8KBZ7-tn1BL_xBcPdtDVHe4JH53iqbaiNSnVQmT02W3u/s1600/testing-as-feedback8.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="150" data-original-width="505" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjExBRN0yPENv_ZjxBbYNA0l_nzQjqo3u8OWb-itubCHt57sZ6BIcjyLouOn3-fq-kBle7vshss00oBrOn_IGI6YsyHoJ4h4FQB8KBZ7-tn1BL_xBcPdtDVHe4JH53iqbaiNSnVQmT02W3u/s400/testing-as-feedback8.tiff" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJQfNdLuJ1oa8I_bcklhkPpbjfTNe-gLp-yWxaPyDj6f2QlZuIn6JlHIwux7mCZBaHZSoWpLJ4nH7g6JRhKnjEWJbbFDKR-hfwAjf-Wjy-6m493xV5uchLRZWUqRKLgu6kTXdhC1A8Oy90/s1600/testing-as-feedback9.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="155" data-original-width="512" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJQfNdLuJ1oa8I_bcklhkPpbjfTNe-gLp-yWxaPyDj6f2QlZuIn6JlHIwux7mCZBaHZSoWpLJ4nH7g6JRhKnjEWJbbFDKR-hfwAjf-Wjy-6m493xV5uchLRZWUqRKLgu6kTXdhC1A8Oy90/s400/testing-as-feedback9.tiff" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTKdEWMEjqNQr3Pa7sOe9GMI2RKBvL28LfTLnjj73jIlFaMhVmVkhuwRdTVe_w_Eee9VhG3j9iPpjAnCpgKEh71Oynqn_tisMtpPcfrEnk6SUcUQYZCMdkC-IzdtG1fr2QGG3uoGcARfmw/s1600/testing-as-feedback10.tiff" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="175" data-original-width="509" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTKdEWMEjqNQr3Pa7sOe9GMI2RKBvL28LfTLnjj73jIlFaMhVmVkhuwRdTVe_w_Eee9VhG3j9iPpjAnCpgKEh71Oynqn_tisMtpPcfrEnk6SUcUQYZCMdkC-IzdtG1fr2QGG3uoGcARfmw/s400/testing-as-feedback10.tiff" width="400" /></a></div>
<br />
<br />
<b>Reflections</b><br />
There are other aspects of product development that one could look at besides feedback.<br />
<div>
<br /></div>
<div>
The timing is implied in these tweets - the implication is that if it takes longer to get feedback and be able to make a decision about it then you (team, product owner or business manager) miss the opportunity to adjust course, make a different decision, conclude that an experiment doesn't / didn't work and adjust investment, etc.</div>
<div>
<br /></div>
<div>
My implication in these tweets is that testing provides valuable feedback to teams and companies. </div>
<div>
<br /></div>
<div>
Another implication is that feedback is in the system of product development and that ultimately it is a social interaction between individuals.</div>
<div>
<br /></div>
<div>
I do not make a claim that testers own feedback. Good testers can contribute to good feedback systems and delivery.</div>
<div>
</div>
<div>
Extending that: the proximity of feedback to when code is written is vital for hypothesis or experiment-based development.</div>
<div>
<br /></div>
<div>
I think I've probably generated a few question marks that need explanation and extension, but I will leave that to another post.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-74838784650260477132017-08-06T22:28:00.000+02:002017-08-06T23:45:34.272+02:00Quality-Value: Heuristic or Oracle?<div style="line-height: normal;">
<span style="font-family: inherit;">The other week <a href="https://twitter.com/qahiccupps" rel="nofollow" target="_blank">James Thomas</a> wrote down <a href="http://qahiccupps.blogspot.com/2017/07/in-two-minds.html" rel="nofollow" target="_blank">some thoughts</a> related to “Quality is value to some person”. I added some diverse thoughts - a mini-brain dump. This post is an expansion on one of those thoughts.</span></div>
<div style="line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><br /></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>Start point</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">I made the assertion that the heuristic/statement (“quality is value to some person”) had been primarily used in a software testing context as a tool to illustrate that stakeholders are the gatekeepers of quality and not necessarily testers.</span></div>
<div style="line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><br /></span></div>
<div style="line-height: normal;">
<span style="font-family: inherit;">To me this is ok - but ultimately not a proactive approach helping the team, company or organisation work out how to delight it’s current and potential customers with good/acceptable/superior value and quality.</span></div>
<div style="line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><br /></span></div>
<div style="line-height: normal;">
<b><span style="font-family: inherit;">Point to explore</span></b></div>
<div style="line-height: normal;">
<span style="font-family: inherit;">The point I made on James’ blog that I want to expand on is:</span></div>
<blockquote class="tr_bq">
<span style="font-family: inherit;"><span style="background-color: white; font-kerning: none;"><i>“4. In what scope is the “quality is value to some person” used in SW testing? I don’t know if it really matches Jerry’s original intent. I think it has been used to find a responsible stakeholder to discuss test results & objectives with, and probably also to aid testers to explain that they are not the sole arbiters of quality. </i></span> </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: inherit; font-kerning: none;"><i>4.1. I read the intent (from Jerry’s QSM) as highlighting a relationship and a perspective (i.e. subjective rather than objective) - which by nature can’t (usually) be static. I haven’t really seen/read of anyone applying it from this perspective to SW testing. I wonder how it would look…”</i></span></blockquote>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">So, the question I will explore- if this is a transient and subjective statement - what use can it have to software testing or software development as a whole? I make some claims (statements and opinions) and questions - partly re-anchoring the context in which the statement* could be used:</span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
</div>
<ol>
<li><span style="font-family: inherit;">Who is making the statement*?</span></li>
<li><span style="font-family: inherit;">When is it useful to make the statement*?</span></li>
<li><span style="font-family: inherit;">To what scope does it apply?</span></li>
<li><span style="font-family: inherit;">It’s bigger than software testing</span></li>
</ol>
<span style="font-family: inherit;"><br /></span>
<br />
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>Who is making the statement*?</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit;"><span style="font-kerning: none;">I assert that it makes no sense for </span><span style="font-kerning: none; text-decoration: underline;"><b>only</b></span><span style="font-kerning: none;"> certain people in a development team or project to have this view (“quality is value to some person”) - therefore, this is a team or project view of quality, or preferably a team together with a product owner or even customer view on quality.</span></span></div>
<div style="background-color: white; color: #666666; line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><span style="font-kerning: none;"></span><br /></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">This starts to imply a synchronised view on acceptable goals for a product, or feature, or vertical slice of a feature - as a goal for the team/group rather than individuals using it as a reminder (or in the worst case a defensive and passive position).</span></div>
<div style="background-color: white; color: #666666; line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><span style="font-kerning: none;"></span><br /></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>When is it useful to make the statement*?</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">I assert it is a useful reminder at the start of work (goals of a product, or feature) - these may be preliminary goals used in a prototyping activity or a hypothesis on what a customer might want. Doing this at the start is establishing a common goal for the team (or project or program, etc). This is not a statement useful for gatekeeping but useful for goal alignment - alignment of subjective views if you will.</span></div>
<div style="background-color: white; color: #666666; line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><span style="font-kerning: none;"></span><br /></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>To what scope does it apply?</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">I assert this applies to the whole development and delivery chain. Therefore, this implies that synchronisation between development and delivery teams (or even a DevOps set-up) would be desirable. Again, the alignment is about aligning common goals, not gatekeeping.</span></div>
<div style="background-color: white; color: #666666; line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><br /></span> <span style="font-family: inherit; font-kerning: none;"></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>It’s bigger than software testing</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">Hopefully, this point is obvious?</span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">Applying the statement* to product development (and delivery) I assert that it soon becomes clear(?) that it is much more than software testing and should be running through the whole chain. It doesn’t need to be a defence mechanism used by testers if alignment on goals has been achieved within the team, program, organisation, company.</span></div>
<div style="background-color: white; color: #666666; line-height: normal; min-height: 14px;">
<span style="font-family: inherit;"><br /></span> <span style="font-family: inherit; font-kerning: none;"></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="-webkit-font-kerning: none; font-family: inherit;"><b>Flip side? From Heuristic to Oracle</b></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">Suppose individuals or teams are using the statement* as a reminder/defence mechanism to illustrate that one or more stakeholders need to take a view on quality - what could this mean? I’d interpret it as a symptom of the team/organisation and its maturity with regards to delivering synchronised development & deployment quality. </span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;"><br /></span></div>
<div style="background-color: white; color: #666666; line-height: normal;">
<span style="font-family: inherit; font-kerning: none;">Another way to look at it: it’s a canary call for silos and local-optimisations. You could say it could be used as an oracle for spotting an organisational problem! More on that in another post…</span><br />
<span style="font-family: inherit; font-kerning: none;"><br /></span>
<span style="font-family: inherit; font-kerning: none;">*Statement: "Quality is value to some person"</span></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-72101357268827621232017-04-25T01:01:00.000+02:002017-04-25T08:48:06.914+02:00Where is Testing ...?<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">It's in people's nature to notice change and differences. It's also in people's nature to make assumptions (or stop and ask questions) when they expect something and, to them, it is missing.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
So, if when trying to optimise a number of teams working together for the benefit of customers, something people are familiar with is not obviously there, then people can get tripped up.</div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Due to this, it can become difficult to discuss and explore new models or approaches due to questions about what is missing - or, really, not visible, not obvious, not understood.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<blockquote class="tr_bq">
<span style="font-kerning: none;"><b>TL;DR -> Aim (Why): Delight Customers; Plan (How): Where to invest effort, observe and act; Execute (What): Experiments to perform and data to gather - to match the "How". These are universal "test skill" attributes - i.e. "testing" is everywhere in well-functioning product development, delivery and operation.</b></span></blockquote>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Example:</b> </span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Consider a model for product development, ideally optimised to get feedback from users of the product and work on customer needs. This can be many software components, many software systems and many configurations. Assume the aim for the company producing a product is: to delight the customers & users of the product.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVSQ-7cPfc4o87xfQkjfF9ww3LXd6JbQd1MX7BH5zCQ-ALrspveGkRN2JLtZgc5fnH2a9jhSG2tvDTP-T5IDZE_9H5Q9tu9yREJ22P9w3b9YM9d_fQ4nmvbBQ57FoZPzlqT0ZtcM_pdqbR/s1600/Product+Feedback+loop2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVSQ-7cPfc4o87xfQkjfF9ww3LXd6JbQd1MX7BH5zCQ-ALrspveGkRN2JLtZgc5fnH2a9jhSG2tvDTP-T5IDZE_9H5Q9tu9yREJ22P9w3b9YM9d_fQ4nmvbBQ57FoZPzlqT0ZtcM_pdqbR/s400/Product+Feedback+loop2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Product Inception, Development, Delivery and Monitoring</td></tr>
</tbody></table>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px; text-align: center;">
<br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Note, that this model can be applied on a team or company level.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Potential problem: Interpretation</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">If 10 people look and study this model there will be more than 10 interpretations. Very often this is a result of how someone <a href="https://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html"><span style="-webkit-font-kerning: none; color: #042eee;">frames the object to study</span></a> (or problem to solve). This can be a factor of where they "sit" in the model/company, what influence they have and what they "want" to do.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Potential question: Where's testing? </b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Some people might look at the model and wonder where is "testing" done? This can be a leading question - sometimes as a function of thinking of "testing" as separated from other activities, sometimes as a function of what someone is used to anchoring to. Sometimes it might be a worry - how do we understand the customer is getting what they asked for.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">I don't see this model reflects a particular development model or even a type of "testing school". Conversely, I'm not sure any testing school has put the work into supporting such a model (for optimising feedback from customers and working on customer needs).</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Potential Approach: Find a place for testing.</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">One way is to find how testing contributes to each box of the model. There is a trap with this - if the whole is not also considered (or at least adjacent boxes) then this approach will tend to a local-optimisation in each box and not necessarily between boxes. It's an approach that tends to place testing in boxes - in the extreme it creates separated testing boxes. In the ultra extreme it creates a standard for SW testing detached from SW development.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Potential assumption: Specialised testing is not needed.</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">If you can't see it in the model then it's not needed, right? But, note - I haven't spelled out product architectural and system design. It doesn't mean they are not needed. So, that leads to the question - what is the model conveying. This model is not a WYSIATI (what you see is all there is) model - or rather it is above a level of practices. </span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"><b><i></i></b></span><br /></div>
<blockquote class="tr_bq">
<span style="font-kerning: none;"><b><i>Ok, so what use is such a model????</i></b></span></blockquote>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"><b></b></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>My take?</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Yes, the model is on a very high level, but that's the point - use an example where it appears as though the thing you want to talk about is absent...</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><br /></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">When I discuss such a picture above and a discussion about testing comes up, my approach is, "think how testing helps each of the above boxes", or really think of the boxes containing:</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUa72cmyijD3JaUj7tuw4jE_3_IEwzunLPa3p3wXg9xmTNxYrUpwAw-eC88628IVmuOT5SBTaEkUqkEx9_6HM0u6HjrQnvsmCSnZ2d0eAcFOXFDGOmDp_Sn2M55NcpLATmjwns5tnDzt64/s1600/Observation-Hypothesis-Experiment-Sensemaking.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUa72cmyijD3JaUj7tuw4jE_3_IEwzunLPa3p3wXg9xmTNxYrUpwAw-eC88628IVmuOT5SBTaEkUqkEx9_6HM0u6HjrQnvsmCSnZ2d0eAcFOXFDGOmDp_Sn2M55NcpLATmjwns5tnDzt64/s400/Observation-Hypothesis-Experiment-Sensemaking.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><table cellpadding="0" cellspacing="0" style="margin: 0px 120.5px 8px; padding: 6px;"><tbody>
<tr><td style="padding: 4px 0px 0px; width: 400px;" valign="middle"><div style="font-family: Times; font-size: 12.8px; line-height: normal; text-align: center;">
<span style="-webkit-font-kerning: none;">Observation-Hypothesis-Experiment-Sensemaking</span></div>
<div>
<table cellpadding="0" cellspacing="0" style="margin: 0px 120.5px 8px; padding: 6px;"><tbody>
<tr><td style="width: 400.0px;" valign="middle"><div style="color: #042eee; font-family: Times; font-size: 16px; line-height: normal; text-align: center;">
<span style="font-kerning: none;"></span></div>
</td></tr>
<tr><td style="padding: 4.0px 0.0px 0.0px 0.0px; width: 400.0px;" valign="middle"><div style="font-family: Times; font-size: 12.8px; line-height: normal; text-align: center;">
<br /></div>
</td></tr>
</tbody></table>
</div>
</td></tr>
</tbody></table>
</td></tr>
</tbody></table>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">To give a number of questions, for example:</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Product in Use & User Feedback</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to observe or get data about the product in use? Getting data about customer opinions, complaints or new wishes and needs?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to make judgements and derive opinions about the data (form hypotheses)?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to create experiments to gauge and observe the product performance in use or product usage?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to evaluate the results of those experiments, data and results?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Product Backlog & Development</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How will observability of the product and product usage be prioritised, developed and in what circumstances?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How should the product architecture and supporting environment look to be observable?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How should the product architecture support (fast) feedback on product changes? (hypothesis)</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How should the supporting environment support product changes? (hypothesis)</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How should experiments be created to observe and gather data on the product, its usage and performance?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to understand the results and data and whether the experiments are giving data on the hypotheses?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Product Delivery</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to observe and understand product delivery and deployment?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to understand if a product delivery will delight or disappoint a customer (new or old)?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to create experiments to gather data on product delivery and potential response from customers?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How to evaluate the data from the experiments of product delivery? What does the experimental data indicate about product delivery and potential (or actual) customer reaction?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"><b><i></i></b></span><br /></div>
<blockquote class="tr_bq">
<span style="font-kerning: none;"><b><i>And finally.... the whole:</i></b></span></blockquote>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"><b></b></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Product Inception, Development, Delivery and Operation</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How do we observe product usage and customer satisfaction?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How do we create an understanding of what the customer wants and is happy with?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How do we create experiments to understand our understanding during development, delivery and operation? Do we have consistency of hypothesis through development & delivery?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">- How do we evaluate the data from development, delivery and operation to a consistent picture? Do we have data to help understand delivery to customers, customer perception, feedback to the product development teams? Do we have data to understand what can be improved?</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Why->How->What</b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Most of these questions are "how" questions. They are predicated on supporting a model that optimises feedback from a customer and providing a product that a customer wants - the why. The "what", the implementation, is the least important - although it is important.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Sometimes "where's testing?" questions are really about "what" rather than the purpose and meaning. This is a check observation to keep in mind.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>And So....</b></span></div>
<ol>
<li style="font-family: Times; font-size: 16px; line-height: normal; margin: 0px;"><span style="font-kerning: none;">Notice, all of the above might be more recognisable as test and fact-based advocacy (observations), test and fact-finding analysis and design (hypothesis & experiments), test and fact-finding execution (experiments and iterating on experiments) and test and (qualitative) data advocacy and reporting (sense making).</span></li>
<li style="font-family: Times; font-size: 16px; line-height: normal; margin: 0px;"><span style="font-kerning: none;">Notice, it is everywhere in the SW development, delivery and operations loop. You might want to be ultra-specialised and constrain your "test" advocacy-design-execution-reporting skills to a small subset of the whole. Or, you might realise that those same observation-hypothesis-experimentation-sensemaking skills are needed (and can be used) everywhere. The trick is to realise that, then to balance the amount of time you want to spend in a small subset of a product development activity - whether as a team, separate team or individual and balance those skills elsewhere.</span></li>
</ol>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b><i>So - the testing skill set tied to observations, hypothesis forming, experimentation and evaluation and sense making are vitally important all through the product inception, development, delivery and operations flow!! In my experience successful teams and organisations have these skill sets in multiple places, not isolated.</i></b></span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">Of course, if your practical skill set (or comfort zone) limits you to a small subset, you might want to work on expanding those boundaries - at least for the good of the people and teams around you.</span></div>
<div style="font-family: Times; font-size: 16px; line-height: normal; min-height: 19px;">
<span style="font-kerning: none;"></span><br /></div>
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;"><b>Potentially Related Post</b></span></div>
<br />
<div style="color: #042eee; font-family: Times; font-size: 16px; line-height: normal;">
<span style="color: black; font-kerning: none;">[1] <span style="-webkit-font-kerning: none;"><a href="https://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html">Framing: Some Decision and Analysis Frames in Testing</a></span></span></div>
<div style="color: #042eee; font-family: Times; font-size: 16px; line-height: normal;">
<span style="color: black; font-kerning: none;">[2] <a href="https://testers-headache.blogspot.se/2016/01/some-software-development-testing.html">Some Software Development & Testing Challenges 2016</a></span></div>
<div style="color: #042eee; font-family: Times; font-size: 16px; line-height: normal;">
<span style="color: black; font-kerning: none;"><br /></span></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-17396055738713760302016-06-19T17:15:00.000+02:002016-06-19T17:15:41.029+02:00Communication Brevity and ContextYou might've seen something like this:<br />
<blockquote class="tr_bq">
"it works != it will work"<br />"it doesn't work != it won't work"<br />"The product can work != the product [does|will] work"</blockquote>
Hmm, let's extend them a bit:<br />
<blockquote class="tr_bq">
"The product doesn't work != the product will never work"<br />"The product doesn't work != the product won't work for you and how you use it"<br />"It works != it works forever"<br />"It works != release it, no worries"<br />"It works != no more work is needed with this product"</blockquote>
These statements seem to be very context-detached - it's not easy to see who is involved or what is really meant.<br /><br />With so little context how much meaning can be drawn? Do airlines work like that? Do they say a plane can work or that it can reach the destination but refuse to say that it will reach the destination?<br />
<br />
What's happening here? "it works" and "it doesn't work" are conveying headline messages - maybe something I'd think of as feelings about a product. And maybe specific to what someone means - which also means that you can't read more into it if you don't know that person or know what they intend with specific words and phrases.<br />
<br />
Can they be interpreted as something else? Yes, of course! But that applies to a majority of statements - there's always a flip side that can be emphasised.<br />
<br />
<b>Communication & Brevity</b><br />
The words we use and with whom will (or should) differ. The interpretation will also differ. They are probably influenced by who we are talking with, shared history, how synchronised we are, dependent on which <a href="http://testers-headache.blogspot.com/2016/06/communication-paradigms-and.html">trading zone</a> we find ourselves in and what level of <a href="https://en.wikipedia.org/wiki/Interactional_expertise">interactional expertise</a> each party has.<br />
<br />
I'd assert brevity of communication works best when the people are "tuned in to each other". I'd also assert that this type of communication is a heuristic summary / short-hand.<br />
<br />
<i><u>Brevity</u></i><br />
Brevity of communication may have a place. But it's not a universal place. This doesn't mean it's bad, it means you should not lift it out of context and generalise as something always "good" or always "bad".<br />
<br />
It's a form of short-hand communication and typically (to me) an invitation to discussion or dialogue.<br />
<br />
<i><u>Example</u></i><br />
When I was working as a system integrator in the "noughties" (2005-9) I would be testing product deliveries or part deliveries together in different combinations for them to progress to a next stage of work. I'd often remark on certain combinations to my project manager as, "good to go", "no, no good". These are quite similar to "this combination works" and "this combination doesn't". In the context of that relationship and product development that was perfectly ok - with that manager and the teams that I knew.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcNltV0PWYkDtILSl0VigOvI9wlpF3Az-NOi0R7WujIRjtznAY5ivo8ainfXD-tHkEEK5Rx2pj6Njx8ufcbnhqfZ9U1Ja2sv_LZFD63HgvtzltHMn3i_OOxjCodsKv28fiyuhGOM6ZR3r7/s1600/ok.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="291" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcNltV0PWYkDtILSl0VigOvI9wlpF3Az-NOi0R7WujIRjtznAY5ivo8ainfXD-tHkEEK5Rx2pj6Njx8ufcbnhqfZ9U1Ja2sv_LZFD63HgvtzltHMn3i_OOxjCodsKv28fiyuhGOM6ZR3r7/s320/ok.png" width="320" /></a></div>
<br />
Note:<br />
<blockquote class="tr_bq">
1. The two "OK"'s sit in different trading zones of language - different contexts and different meanings.<br />
2. "OK" & "Not OK" are headlines - they don't really contain all communication. They would be accompanied by some form of report - whether verbal or not - of what was seen to work, what was not tested and where concerns might lie.</blockquote>
The people I worked with wouldn't be satisfied with "ok" or "not ok" on it's own. And I wouldn't be satisfied with giving such a simple message in total - i.e. it might be ok as a headline but not the whole story. Naturally, a team would want to know why something wasn't "ok" upon hearing "Not OK" and very often a team would want to know if something wasn't "ok" upon hearing "OK".<br />
<br />
To me these are characteristics of each party in the communication chain.<br />
<br />
This example might not look likes it's really relevant in the "devops" world but wherever there is a producer and receiver of a product (or information) I suspect the model applies. The content ("OK" or "Not OK" or "<something else>") is highly dependent on the context (who is involved, the different trading zones and different levels of interactional expertise).<br />
<br />
<i><u><b>Lessons:</b></u></i><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
1. Brevity of communication is very context-specific, and I'd assert has a personal element.<br /> 2. Communication content is driven by context - i.e. the vocabulary you use might well be different depending on how well you know someone.<br />3. Brevity of communication shouldn't be the totality of communication. It should be an intro to a discussion. So it can only ever be a headline.<br />4. Lifting brevity of communication into a generalised form is - to me - misusing and misapplying the example. I.e. it's incorrect to say if particular words between two parties isn safe or not.<br />5. Using a concise form of words detached from context as a good/bad usage example is misleading - and one should beware that it might be a a form of straw man argument, or <a href="https://en.wikipedia.org/wiki/Ludic_fallacy">lucid fallacy</a>.</blockquote>
</blockquote>
<b>Important: Content over Labels: Context drives Content</b><br />So, to me, in communication - the words and phrases used are extremely context-dependent. They depend on the parties communicating. The exchange of ideas and intent should be dictated by the people in the conversation, discussion or dialogue, an appreciation of their respective contexts and level of interactional expertise.<br />
<br />
<b>Related Posts</b><br />
<a href="http://testers-headache.blogspot.com/2016/06/communication-paradigms-and.html">Communication, Paradigms and Interactional Expertise</a><br />
<a href="http://testers-headache.blogspot.com/2010/09/lessons-in-clarity-is-this-ok-with-you.html">Lessons in Clarity: "Is this OK with you?"</a><br />
<a href="http://testers-headache.blogspot.com/2014/04/on-test-results-and-decisions-about.html">On Test Results and Decisions about Test Results</a><br />
<a href="http://testers-headache.blogspot.com/2012/08/testing-chain-of-inference-model.html">Testing Chain of Inference: A Model</a><br />
<a href="https://storify.com/YorkyAbroad/there-is-no-test-for-it-works-every-time-always">There is no test for "it works, every time, always"</a><br />
<br />
<br />Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-34582028168819071662016-06-05T16:04:00.000+02:002016-06-05T16:04:35.959+02:00Communication, Paradigms and Interactional ExpertiseOver several years I have been looking at communication involving testing, <span style="color: #042eee; font-family: "times"; font-size: 16px; text-decoration: underline;"><a href="http://www.slideshare.net/YorkyAbroad/test-reporting-tonontesters2010">here</a></span><span style="font-family: "times"; font-size: 16px;"> and <span style="-webkit-font-kerning: none; color: #551a8b;"><a href="http://testers-headache.blogspot.com/2010/09/lessons-in-clarity-is-this-ok-with-you.html">here</a></span></span> are examples from 2010, and it never ceases to amuse (or concern) me how little we communicate well; then come the knots we tie ourselves up in to cope with this lack of information and content (and indeed communication, discussion and relation to context).<br />
<br />
Occasionally the problem is brevity - information (context) is omitted. Other times it is a different paradigm - a different way of thinking about contexts. Sometimes it is a framing problem - not realising the background argument (or concern) someone has.<br />
<br />
I have constant contact with senior managers and I know that they are always interested in content and context - what does it (some piece of information) mean in our product's and company's situation? A stakeholder or manager wants to know how the information you have found relates to their business context.<br />
<br />
The last thing I want from a manager or stakeholder is an appreciation of "no added value" due to brevity, lack of content or context or because I don't engage their language or understanding of the situation. Something alone the lines of the Billy Madison clip:<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/5hfYJsQAhl0/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/5hfYJsQAhl0?feature=player_embedded" width="320"></iframe></div>
<br />
<br />
I will look at some issues with stripped out information (brevity) in a following post. In this one let's take a look at the differing views and paradigms that can make us stumble or not notice how communication is failing.<br />
<br />
<b>Paradigms & Different Views</b><br />
Paradigms - these are where different people hold different world views of something, for example what testing is, it's place in product development and what it can, can't and should do. We have probably met people with very different views and appreciation of the work we each do and where it fits into the bigger scheme of things.<br />
<br />
There are a few ways to approach this problem. One is to educate people into your way of thinking. This might be the optimal goal, but it isn't always realistic - especially with senior managers. It's not always possible if someone doesn't understand that there might be a different way to look at the topic.<br />
<br />
Another approach is to find common ground - a trading zone for language.<br />
<br />
Consider the case with three actors: How do three people communicate, one with an ISTQB-view of "words", another with a non-ISTQB view of "words" and a stakeholder (business leader) with his own understanding / view of words? Do you convert everyone to one view, or do you look for an alternative?<br />
<br />
To talk to a business manager about testing do you insist he must know all the terminology you wish to use, or do you look for an alternative?<br />
<br />
In 2007 Collins, Evans & Gorman [1] explored the idea of trading zones and interactional expertise. On trading zones:<br />
<blockquote class="tr_bq">
"<i><span style="font-family: "timesnewromanpsmt"; font-size: 12pt;">Peter Galison introduced the term `trading zone’ to the social studies of science.</span><span style="font-family: "timesnewromanpsmt"; font-size: 7pt; vertical-align: 5pt;"> </span><span style="font-family: "timesnewromanpsmt"; font-size: 12pt;">His purpose was to resolve the problem of incommensurability between Kuhnian paradigms: How do scientists communicate if paradigms are incommensurable?</span></i>"</blockquote>
I.e. how do the three actors communicate with each other in a reasonable way when they each operate in different views of the world (paradigms)? Hence the concept of a <a href="https://en.wikipedia.org/wiki/Trading_zones">trading zone</a> for language.<br />
<blockquote class="tr_bq">
"<span style="font-family: "times"; font-size: 12pt;"><i>Two groups can agree on rules of exchange even if they ascribe utterly different signifcance to the objects being exchanged; they may even disagree on the meaning of the exchange process itself. Nonetheless, the trading partners can hammer out a local coordination, despite vast global differences.</i></span>"</blockquote>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNQUv5lwwTQuCVW4LZuQbvYemYqldMgsdhQdOUBj-xKLYzWxsE6QV9HDzDyAAvxpfD_4RPtDIhkdbrTnVltZPN-qsQ5mlAe4mHDK3BSK39_pkNLEqFl1pt5BzBQT2eveIPBEKe1tOs9uyg/s1600/LanguageTradingZones.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNQUv5lwwTQuCVW4LZuQbvYemYqldMgsdhQdOUBj-xKLYzWxsE6QV9HDzDyAAvxpfD_4RPtDIhkdbrTnVltZPN-qsQ5mlAe4mHDK3BSK39_pkNLEqFl1pt5BzBQT2eveIPBEKe1tOs9uyg/s320/LanguageTradingZones.png" width="320" /></a></div>
And one important form of these trading zones is interactional expertise.<br />
<blockquote class="tr_bq">
"<span style="font-family: "timesnewromanpsmt"; font-size: 12pt;"><i>Interactional expertise is the product of a successful linguistic socialization. Although expressed as language alone, it cannot be too heavily stressed, interactional expertise is tacit knowledge-laden and context specific.</i></span><span style="font-family: "timesnewromanpsmt"; font-size: 12pt;"> </span>"</blockquote>
<b>Interactional Expertise</b><br />
Applying Collins’ view [2] of interactional expertise from the sociologist-scientist relation to a tester/developer-stakeholder relation [<i>italics and </i><strike>strikethrough</strike><i> are my application</i>]<br />
<blockquote class="tr_bq">
“… where interactional expertise is being acquired, there will be a progression from “interview” to “discussion” to “conversation” as more and more of the <strike>science</strike> [<i>business/stakeholder context</i>] is understood.” </blockquote>
<blockquote class="tr_bq">
“ Above all, with interactional expertise, conversation about technical matters has a normal lively tone and neither party is bored. As things develop the day may arrive when, in response to a technical query, a <strike>respondent</strike> [<i>stakeholder</i>] will reply “I had not thought about that,” and pause before providing an answer to the <strike>sociologist’s</strike> [<i>developer/tester’s</i>] technical question. When this stage is reached, respondents will start to be happy to talk about the practice of their science [business context] and even give studied consideration to critical comments. Eventually <strike>respondents</strike> [<i>stakeholders</i>] will become interested in what the <strike>analyst</strike> [<i>developer/tester</i>] knows about the field because he or she will be able to convey the <strike>scientific</strike> [<i>business context</i>] thoughts and activities of others in a useful way. ”</blockquote>
<blockquote class="tr_bq">
“Where there is no developing interactional expertise […] the conversations never become interesting to either party, the analyst [developer/tester] can never transmit information, take a devil’s advocate position or, crucially, distinguish between “pat” answers and real conversational interchange, nor between jokes and irony on the one hand and serious responses on the other. Worse still, though a field might be riven with controversy […] the analyst cannot understand what the protagonists are disagreeing about, nor how deep the disagreements run, nor, with any certainty, who disagrees with whom! ”</blockquote>
Note, this relationship goes both ways - stakeholder <-> developer/tester. Over time the stakeholders / business leaders worth their salt will develop their interactional expertise to talk with the product developers / testers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3VRk0jWa_oR0SNVHyLS-mjo-ZFhz3PwFl9_ALtrI5bVdqjDoR0wlM0uKm33C9mzmYbLKcuyFBxcnXKwjI3OuZR3A2AMsU89WNj18l14QFkjgV4Q3B0K-cKgZY4RIiJtaMPDy_MgSMoF9f/s1600/InteractionalExpertise2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3VRk0jWa_oR0SNVHyLS-mjo-ZFhz3PwFl9_ALtrI5bVdqjDoR0wlM0uKm33C9mzmYbLKcuyFBxcnXKwjI3OuZR3A2AMsU89WNj18l14QFkjgV4Q3B0K-cKgZY4RIiJtaMPDy_MgSMoF9f/s320/InteractionalExpertise2.png" width="320" /></a></div>
<br />
<br />
<b>Importance of these concepts in your daily work</b><br />
Some important points to highlight:<br />
<ul>
<li>Acknowledging (or being aware) that language trading zones exist between different people or stakeholders then a natural way to cope with this is to advocate a greater need for interactional expertise.</li>
</ul>
<ul>
<li><i><b>Plainly:</b></i> Testers need greater interactional expertise in dealing with stakeholders and business leaders. This means understanding their concerns, how they think about problems, listening for signals in which pieces of information they react to (and don't) - why are some pieces interesting. To a certain extent it means translating information into their language. What do the risks in your testing mean to them and the customers? How might they word or translate it?</li>
</ul>
<ul>
<li>What type of information do they need to translate and why. It's not usually just about - can a product version release in the next quarter, sometimes it's also about understanding their risk picture - and what other contingencies they might need to prepare for.</li>
</ul>
<ul>
<li>In fact, if you can talk with and listen to your business leaders more and more - understand how they think and talk and what their concerns are - you will get familiar with the trading zone language. For example, why do they emphasise certain ideas and concepts - and how does my work relate to that?</li>
</ul>
<ul>
<li><b><i>Tip:</i></b> Ask your stakeholders or business leaders if they are using language or concepts you don't understand.</li>
</ul>
<ul>
<li><b><i>Corollary#1:</i></b> Business leaders naturally need to listen to their product development organisation - for information about how they talk, which information they emphasise. In general, business leaders are more adept (used to) doing this - as they need to communicate cross-business.</li>
</ul>
<ul>
<li><b><i>Corollary#2:</i></b> There is no one right way to talk about a subject if two people are located in different paradigms (or even namespaces) - without acknowledging the need of language trading zones and even understanding which context the other person is grounded.</li>
</ul>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiueFSpc7ebl6erQ3y4f4DmcbRnMch01jwRJGFOrjKF9D0Sq5Mj2Z1T3O-WY4sf10jiZHtKmswVaaR_cMv5OPPgnzQAzgkeIJyhQ9Irl1-BhasXKDo7Un3dy3tg4MHaSFDh962VXP6eS-PL/s1600/TestParadigms.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="169" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiueFSpc7ebl6erQ3y4f4DmcbRnMch01jwRJGFOrjKF9D0Sq5Mj2Z1T3O-WY4sf10jiZHtKmswVaaR_cMv5OPPgnzQAzgkeIJyhQ9Irl1-BhasXKDo7Un3dy3tg4MHaSFDh962VXP6eS-PL/s320/TestParadigms.png" width="320" /></a></div>
<br />
<br />
<b>Potentially Related Posts</b><br />
<a href="http://testers-headache.blogspot.com/2016/05/some-communication-patterns.html">Communication Patterns</a><br />
<a href="http://testers-headache.blogspot.com/2016/02/the-conway-heuristic.html">The Conway Heuristic</a><br />
<div>
<br /></div>
<b>References</b><br />
<div style="font-family: Times; font-size: 16px; line-height: normal;">
<span style="font-kerning: none;">[1] Collins, Harry, Evans, Robert and Gorman, Michael. 2007; Studies in History and Philosophy of Science Part A -- Trading Zones and Interactional Expertise</span></div>
[2] Collins. 2007. Rethinking Expertise (Chap 1, “Origins of Interactional Expertise”)Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-36537109144881600642016-05-28T10:17:00.000+02:002016-05-28T10:17:03.890+02:00Some Communication Patterns<div>
Communication is fundamental. I've been visiting it recently (see related posts below).</div>
<div>
<br />
Good product or software development generally has good communication involved. Yes, you will alway find outliers that are specialists at something that have poor communication skills. But, in my experience, the best managers, the best developers, the best architects, the best testers have good to very good communication skills.<br />
<br /></div>
<div>
In the late nineties I read "The User Illusion" [1] - a difficult read but one I found enlightening. One of the main points I took away from it was how we communicate, how we exchange and discard information and some of the pre-requisites to information exchange.</div>
<div>
<br /></div>
<div>
There is an implicit need to synchronise before we really exchange information and communicate. Synchronise - to understand (to some extent) the context of the person(s) one is communicating with.<br />
<blockquote class="tr_bq">
Tip: Remember this!</blockquote>
</div>
<div>
In the last 6-7 years I've encountered other models - the Satir Interaction model [2], the idea of dialogues as a means to understanding [3] and even ideas around idea recording [4]</div>
<div>
<br /></div>
<div>
<b>Communication Skills or System of Communication?</b><br />
Communication skills are not about broadcasting messages, or being loud - prepared to stand on a soapbox or just being talkative or even argumentative.<br />
<br /></div>
<div>
Here, for communication skills read: the set of skills to help someone be understood - discussing an idea or message in a way appropriate to the other person/people, and also to listen and reflect on what the other person/people are saying.<br />
<br />
A system of communication is the interaction - where an idea gets refined or examined and how. So people can have great communication skills but the communication (system) doesn't work for a number of reasons.<br />
<br />
<span style="font-size: large;">Spotting dysfunctional communication</span></div>
<div>
<br /></div>
<div>
<b>Some Communication Anti-Patterns [and Antidotes]</b></div>
<div>
<br /></div>
<div>
- Being fixed on one explanation.<br />
<blockquote class="tr_bq">
[Make alternative explanations visible or ask for alternative explanations. See Weinberg's "rule of three"]</blockquote>
</div>
<div>
- Listening for a gap rather than listening to understand. Waiting to speak, make your own statement rather than digging into what the person is saying, exploring and understanding.<br />
<blockquote class="tr_bq">
[Try, "Tell me more", "why do you say that?", "should I try to explain my reasoning?", "shall we follow my train of thought so we understand what it's grounded in?", "shall we take this discussion at another time?"]</blockquote>
</div>
<div>
- Not asking for explanations or giving examples.<br />
<blockquote class="tr_bq">
[Try, "Can you give an example?", "Shall I try to give an example in use?", "Is it clear or could it be misinterpreted without example?" ]</blockquote>
- Hidden frames, assumptions or agendas. This can be the case that someone is following a particular line of thought or argument and doesn't want to divert from that.<br />
<blockquote class="tr_bq">
[Try, "can you explain the concern or importance for this particular idea". Tip: what is your own set of frames, assumptions and implications? Are they clear or hidden?]</blockquote>
- Dismissive elements, e.g. "that's a rubbish/stupid idea/thought".<br />
<blockquote class="tr_bq">
[This is probably a symptom or result of one of the above patterns. Tip: take a break, pause and reflect.]</blockquote>
</div>
<div>
<b>A Communication Checklist</b></div>
<div>
<br /></div>
<div>
- Did I understand what the other means?<br />
<blockquote class="tr_bq">
[Are we "synchronised", do I understand their context?]</blockquote>
</div>
<div>
<blockquote class="tr_bq">
Tip: Ask for help? "I don't understand", "can you give an example"]</blockquote>
- Is there a value (or risk) in this person's idea/opinion?<br />
<blockquote class="tr_bq">
[What's their frame of reference? Can they contribute something I've missed?]</blockquote>
Wait, that's rather a short checklist?????? If you combine these and iterate on the communication anti-patterns above, it might be all you need. [Challenge: add & refine this!]<br />
<br />
<b>Implications</b><br />
The motivation (goal) should be to understand rather than change. If you start from the premise that, "this person is wrong" you're probably not open to the signals (consequences) of a particular line of thought.<br />
<br />
Communication is to exchange, spread and refine ideas. And I assert that "healthy" communication is subject to the scientific approach.<br />
<br />
If you take a "scientific" approach then you are examining data/ideas and understanding if they change your own ideas or ways in which they should be expressed. There's no way you can know your ideas are correct for ever. Anyone can bring a point of view, perspective or consequence that hasn't been examined before.<br />
<br />
If someone brings an idea that has "bad" consequences (from your perspective) then point it out - demonstrate it.<br />
<br />
But, what about the "crazies" or people who won't listen to reason? Well, if you've pointed to your reasoning and demonstrated your case (check: have you?) and are still convinced that your idea/opinion is correct or better --> then either walk-away or stick around and put up with it (if the value/potential gain of sticking around is greater than the risk of walking away).<br />
<blockquote class="tr_bq">
This applies to personal communication between friends/colleagues and between you and your manager/stakeholder in the workplace. </blockquote>
<b><i>Note:</i></b> I listen out for bad ideas - not necessarily to confirm the correctness of my own ideas (there is always a risk for this). Rather, it can be a useful tool to look for flaws in ideas and arguments. That's feedback on your own ideas and the way that you have tried to spread them. It's useful feedback on whether your own system of communication works.<br />
<br />
If you don't take a "scientific" approach - what are you doing? Are you creating a belief system, cult or echo chamber? There are plenty of those....<br />
<br /></div>
<div>
<b>Related Posts</b></div>
<div>
<a href="http://testers-headache.blogspot.com/2016/02/the-conway-heuristic.html">The Conway Heuristic</a></div>
<div>
<a href="http://testers-headache.blogspot.com/2016/04/communication-heuristic-use-cases.html">Communication Heuristic: Use Cases</a><br />
<a href="http://testers-headache.blogspot.com/2016/02/testing-what-was-question.html">Testing. What was the question?</a></div>
<div>
<b><br /></b>
<b>References</b><br />
[1] <a href="http://www.amazon.com/User-Illusion-Cutting-Consciousness-Penguin/dp/0140230122/">The User Illusion (Norretranders)</a></div>
<div>
[2] <a href="http://www.donaldegray.com/debugging-system-boundaries-the-satir-interaction-model/">Example of the Satir Interaction model (Gray)</a></div>
<div>
[3] <a href="http://www.amazon.com/Dialogue-Skill-Tacit-Knowledge-Goranzon/dp/0470019212/" style="color: #7d181e; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; text-decoration: none;">Dialogue, Skill & Tacit Knowledge (Göranzon, Hammaren, Ennals)</a></div>
<div>
[4] <a href="http://www.amazon.com/Protocol-Analysis-Revd-Verbal-Reports/dp/0262550237/ref=sr_1_1?s=books&ie=UTF8&qid=1464422699&sr=1-1&keywords=Protocol+Analysis%3A+Verbal+Reports+as+Data">Protocol Analysis: Verbal Reports as Data (Ericsson, Simon)</a><br />
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-22087476034167396232016-05-15T15:14:00.000+02:002016-05-15T15:14:09.247+02:00A Thought Experiment on Definitions<a href="https://en.wikipedia.org/wiki/Thought_experiment">Thought experiments</a> are a very powerful tool. You probably use them a lot without realising! Any time you wonder "what would happen if..." or "what is a possible consequence of that?" then you're making your own mini thought experiment.<br />
<br />
Einstein used them to develop his ideas around relativity. A recommended documentary on this <a href="https://www.youtube.com/watch?v=vk3KrP5F1Ao">here</a>.<br />
<br />
I recently posed a thought experiment on twitter<br />
<br />
<span style="background-color: white; color: #292f33; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 26px; letter-spacing: 0.25999999046325684px; white-space: pre-wrap;">thought experiment: what happens when one agrees on purpose behind a definition but not agree on it's usage..</span><br />
<br />
There are a number of layers to this question.<br />
<ul>
<li>Purpose</li>
<li>Definition</li>
<li>Agreement</li>
<li>Usage</li>
</ul>
<b>Purpose</b><br />
This is the "why?" question. What problem are you trying to solve, and does the definition and usage examples help solve it?<br />
So, what might happen if a usage of a definition doesn't appear to agree with it's intent, i.e. they are not congruous.<br />
<br />
<b>Definition</b><br />
This is getting into the <a href="https://en.wikipedia.org/wiki/Fallacies_of_definition">correctness</a> and relevance area. Is the definition too narrow or broad? Is it circular? Is it complex to understand. Is there some guidance to help understanding?<br />
<br />
<b>Agreement</b><br />
Complex or obscure definitions may be harder to agree with. Is the definition accessible, useable and <a href="https://en.wikipedia.org/wiki/Fallacies_of_definition">congruous</a>. Is there controversy or disagreement? Is that due to the purpose-definition-usage parts not being in synch? Is the definition generally accepted - de facto agreement?<br />
<br />
<b>Usage</b><br />
Is it clear how such a definition would and wouldn't be used? Are there any examples, or patterns and anti-patterns of usage somewhere - or indeed any guidance at all.<br />
<br />
It's not necessary for a definition to have usage examples or guidance. But it might help the case. Think about dictionaries - do they often, usually or seldom include examples of usage or guidance notes? (I think the answer would, of course, vary with the dictionary used.) This question would seem to be more relevant if the definition is complex or is difficult to accept.<br />
<br />
<b>What might symptoms of non agreement between definition and usage look like?</b><br />
<ul>
<li>Dislike of the definition (fit for purpose? relevance?)</li>
<li>Aversion or uneasiness with the definition (understanding, clarity?)</li>
<li>Misuse of the definition (understanding, clarity?)</li>
<li>Non-use (relevance, clarity, understanding?)</li>
</ul>
<div>
<b>Conclusion</b></div>
<div>
To me there are a number of consequences if such a contradiction crops up between usage and definition.<br />
<ul>
<li>The definition is not clear or complete.</li>
<li>The usage of the definition is not clear or illustrated.</li>
<li>The definition is misunderstood.</li>
<li>The definition is communicated in a way that doesn't align with the definition.</li>
<li>There is resistance to the definition and/or usage - emotional response.</li>
<li>There is resistance to the definition and/or usage - different paradigm.</li>
<li>There is resistance to the definition and/or usage - different dictionary references.</li>
<li>There is resistance to the definition and/or usage - frames of reference.</li>
<li>There is resistance to the definition and/or usage - little value add visible.</li>
<li>A combination of the above or even something else.</li>
</ul>
</div>
<div>
So, good definitions are generally robust. Unfortunately in the world of software testing many definitions would fail a lot of these tests above. Go look in the ISTQB Standard Glossary of Terms used in Software Testing and try it. Do you find any terms that "don't add value"? </div>
<div>
<br /></div>
<b>Example?</b><br />
Ok, so if I wanted play the school ground bully and pick on the weak I'd start with the ISTQB glossary, but I have higher intellectual ambitions, so...<br />
<br />
I've been thinking about <a href="http://testers-headache.blogspot.com/2016/04/thoughts-around-label-checking.html">checking</a> recently, let's try there.<br />
<br />
<b>Checking</b><br />
I would say I have had a certain uneasiness with the definition - for reasons I don't think I've always been able to articulate. This could boil down to my understanding or the clarity of the definition or something else.<br />
<br />
It could be that this feeling is also reflected elsewhere - as recently appeared on the <a href="http://www.softwaretestingclub.com/forum/topics/the-art-of-icky-and-good-words-in-software-testing">software testing club.</a> The reasons others may give for their "Icky feeling" may be unconnected from my observations, but it would be interesting for them to give their reasons.<br />
<br />
Ok, so let's take the <a href="http://www.satisfice.com/blog/archives/856">checking definition</a> from RST:<br />
<blockquote class="tr_bq">
<strong style="border: 0px; color: #373737; font-family: Georgia, 'Bitstream Charter', serif; font-size: 15px; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Checking</strong><span style="border: 0px; color: #373737; font-family: "georgia" , "bitstream charter" , serif; font-size: 15px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"> is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.</span></blockquote>
<ul style="border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; list-style: square; margin: 0px 0px 1.625em 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">“evaluations” as a noun refers to the product of the evaluation, which in the context of checking is going to be an artifact of some kind; a string of bits.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">“algorithmic” means that it can be expressed explicitly in a way that a tool could perform.</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"><span style="font-family: inherit; font-style: inherit;">“specific observations” means that the observation process results in a string of bits (otherwise, the algorithmic decision rules could not operate on them).</span></li>
</ul>
Now let's apply it to a stochastic process - eg speech recognition.<br />
<br />
According to the definition I can make specific observations (samples of audio) and apply an algorithm to them (for example a speech recognition algorithm). The interesting thing here is that the result is non deterministic (due to speech/accent/pronunciation variation - making the test data design problem difficult) and is going to need some engagement - both for input threshold parameters and analysis of the output. I might get a boolean output (match/no match) or I might get a range (78% match) - and that is a function of the input parameters and the specific observations I ran the algorithm with.<br />
<br />
Now the actual algorithm that is making the comparisons is the "checking" part of the process. But this becomes a very small part of the whole - because I need to put effort (more effort and time than the algorithm takes) beforehand and afterwards.<br />
<br />
To make this example fit into the current definition I'd have to have all possible samples for certain speech snippets (infinite) or I'd have to define the sample population (this is the test design part of the process - by implication this is part of "testing"). (I won't get into the problematics of the sampling mechanism I use.) So, I'm narrowing the checking part of the whole even more.<br />
<br />
So, the question becomes (for me) - should I only use checks where I am certain of the wanted outcome - i.e. a binary answer (which might be "yes/no", "pass/fail", "above 78% threshold/not above 78% threshold"). And here's the problem - I'm quite happy to use scripts as change checkers - or early/leading indicators - they are a mechanism to draw my attention to a result and then ask a question, "should I investigate more or what does this result tell me?". As soon as I am paying attention to the result or thinking about it I am not checking anymore - that's testing.<br />
<br />
In this example, checking becomes a very small part of the whole - compared with all the other parts of requirement and test analysis, test design, test set-up and result analysis that make up testing. Then I wonder what value it really adds.<br />
<br />
Am I using the definition incorrectly? I don't see any <u>usage examples anywhere</u>, so maybe the <u>definition is incomplete</u>. Or maybe guidance is incomplete. Or maybe the terminology is just not useful for me.<br />
<br />
Divergent thought: In the definition of checking it's not clear to me if the algorithm can be a non deterministic algorithm. It could be read in that way - then here's another thought experiment --> what would the consequences of that be?<br />
<br />
If I was to revisit the <u>purpose</u> and <u>intent</u> behind this definition I'm not sure that it achieves what it wanted. The checking part is quite small - the other activities in testing are not described so the importance of checking seems to be artificially increased. This is a problem! To me, it would be better to list different tactics of test execution and highlight that checking is one of them.<br />
<br />
So, in this example, the "checking" is a very small part of the whole and falls into (for me) a very narrow definition, <a href="http://www.developsense.com/blog/2016/04/you-are-not-checking/">with a certain amount of ambiguity.</a> (It's narrow as it is contrasted with testing. This is analogous to a "testing vs test design post".) The definition is incomplete and/or incongruous (no usage example and generates confusion and discussion) and fails to add value (as it seems to artificially inflate the importance of checking in relation to other testing activities).<br />
<br />
Note, it's taken me quite a while to come to this conclusion - I have needed to put an amount of time thinking around this. It's certainly not an obvious conclusion. And I can also understand if others don't have the time, energy or inclination to do this type of thought journey and treat it as a heuristic to help in their communication. And I also understand that this term is helpful for some people and they have success in using it with their stakeholders - again if this heuristic communication works for you - fine.<br />
<br />
<b>Final word</b><br />
<span style="color: red;">It seems to me that there are many definitions around in the testing and software testing community that could benefit from this type of approach. Do you agree? Which would you try it on first?</span><br />
<br />
<b>Potentially Related Posts</b><br />
<br />
<a href="http://testers-headache.blogspot.com/2016/02/the-conway-heuristic.html">The Conway Heuristic</a><br />
<a href="http://testers-headache.blogspot.com/2016/04/communication-heuristic-use-cases.html">Communication Heuristic: Use Cases</a><br />
<a href="http://testers-headache.blogspot.com/2016/04/thoughts-around-label-checking.html">Thoughts around the label "Checking"</a><br />
<br />Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-16949496346221892182016-04-24T20:18:00.000+02:002016-04-24T20:33:32.067+02:00Communication Heuristic: Use CasesIn the last year I've been working with different aspects of software development - many aspects related to Continuous Integration and Testing, but also other areas. Part of this work has involved developing new ideas to test, reflect on and potentially spread. A common challenge I've had from a number of colleagues when I'm talking about an idea or concept is:<br />
<br />
"Can you give me or describe it in a use case?"<br />
<br />
People want to relate to the idea. Sometimes a user story is really meant, but that doesn't matter. <b>The real power here is the invitation to a discussion and dialogue</b> about the topic, concept, way of working, product, etc. It's a way of saying. "let me understand your idea in use".<br />
<br />
The use case doesn't necessarily describe the entirety of the idea but it starts the discussion - at least from one angle.<br />
<br />
It's not always easy to do either - because sometimes it generates discussions in unexpected areas. This might be because people interpret the need differently. Or they've framed the problem differently. But that is good, and indeed useful, to generate such a discussion. It helps weed out misunderstanding.<br />
<br />
When people (and groups) have worked with (or thought about) the idea then they will naturally develop new ideas about it or generate new questions. Some of this is testing the idea or concept, sometimes it's information gathering and sometimes it's clarification. Usually the testing of the idea explores ways that it could be misunderstood or produce unwanted results.<br />
<br />
This is a very useful tool not only for product development but communication in general. It is common to use this in product specification and requirement capture, but it's also very useful in concept/idea discussion.<br />
<br />
<span style="color: red;">It is a heuristic approach to communication.</span><br />
<span style="color: red;"><br /></span>Does this all seem abstract? I used this heuristic recently to discover that I didn't have a common understanding - at least via use case - for the usage of the word "checking".<br />
<br />
<b>Potentially Related Posts</b><br />
<a href="http://testers-headache.blogspot.com/2016/02/the-conway-heuristic.html">The Conway Heuristic</a><br />
<a href="http://testers-headache.blogspot.com/2016/02/testing-what-was-question.html">Testing. What was the question?</a><br />
<a href="http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html">Framing: Some Decision and Analysis Frames in Testing</a><br />
<a href="http://testers-headache.blogspot.com/2016/04/thoughts-around-label-checking.html">Thoughts around the label "Checking"</a>Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-73268036912643406812016-04-24T17:35:00.002+02:002016-04-24T17:35:54.038+02:00Thoughts around the label "Checking" Observant readers of twitter and my occasional (6) blogs related to checks, testing and checking will know that I have done some head-scratching over the term "checking".<br />
<br />
<b>Some of My Heuristic Triggers</b><br />
Why, what has triggered this for me? I have a number of factors - mainly these heuristic sources:<br />
<blockquote class="tr_bq">
Are You're Lights on? [1], Reason for testing (term usage) [8], Framing testing terms [9]<br />
<i>--> My questions: What problem are we trying to solve? And does this succeed?</i></blockquote>
<br />
<blockquote class="tr_bq">
Weinberg's Rule of Three [2] and the Conway Heuristic [5]<br />
<i>--> My questions: What alternative interpretation is there? How could this be interpreted, or what consequences might it create (unintended or not)? How could it be misinterpreted?</i></blockquote>
<br />
<blockquote class="tr_bq">
Communication, Exploring Requirements [3], Dialogue & tacit knowledge [4], Understanding Arguments [6] and Heuristic wear-n-tear / refinement [7]<br />
<i>--> My questions: Do the statements & arguments stand up? Does something get lost in interpretation?</i></blockquote>
<b><br /></b>
<b>Towards Clarity (for me)</b><br />
These are usually with me in my toolkit as aids - some more than others. So, I read Michael Bolton's recent post [11] on checking and what isn't checking - this got me thinking on this topic again.<br />
<br />
As I stated on his blog - after a challenge for clarity from Thomas Ponnet:<br />
<blockquote class="tr_bq">
Checking <i>"... is a label that should only be used (imo) for an observation of something that is happening or has happened. It’s a post event rationalisation (it’s “a posteriori” knowledge). </i> </blockquote>
<blockquote class="tr_bq">
<i>If you state your intention to include checking in your activities you are really describing your testing – because the intent, analysis, selection and discussion of results is testing – even if checks were used.</i> </blockquote>
<blockquote class="tr_bq">
<i>Then I would find it more accurate to talk about the testing that /will be/ aided by checks. Afterwards it might be accurate to describe the testing has included checking. Planning checking, intending checking is by implication testing – and I (personally) don’t see the added value and I question it’s accuracy.</i> </blockquote>
<blockquote class="tr_bq">
<i>But, that doesn’t, of course, stop anyone using the term however they want and in ways they find useful."</i></blockquote>
I am making an assertion for how checking might be used in a less-ambiguous way.<br />
<br />
<b>Shallow-Agreement?</b><br />
I was challenged that I might be in shallow-agreement with the RST meaning of checking. Of course, this is always possible - I went and re-read the testing-checking-refined [10] and Michael's post [11]. I didn't find any examples of the usage of "checking" (as of 23 April 2016) - i.e. examples of how it would be used in speech or written form.<br />
<blockquote class="tr_bq">
<b><i>Thus, there is a risk for shallow agreement with something that isn't demonstrated. Whether that is small or high, how would you know?</i></b></blockquote>
<blockquote class="tr_bq">
<b>A couple of questions then occurred to me:</b> </blockquote>
<blockquote class="tr_bq">
<ul>
<li>What's the guide for shallow agreement on the definition and use of "checking"? </li>
</ul>
</blockquote>
<blockquote class="tr_bq">
<ul>
<li>What's the accepted form for agreement when the main post doesn't demonstrate it?</li>
</ul>
</blockquote>
<b>Does it matter?</b><br />
What have I been doing? I've been testing the concept of "checking" - trying to understand ways it will work and risks associated with its usage. I've given examples (in the blog [11]) of potentially unintended consequences of its usage.<br />
<br />
Would examples and the above guide for "checking" help? Maybe, maybe not. It might be that the problems I have highlighted "don't really matter" or isn't a "high priority issue". That's fine, I can live with that. It's possible I'm getting stuck in the semantics... Oh!<br />
<br />
<b>My context/background: </b><br />
I started writing, questioning and exploring these issues in September 2009. I was one of the vocal parts of the discussion that led to the re-drawing/refinement of the diagram in the testing-checking-refined post [10].<br />
<br />
I've called out and questioned people - typically on twitter - that might be using "checks" and "checking" in unsafe ways. I don't typically use the word "checking" in my work - partly due to some worries I've seen in misunderstanding - and also that I can distinguish between testing and checks without "checking".<br />
<br />
<span style="color: red;">Not using "checking" myself doesn't mean I can't usefully "test" it, its usage and risks associated with its usage. Can you? And if so, what heuristic guides would you use?</span><br />
<br />
<b>References</b><br />
[1] <a href="http://www.amazon.com/Are-Your-Lights-Figure-Problem/dp/0932633161/">Are Your Lights on? (Gause, Weinberg)</a><br />
[2] <a href="http://www.amazon.com/Secrets-Consulting-Giving-Getting-Successfully-ebook/dp/B004J35LHQ/">The Secrets of Consulting (Chapter 5; Weinberg)</a><br />
[3] <a href="http://www.amazon.com/Exploring-Requirements-Quality-Before-Design/dp/0932633137/">Exploring Requirements: Quality before Design (Gause, Weinberg)</a><br />
[4] <a href="http://www.amazon.com/Dialogue-Skill-Tacit-Knowledge-Goranzon/dp/0470019212/">Dialogue, Skill & Tacit Knowledge (Göranzon, Hammaren, Ennals)</a><br />
[5] <a href="http://testers-headache.blogspot.com/2016/02/the-conway-heuristic.html">Tester's Headache: The Conway Heuristic</a><br />
[6] <a href="http://www.amazon.com/gp/product/0495603953/">Understanding Arguments: An Introduction to Informal Logic (Sinnot-Armstrong, Fogelin)</a><br />
[7] <a href="http://testers-headache.blogspot.com/2014/04/on-thinking-about-heuristic-discovery.html">Tester's Headache: On Thinking about Heuristic Discovery</a><br />
[8] <a href="http://testers-headache.blogspot.com/2016/02/testing-what-was-question.html">Tester's Headache: Testing. What was the question?</a><br />
[9] <a href="http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html">Framing: Some Decision and Analysis Frames in Testing</a><br />
[10] <a href="http://www.satisfice.com/blog/archives/856">James Bach: Testing And Checking Refined</a><br />
[11] <a href="http://www.developsense.com/blog/2016/04/you-are-not-checking/">Michael Bolton: You Are Not Checking</a>Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-40015393199254507412016-02-19T21:17:00.000+01:002016-02-19T21:23:54.612+01:00The Conway HeuristicIf you have not read Polya's "How to Solve It", [1], and have an interest in heuristic approaches to problem solving then I'd recommend it.<br />
<br />
There are many heuristic approaches to problem solving that you probably use without realising it. The book may just help spot and discover new problem solving heuristics in the wild, [2].<br />
<br />
It was whilst reading a version of Polya's book, with a forward by <a href="https://en.wikipedia.org/wiki/John_Horton_Conway">John Conway</a>, that I read this passage:<br />
<blockquote class="tr_bq">
<i>It is a paradoxical truth that to teach mathematics well, one must also know how to misunderstand it at least to the extent one’s students do! If a teacher’s statement can be parsed in two or more ways, it goes without saying that some students will understand it one way and others another, with results that can vary from the hilarious to the tragic.</i></blockquote>
This has a clear application (to my mind) to testing. Think of specifications, requirements or any type of communication. We are generally very good at imprecise language so the risk for miscommunication, misunderstanding, hidden assumptions remaining hidden or unsynchronised priorities is real. I'm sure we all have our favourite misunderstanding war stories.<br />
<br />
In about 2012, [3], I re-stated this paragraph specifically for communication as:-<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoVaKHfe4Hx2k-mCSDx7pzteicnaMMLrsUX_UMfssJOEpgb_5kyIjBhba2DO4cjdN9gtT9BNndVNyK2fIPByLd6LiO0RT4721YrtrDophUNYc3rl0kT_vTcOFxmKcM6V6vCiJmGqQ4NDbF/s1600/ConwayHeuristic.tiff" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="255" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoVaKHfe4Hx2k-mCSDx7pzteicnaMMLrsUX_UMfssJOEpgb_5kyIjBhba2DO4cjdN9gtT9BNndVNyK2fIPByLd6LiO0RT4721YrtrDophUNYc3rl0kT_vTcOFxmKcM6V6vCiJmGqQ4NDbF/s400/ConwayHeuristic.tiff" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I called it the Conway Heuristic. It's something I use from time to time - to remind me that there might be a loose agreement or understanding - or a shallow agreement. I tend to think of this when there are <a href="https://en.wikipedia.org/wiki/Weasel_word">weasel words</a> or phrases about. For example, saying something is "tested" - which can be interpreted as fault-free, where in fact there may be several issues remaining.<br />
<br />
This is a heuristic for a couple of reasons: (1) it's a guide, and not fool-proof; (2) it's impossible to think of all the ways something would be misinterpreted. So, this is like a useful check question, "is there or could there be a problem here?" If it helps remove the big mistakes then it's done its job.<br />
<br />
So, anywhere where there are context-free questions or context-free reporting - or really, generalised or context-free communication, then keep this heuristic in mind. There be dragons....<br />
<br />
<b>References</b><br />
[1] <a href="https://draft.blogger.com/How%20to%20Solve%20It:%20A%20New%20Aspect%20of%20Mathematical%20Method">How to Solve It: A New Aspect of Mathematical Method</a>; Polya<br />
[2] <a href="http://testers-headache.blogspot.com/2014/04/on-thinking-about-heuristic-discovery.html">On Thinking about Heuristic Discovery</a><br />
[3] <a href="http://www.slideshare.net/YorkyAbroad/testing-lessons-from-the-rolling-stones">Testing Lessons from The Rolling Stones</a>Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-27101462182900797592016-02-14T17:25:00.000+01:002016-02-14T17:25:17.334+01:00Testing. What was the question?Details, details, details. It seems sometimes people get sucked into debates and discussions on details and forget a bigger picture - or that there is a bigger picture. <div>
<br /></div>
<div>
<b>TL;DR</b> -> Jump to the <b><i>Thoughts and Observations</i></b> part<br /><div>
<br /></div>
<div>
Do you recognise any of these?</div>
<div>
<ul>
<li>Is it right to say I am automating a test?</li>
<li>Is it ok to have lots of automated tests?</li>
<li>Is it right to have a standard for software testing?</li>
<li>Do I need to distinguish between scripts that I code or encode and my testing?</li>
</ul>
</div>
<div>
Notice any pattern? The discussion usually focusses on the detail. This might be ok... But sometimes it's useful to have a tool, aid or heuristic to help judge if this detail is needed or not. So, here's my control question for you, if you end up in such a discussion. </div>
<div>
<ul>
<li><b style="font-style: italic;">What set of questions are you working with that you want your testing to help answer? </b></li>
</ul>
</div>
<div>
Meaning, what is the relation to your teams', or projects' or programs' development activity? So, really,</div>
<div>
<ul>
<li><b><i>WHY</i></b> are you testing?</li>
<li><b><i>HOW</i></b> are you testing?</li>
<li><b><i>WHAT</i></b> are you testing?</li>
</ul>
</div>
<div>
Yes, it starts with "why". Depending on how you view or frame, [1], your testing will help you understand if the detail is needed, needed now or not.</div>
<div>
<br /></div>
<div>
The <b><i>WHY</i></b></div>
<div>
This could circle around ambitions, visions or goals of a team, company, product, campaign, program etc. </div>
<div>
<ul>
<li>Who is the receiver of my test information? Are there safety critical or contractual aspects involved? (e.g. <i>Visibility of testing whether by artefact or verbal report? Have I made a stakeholder analysis?</i>) </li>
<li>How long will the product live and what type of support/development will it have? (e.g. <i>Support needs, re-use of scripts as change checkers? Supporting tools and framework to leave behind?</i>) </li>
<li>What are the over-aching needs of the information from testing? (e.g. <i>See reference [2]</i>)</li>
<li>Are there other factors to work with? (e.g. <i>product or company strategic factors</i>)</li>
</ul>
<ul>
<li>Check-question: </li>
<ul>
<li>Does answering the "why" help describe if testing is needed, in which cases it supports the product or team? Answering this question usually helps working with the "how" and "what" of testing.</li>
</ul>
</ul>
</div>
<div>
The <i><b>HOW</b></i></div>
<div>
Depending on the factors that the team, program or project comes up with to work out the WHY some testing will be needed, the HOW usually gets into the detail and factors affecting the approaches of testing.</div>
<div>
<ul>
<li>What type of test strategy is needed, or what factors should I think about? (e.g. <i>see references [4]</i>)</li>
<li>What items or types of information do my stakeholders really want? (e.g. <i>see references [3]</i>)</li>
<li>How does the development and/or deployment pipeline look? Staging areas? Trial areas?</li>
<li>Does the team have dedicated equipment, share or use CI machinery to support? Orchestration factors?</li>
<li>Will the test strategy need tool support?</li>
<li>Is the test target a GUI, a protocol layer, some back-end interface, combination or something else?</li>
<li>How do I and my team iterate through test results, assessing where we are now, and where we need to be? Do we have the right skills? (e.g. <i>see references [6] & [7]</i>)</li>
</ul>
<ul>
<li>Check-question: </li>
<ul>
<li>Does answering the "how" help describe where the testing fits into the development of the product (i.e. an end-to-end view)? If not, then you might not be done answering this question.</li>
</ul>
</ul>
</div>
<div>
The <b><i>WHAT</i></b></div>
<div>
If and when you have worked out the "why" and the "how" then the artefact-heavy part of testing might be the next set of questions to work with. </div>
<div>
<ul>
<li>Split between existing script re-use and new script development</li>
<li>What heuristic inputs do I use? Are they sufficient? Do I notice new ones or refine some? (e.g. see references [5])</li>
<li>Now you can think about test design techniques (pick your favourite book, blog or list).</li>
<li>Extra tooling to create to support the testing (e.g. test selection and filtering aids).</li>
</ul>
<ul>
<li>Check-question:</li>
<ul>
<li>The techniques and components chosen here should support the "how" and the "why". Do they? What might be missing?</li>
</ul>
</ul>
</div>
<div>
<b>Thoughts and Observations</b></div>
<div>
Notice that the "details" usually fall into the "what" or sometimes the "how" of testing? But - if you're not connecting it to <i><u><b>YOUR</b></u></i> "why" of testing then you might be getting yourself into a rabbit-hole. You might be - from a team, product or product perspective - locally optimising the effort.</div>
<div>
<ul>
<li>So, details are fine as long as you're explaining the why, the how and the what together - and for what (product, company, team). This is the context.</li>
</ul>
</div>
<div>
Another way to look at it - if you're getting caught in details or not connecting the "why", "what" and "how" of testing in the context of your product, program, company or team then you might just be trying to solve the wrong problem. For help on trying to work out the right problem I'd recommend starting with Gause & Weinberg [8].</div>
<div>
<br /></div>
<div>
References</div>
<div>
[1] <a href="http://testers-headache.blogspot.com/2011/08/framing-some-decision-and-analysis.html">Framing: Some Decision and Analysis Frames in Testing</a> </div>
<div>
[2] <a href="http://testers-headache.blogspot.com/2012/12/mapping-information-value-in-testing.html">Mapping Information Value in Testing</a></div>
<div>
[3] <a href="http://testers-headache.blogspot.com/2013/01/identifying-information-value-in-testing.html">Identifying Information Value in Testing</a></div>
<div>
[4] <a href="http://testers-headache.blogspot.com/2013/01/a-test-strategy.html">A Test Strategy</a> </div>
<div>
[5] <a href="http://testers-headache.blogspot.com/2014/04/on-thinking-about-heuristic-discovery.html">On Thinking about Heuristic Discovery</a></div>
</div>
<div>
[6] <a href="http://testers-headache.blogspot.com/2014/04/on-test-results-and-decisions-about.html">On Test Results and Decisions about Test Results</a></div>
<div>
[7] <a href="http://testers-headache.blogspot.com/2015/07/testing-tests-story.html">Testing & Tests: A Story</a></div>
<div>
[8] <a href="http://www.amazon.com/Are-Your-Lights-Figure-Problem/dp/0932633161/">Gause, Weinberg; 1990; Are Your Lights On?: How to Figure Out What the Problem Really Is</a> </div>
<div>
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com1tag:blogger.com,1999:blog-5948141587422980338.post-42640843520871614672016-01-10T12:28:00.000+01:002016-01-10T12:28:25.065+01:00Some Software Development & Testing Challenges 2016So it's 2016 and I have been reflecting on some of the challenges I see for Software Development, with emphasis on Software Testing.<br />
<br />
<b><u>Continuous Integration</u></b><br />
<br />
Everyone knows what this is right? The concept has been around a while* - everyone has been there and done that, if you read the hype. But who is innovating?<br />
<br />
A lot has been written about CI and its place in support of testing... Or has it?<br />
<br />
<b><i>Some Challenges</i></b><br />
<br />
<ol>
<li>Massive parallel script execution against the same target drives a re-think on test framework design, modelling and creation - impacting data modelling and needs for flexibility in frameworks and harnesses</li>
<ul>
<li>This is a move away from single isolated and independent "tests" on a stateful application. It will trigger a change in test script approaches. Where is the work on that? Pointers gratefully received...</li>
<li>I have seen some research on "multi-session testing of stateful services", but more is needed.</li>
</ul>
<li>CI script execution and studies showing the effectiveness of dynamic test script selection strategies for team or testing support</li>
<ul>
<li>I see that as a rule-driven approach to setting a series of checks on commits, e.g. (1) which checks cover my updated code-base (execute and result), (2) which whitespots in my codebase are there now (report)</li>
<li>Where are the studies or experience reports, where is the work?</li>
</ul>
<li>There are socio-technical challenges with CI use and implementation. Technology is the easy part, the soci-technical part comes in when organisational issues and preferences distort the technology choices. This might range from "we have always done it this way" to "the language or framework of choice here is X so everyone must adapt to that".</li>
<ul>
<li>CI is a development approach, and is distinct from testing. It's like an extension to compiler checks**. Thinking around selecting and adding to those "compiler checks" needs to be dynamic. Experience reports, empirical studies for this?</li>
<li>There is a danger that "testing" is driven into a TDD/Acceptance Test-only mode.</li>
<li>I would like to see more research on organisational and soci-technical challenges around software development...</li>
</ul>
<li>Are people really going all-in on cloud and virtualization technologies to solve their CI related bottlenecks? Mmmm...</li>
</ol>
<u><b>Software Testing</b></u><br />
<br />
<b><i>Some Challenges</i></b><br />
<br />
<ol>
<li>Detachment from Software Development</li>
<ul>
<li>This can be seen in various forms</li>
</ul>
<ol><ol>
<li>Distillation down to process on "testing" only - the ISO 29119 papers are a classic example of this. This is the "reductionist" approach to a wicked organisational problem - not very sophisticated and with a high risk of solving the wrong problem.</li>
<li>Other examples are some/many software testing only books - yes, it can be good to highlight the testing and testers role and special challenges there, but until you start from software development as a whole (systems thinking approach) then there is a high risk that you are making a local optimisation. Another reductionist approach, liable to solving the wrong problem.</li>
</ol>
<ul><ul>
<li>So, software testing focus -> good; software testing interplay and interaction and CONNECTION to the creative process -> better.</li>
</ul>
</ul>
</ol>
<li>Mis-understanding of the software testing challenge - how to link a creative process (software creation and positive and confirmatory tests and checks) to a challenging process (testing for issues, highlighting problems, testing for risks)</li>
<ul>
<li>Many organisations focus on confirmatory tests - a proxy for customer Acceptance Tests - as an MVP (minimum viable process), i.e. a proxy "get out of gaol card". See Myers [2] example of testing in an XP approach is an example here.</li>
<li>Myers [2] first wrote about the psychology of software testing. However, Martin et al [4] make the case for reframing this as an organisational approach/problem. Rooksby et al [5] observe the cooperative nature of software testing.</li>
<ul>
<li><b>More studies on satisficing the organisational needs please!!</b></li>
</ul>
</ul>
<li>Lack of soci-technical studies and research into software testing and its part in software development. Rooksby & Martin et al [4] & [5] performed ethnographic studies of software testing to highlight its cooperative and satisficing nature. This called for further research</li>
<ul>
<li>Sommerville et al [6]:</li>
<ul>
<li>"<i>An over-simplification that has hindered software engineering for many years is that it is possible to consider technical software-intensive systems that are intended to support work in an organization in isolation. The so-called ‘system requirements’ are the interface between the technical system and the wider socio-technical system yet we know that requirements are inevitably incomplete, incorrect and out of date.</i>"</li>
</ul>
</ul>
</ol>
The sooner we stop treating software development, and especially testing, in reductionist approaches, consider the socio-technical aspects - especially for large and complex systems - the better. And, today, most systems are inevitably complex.<div>
<br /></div>
<div>
Got any pointers to recent advances? I'm all ears...<br /><div>
<br /></div>
<div>
<b><u>References</u></b></div>
<div>
[1] Li, Chou, 2009, IEEE; A Combinatorial Approach to Multi-session Testing of Stateful Web Services</div>
<div>
[2] Myers, 2011, 3rd ed; The Art of Software Testing</div>
<div>
[3] Rethinking Experiments in a Socio-Technical Perspective: The Case of Software Engineering</div>
<div>
[4] Martin, Rooksby, Rouncefield, Sommerville, 2007, IEEE; ‘Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company</div>
<div>
[5] Rooksby, Rouncefield, Sommerville, , Journal of CSCW; Testing in the Wild: The Social and Organisational Dimensions of Real World Practice</div>
<div>
[6] Sommerville, Cliff, Calinescu, Keen, Kelly, Kwiatkowska, McDermid, Paige, 2011, Communications of the ACM; Large Scale Complex Systems Engineering</div>
<div>
<br /></div>
<div>
*I led the architecture work on a multi-stage CI system in ¨2010</div>
<div>
**yes, a big simplification.</div>
</div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-47317844251361419812016-01-09T10:50:00.000+01:002016-01-09T12:33:41.802+01:00Understanding Arguments... and "Trolls"In the autumn of 2013 I took an online Coursera course called "<a href="https://www.coursera.org/course/thinkagain">Think Again: How To Reason and Argue</a>".<br />
<br />
Many readers here would be familiar with a number of the concepts but the course was useful to me to help structure some concepts around statements and arguments, strategies in analysing them and eventually trying to understand the viewpoint of the person making the statements (arguments).<br />
<br />
The course was good and something I'd recommend as an exercise in helping to understand and categorise ones own approach to argument analysis and deconstruction.<br />
<br />
The course also gave online access to the book <a href="http://www.amazon.com/Understanding-Arguments-Introduction-Informal-Logic/dp/0495603953/">Understanding Arguments: An Introduction to Informal Logic</a>, a comprehensive and useful reference book.<br />
<br />
<b><u>An Important Lesson</u></b><br />
<br />
On element that was re-inforced and stood out early on in the course was to treat all arguments and statements sympathetically. This is like a safety valve when you see or hear a statement that might infuriate, irritate or wind up.<br />
<br />
It's an approach to help one get to the root meaning of a statement and understand the motives of the person (or group).<br />
<br />
I used this approach when first looking at the ISO 29119 documents and supporting material, ref [2] [3].<br />
<br />
<b><u>Challenges & Trolling</u></b><br />
<br />
I often get challenged about my reasoning and thinking. I welcome this, it's usually very good feedback not just on my thinking but also the way I communicate an idea. So, I try to treat all challenges and criticism sympathetically.<br />
<br />
But, when does it become "trolling"?<br />
<br />
I saw a <a href="http://testsheepnz.blogspot.se/2016/01/why-i-am-tired-of-having-to-justify.html">post</a> this week from Mike Talks (<a class="ProfileHeaderCard-screennameLink u-linkComplex js-nav" href="https://twitter.com/TestSheepNZ" style="color: #8899a6; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; text-decoration: none !important;"><span style="color: #8899a6; font-family: Helvetica Neue, Helvetica, Arial, sans-serif;"><span style="font-size: 14px;">@</span></span><span class="u-linkComplex-target" style="color: #8899a6; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; text-decoration: none !important;">TestSheepNZ</span></a>) - I liked the post - but I also recognised the source that triggered the post.<br />
<br />
<b><i>Troll?</i></b><br />
<br />
Well, when it comes to software development - and especially software testing (due to <a href="http://testers-headache.blogspot.com/2009/08/expert-itis-diagnosis-and-treatment.html">expert-itis</a>, amongst other things - I need to update that post) - there might be some tells to help judge:<br />
<br />
<ol>
<li>Does the person claim to be an expert/authority in the field, but without evidence or catalogue of work? (this is a form of avoidance)</li>
<li>Whether an expert or not, do they listen and engage in discussion - not just <a href="https://t.co/TizMPzFiIk">avoidance</a>? (hat tip to James <a class="ProfileHeaderCard-screennameLink u-linkComplex js-nav" href="https://twitter.com/jamesmarcusbach" style="color: #8899a6; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; text-decoration: none !important;"><span style="color: #8899a6; font-family: Helvetica Neue, Helvetica, Arial, sans-serif;">@</span><span class="u-linkComplex-target">jamesmarcusbach</span></a>)</li>
<li>Twitter - due to the 140 char limit - can make people appear to be buzzword, soundbite or even BS generators. Do they resort to soundbites or other indirect responses when questioned or challenged? (this is a form of avoidance)</li>
</ol>
<br />
Essentially, the keyword in the above is avoidance. If so, you might have a hopeless case on your hands.<br />
<br />
<i><b>Treatment?</b></i><br />
<br />
You can google how to deal with internet trolls, but my approach:<br />
<br />
<ol>
<li>Start with sympathetic treatment. If this doesn't help you understand the statements, arguments or motives (and see the list above), then</li>
<li>Detach, ignore, unfollow.</li>
<li>Give yourself a retrospective. Was there a potential feedback element - can you communicate your message differently, is there some fundamental insight that you could use? (this is a topic for a different post). I.e. "trolls" can have their limited use even when their interpersonal skills let them down...</li>
<li>Learn and move on. </li>
<li>I'm also a fan of humuorous treatments & critique. I'm reminded of The Not The Nine O'Clock news approach changing attitudes in World Darts (you can google it...) Sometimes these are forms of <a href="https://en.wikipedia.org/wiki/Reductio_ad_absurdum">reductio ad absurdum</a>.</li>
</ol>
<br />
If you have other perspectives on understanding arguments I'm all ears!<br />
<br />
<b><u>References</u></b><br />
[1] <a href="http://www.amazon.com/Understanding-Arguments-Introduction-Informal-Logic/dp/0495603953/">Understanding Arguments: An Introduction to Informal Logic</a><br />
[2] <a href="http://testers-headache.blogspot.com/2014/08/iso-29119-questions-part-1.html">ISO 29119 Questions: Part 1</a><br />
[3] <a href="http://testers-headache.blogspot.com/2014/09/iso-29119-questions-part-2.html">ISO 29119 Questions: Part 2</a><br />
[4] <a href="https://en.wikipedia.org/wiki/Internet_troll">Internet Troll</a><br />
<br />
<br />
<br />Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-34725779585640966572015-07-18T03:43:00.001+02:002015-07-18T03:46:51.924+02:00Testing & Tests: A Story<div>
<span style="font-family: Helvetica;"><span style="font-size: 12px;"><b>Are you questioning your questioning?</b></span></span></div>
<ul>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">What do you do when you have limited access to the product to be tested?</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">What if your test lab is expensive - meaning you have little time there or have little preparation time on site?</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">What if you have (in the past) had limited compute time to perform calculations?</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">How good is my test cheat sheet or template?</li>
</ul>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Once upon a time…</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
…access to computing power was limited. Mainframe access (as it was sometimes) was time limited or budgeted. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>And even today…</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
…some test lab set-ups are expensive, meaning preparation and execution time (or access) can be limited.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>This may (and has) lead to</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
…extensive preparation of what to be done. Sometimes in terms of steps, sometimes in scripts, sometimes in code.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I’ve worked in all these scenarios - during formal education, during research and during software testing.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
There are skills connected to working in this type of environment, in particular, and also investigation in general. Whether it is testing or performing a calculation - in some of my cases, a complex algorithm - there is the calculation to perform (a piece of code to produce some output). There might be various input data - which might be fed sequentially, or have another algorithm (program, or wrapper) iterate through the data and config combinations.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Then there might be complimentary tests (calculations) to check the output, calibrate it or look for patterns in output, or to compare with other data sets. There is the coding of these calculations (scripts & programs), construction of data and investigation of configuration. There is the paper work and head-scratching (scribbling and calculation) of what combinations make sense, derivation of new algorithms, exploration for meaning and then how and what to communicate and discuss.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Then there is the evaluation of the results. Do they make sense, do they answer the original question. which new questions do they raise, are the results reasonable, how do I know?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Whither Testing Skill?</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Where is the testing - and testing skill? <i><u>It’s everywhere</u></i> - but especially in the parts around working out the types of questions (or calculations, or algorithms) to make and interpreting the data, evaluating the results and deciding if complimentary work is needed.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
How to work out the types of questions? Sometimes people will answer that this falls into the category of test techniques, but before that there may be experience and discussion helping to direct which techniques to try, which type of questions are useful (or interesting), before even discussing what type of implementation (and limitations they might have).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
All of this broadly applies to scientific research as much as software testing. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Questions, questions</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
I was reminded recently by a colleague of a seminar I gave some years ago. He said that something I’d said stuck with him: </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>“A key skill of good testing is about asking good (relevant) questions and assessing the answers - either to help with more questions or to help understand how good the questions (and testing) were.“</i></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I.e. you must know how to evaluate the quality of the testing - when the results still make sense, what their “shelf life” is and when to ask new questions (when the gap in knowledge about the product grows).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Investigation activities</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Notice, the investigation activities and skills fall into some broad categories </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
(1) working out the good, useful and relevant questions,</div>
<div style="font-family: Helvetica; font-size: 12px;">
(2) working out how to get answers for them, </div>
<div style="font-family: Helvetica; font-size: 12px;">
(3) getting the data for the questions, </div>
<div style="font-family: Helvetica; font-size: 12px;">
(4) documenting - lab notes, procedures, </div>
<div style="font-family: Helvetica; font-size: 12px;">
(5) analysis of results, </div>
<div style="font-family: Helvetica; font-size: 12px;">
(6) new questions to ask, </div>
<div style="font-family: Helvetica; font-size: 12px;">
(7) communication of results and assessment </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
This type of pattern can be seen in scientific research and good software testing. Also in both, all these steps need to “work” - great execution of the wrong question (test) or misinterpretation of a good test might be useless.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Note: none of this means that all these activities have to be done and controlled by a single tester (or only by testers) - they could be distributed in a team - or really - the team has the same goal as these activities and keeps the product development AND software testing true to its purpose. In scientific research this is the equivalent of peer review.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Testing & Tests</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
So, in this story…</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Tests</i></span> are very much related to the questions that might be asked about a product. They could take different implementation forms (types of test design or test techniques) and could be encoded (or instantiated) in different ways - a script or a data/configuration set or set-up, evaluating some data points.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Testing</i></span> is the activity of working out which questions (tests) are useful or relevant in a given situation, what to make of the data and results for the tests, evaluating how to change or proceed and what type of information is useful to communicate.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The starting point above were typical scenarios using test scripts and scripting. However, the testing skill was not purely reliant on or a result of the test scripts.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Does this apply to me?</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Suppose you don’t work with expensive test labs or that you have extensive access to the software to be shipped. Does any of the above apply to you? You might think - <i>“I have a handy cheat sheet or list of things to do or nice reference book or a standard to follow.”</i></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Then ask yourself:</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
1. How do I improve or test my own cheat sheet, list of heuristics, guide, template or standard?</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. How do I know they are relevant or useful?</div>
<div style="font-family: Helvetica; font-size: 12px;">
3. What do I need to create or practice for myself to help improve the value I add to my team, group, company or customer? </div>
<div style="font-family: Helvetica; font-size: 12px;">
4. What are my (or my team’s) weak areas? And how do I get help on them?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In my experience, good testers and teams ask themselves these types of questions.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>And finally…. To script, or not to script, that is the question. </b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Wrong that’s a distraction!</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Some people think that a script being used or not is an indication of testing skill or test quality. There might be correlation but not usually causation!</div>
<div style="font-family: Helvetica; font-size: 12px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Some people think that good testing is distinguished by how good the artefacts are that are left behind. To some extent this is true, but a major part that drives which artefacts are left behind (or avoided), including their content and quality, lies in the thought and investigation that goes in upfront, the analysis, discussion and evaluation afterwards.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
If you (or your team) are focusing on one of the categories above (investigation activities) at the expense of another, then that’s a warning sign…..</div>
<div style="font-family: Helvetica; font-size: 12px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
If you (or your team) can account for and question the categories above (investigation activities) then you have a chance of doing good testing. <u>Keep questioning your questioning!</u></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-70337042923121722202014-09-14T17:42:00.000+02:002014-09-14T17:42:49.075+02:00ISO 29119 Questions: Part 2<div style="font-family: Helvetica; font-size: 12px;">
<b>Content questions</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<b></b><br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
There is an ISO software testing standard (ISO/IEC 29119) parts 1-3 published, parts 4-5 in draft, ref [1-4]. I have read parts 1-3 and a draft version of part 4 and am and have been forming several questions for some time. Some questions might be regarded as problems, issues and others as need for clarification.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I will use a series of 3 posts to document. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 1: Looks at the reasoning for the standard - based on the information publicly available, ref [8].</div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 2: Looks at some of the content.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 3: Looks at the output and usage of the standard and some implications.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
This is a snapshot of some items analysed in 29119, it is not a full review - but the comments are representative of what I have found in parts 1-4.</div>
<div style="font-family: Helvetica; font-size: 12px;">
It looks at the validity of the model, the aspect of compliance/conformance to 29119 and some of the language and descriptions used.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Process Model</b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;">Process Model Validity?</span></div>
<div style="font-family: Helvetica; font-size: 12px;">
The standard presents a set of definitions, models for processes and templates for artefacts. The implication (that I assume) is that these are needed, required or observed where “good” software testing happens. I make this assumption as anyone would hardly take the effort to standardise approaches that are ineffective, would they?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
For these process models to have any validity I’d expect external and internal validity to be demonstrated or the basis on which they are claimed to be shown. In fact, you’d expect research of model types that work, and which type of populations (organisations, products and project specifics) to be the basis for a standardisation effort.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Internal process model validity</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
This handles the question of cause and effect. By following this process model it produces something. Good software, happy customers, good testing? It isn’t really stated. A question I have with a process model - that is presented as a standard - is where is the relation between the model (construct) and the result/interpretation (“good” software, or “good” software testing).So what’s the purpose of the model - and what will happen if I follow it. The effects are not stated. There is no evidence of it being applied (tried or tested). Therefore, internal validity of the process model cannot be assumed. I would suggest this is needed for standardisation. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>External process model validity</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
This handles the question to which the process model can be applied generally - across which population of products, projects and organisations. Again there is no indication (evidence) about which range of products and organisations this can be <span style="text-decoration: underline;"><b>reliably</b></span> applied to. It is claimed that it can be applied for any organisation or software testing. This claim is not supported by evidence - reference to the input cases for which this claim is made. Therefore, the claim of application (any software testing and any organisation) is unsupported. <span style="color: #ff2600;">Rhetoric and guesswork</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Process Model - Default Form: Waterfall</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
It is striking looking at the process model flows how waterfall like they are. In 29119-2, 7.2.1, the test planning process is shown. It is also visible here, from ref [6]. There is a note saying that it may be done iteratively. I think leaving the default model being a form that Royce said “is risky and invites failure”, in 1970 ref [9], is poor work. It seems that some real experience “that can be applied anywhere” is not the basis of this flow. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The last part of “Gain consensus of test plan” (which is labelled TP8 in 29119-2) is that approval is gained from the stakeholders. The last part of “Publish Test Plan” (labelled TP9 in 29119-2) is to communicate the availability of the test plan. Wait, the test plan your stakeholders have approved in the previous step - you’re now going to tell them that it’s been approved? Do you think this might be annoying, appear wasteful or potentially just incompetent? I think there’s a risk for that - if you follow it by rote.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Test status and reporting is only described within the context of a completion process. Communication is framed as a handover (or asset even) rather than something that starts early and is continuous. This seems to be framed as a contractual type of reporting. The lack of integration with a software development process model is striking - there is no linkage to software development other than “design then test” - so this seems to be aimed at non agile shops, non incremental or iterative development shops. So who is it aimed at? Testing and test results can and should affect decisions within software development (including requirement refinement and discovery), but this seems to be absent in 29119 - possibly, I hypothesise, because it is a pure waterfall-test-separate-from-development model.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The process model is a mine field of contradictions and potential question marks (for me) - which is one reason I can think that it hasn’t been tested / used in real-life. Note, I have read that a draft of 29119 has been used in a study - this will be the subject of a future study. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Process Model Conclusion</i></span> </div>
<div style="font-family: Helvetica; font-size: 12px;">
The problem for me is that definitions are stated and process models are stated. But the validity of the process models producing good (meaning to me: efficient, effective and reasonable) software testing is not stated, linked, demonstrated or explained.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I’d like to see - and expect a linkage to be demonstrated - in a model that is at the basis of a standard - between theory (what is intended to be the output of such a model) and process model, and between study of practice (evidence to support the model that is intended to be standardised) and the process model. Where is the linkage, or evidence of this linkage - I can’t find it in 29119? Therefore, I cannot see any basis for this process model or any internal or external model validity. Conclusion: <span style="color: #ff2600;">speculation, gamble</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The standard appears to lay the groundwork for a paper-trail between test requirement/inception and final test report. However, the rigour that 29119 might want to put into this paper-trail is strikingly absent in it’s own “trail” (or line of reasoning) between “test need”, “test output” and “test effectiveness and efficiency” - i.e. it omits the evidence for it’s claim that this is a model that facilitates “effective and efficient” testing, or is even a useful model. How did that happen? It seems to be a “Don’t do as we do, do as we say” approach - and I can’t decide if it’s sloppiness or lack of rigour or some oversight on the part of the WG or ISO or both. I do wonder if the result might have been different if ISO had recruited (with interviews) people to draft a standard rather than leave it to volunteers.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;">Note</span></div>
<div style="font-family: Helvetica; font-size: 12px;">
Perhaps the standard - and any accompanying audit according to the standard - is meant to facilitate demonstration by the tester (or organisation) that they have formed a traceable approach to their work and can connect meaning for the work and results to a stakeholders needs. This is a potentially valid and useful aim. But the standard in itself will not necessarily facilitate this - an organisation could produce masses of documentation according to the standard and still have work that isn’t effective, efficient or even repeatable. I will dig more into this question in part 3 of this post series.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Conformance</b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
An organisation can claim full or tailored conformance to 29119.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In part 29119-2, section 2, the processes (meaning - I think - activities) involved are described as the basis for conformance (or not). Due to the concerns above (with the model validity) this could lead to a number of scenarios where an organisation is requested to state its conformance to 29119.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
1. Full conformance is claimed (but there is no understanding of the implications or the organisation hasn’t spotted the problems in the model).</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. Tailored conformance is claimed (but there is no understanding of the implications or the organisation hasn’t spotted the problems in the model).</div>
<div style="font-family: Helvetica; font-size: 12px;">
3. Tailored conformance is claimed (and no part of the process model is followed). The implication here is that the organisation understands (and can demonstrate) what they are doing and maybe even sees 29119 as an obstacle to efficient and effective testing.</div>
<div style="font-family: Helvetica; font-size: 12px;">
4. Non conformance is claimed and the organisation understands (and can demonstrate) what they are doing and maybe even sees 29119 as an obstacle to efficient and effective testing.</div>
<div style="font-family: Helvetica; font-size: 12px;">
5. Some combination of the above.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="color: #ff2600; font-family: Helvetica; font-size: 12px;">
<span style="color: black;">So, </span>a statement of conformance potentially doesn’t say so much<span style="color: black;">.</span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Language and Claims</b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
I don’t know if the editing, writing style or understanding of the topics is at fault but I know poorly formulated or illogical statements don’t help in understanding 29119. This is meant to be a standard, developed and reviewed over quite a time - not a soundbite or copy/paste/hope-it-makes-sense effort. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
For instance, the notes for test strategy (29119-2, 7.2.4.5) state that the estimate of “elapsed time” should be made. This is like saying, “estimate the <i>actual time it takes</i> rather than the time you <i>think it might take</i>…”</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Context, Understanding</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
On the part about understanding context (29119-2, 7.2.4.1) it’s not actually stated what is meant with “context” - so a statement such as “understanding of context and software testing requirements is obtained” is simplistic, misleading and poorly written. On “understanding … is obtained” - how should that be measured? Well the rest of the statement is “to support the test plan”. So the implication (as I can read it, as there is no other guidance on interpretation, once there is a scope in a test plan - or really test plan - then “understanding is obtained”.) As someone who talks to plenty of non-testers you cannot point to a document and claim that understanding is “obtained”.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Ah, but wait, point (b) in the clause might help. You should identify and “interact” with “relevant stakeholders”. Oh, not any old stakeholder (which would be an oxymoron, I think) but relevant stakeholders. Clumsy language -> like saying, “talk to the <span style="text-decoration: underline;"><b><i>right</i></b></span><b><i> </i></b>people<b><i>”. </i></b>So it appears as though the standard is using a convoluted way of saying, “talk to the right people”. <span style="color: #ff2600;">Simplistic and obtuse</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Section 29119-1, 5.2, does give some words on project and organisation context. However, this is framed in the model of 29119-2. This model has a number of validity issues (see above) and is, in essence, forming a circular argument. If you adopt the process model (from 29119-2) then form an understanding of the organisational and project context then the process model (in 29119-2) can be followed. But the context understanding is framed by the process model and the context understanding should be formed somewhat separately (independently) of the process model. Or looking at it starting with 29119-1 (5.2) to understand the project context you need to adopt the process model frame from 29119-2 - same circular dependency. <span style="color: #ff2600;">Circular reasoning -> faulty basis for usage, and potentially misleading</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Stakeholders</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
Reference to “stakeholders” or “the stakeholders” occurs quite often in parts 1 & 2 without actually suggesting who they might be. I think putting a lot of time into describing a model of a process without describing the actors in the model is a serious flaw in a model. It means it is so generic - to be applied (or maybe really overlaid) anywhere. It also suggests the validity of the model or actual usage was never considered. <span style="color: #ff2600;">Unclear and potential for misunderstanding</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Engineering Heuristics</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
In 29119-1, clause 5.1.3, testing is described as an heuristic approach, although it is described in a muddled fashion. It correctly states that heuristics are fallible but that knowing this allows multiple test strategies to be used to reduce the risk of using an ineffective test strategy.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Questions: </div>
<ul>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">Why would you knowingly use an ineffective test strategy (I can think of a reason, but the impression is that you shouldn’t use one, so the implication is that an ineffective test strategy could be unwittingly used…)? </li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">Or does this mean that multiple test strategies should always be used because you don’t know which are effective or ineffective? The implication is that we don’t know the consequence, limitations and benefits of any test strategy - and that I’m not sure I can agree with.</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">Of course, the standard gives no guidance on what is effective or not. So the purpose of the standard is what exactly? </li>
</ul>
<div style="font-family: Helvetica; font-size: 12px; margin-left: 36px; min-height: 14px; text-indent: -36px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; margin-left: 36px; text-indent: -36px;">
It seems to be advocating - do multiple things because you might not know what you’re doing. I take the view that if you don’t know what you’re doing then you need help, training, practice and even coaching, not a blunderbuss approach that may do more harm than good. <span style="color: #ff2600;">Clumsy and Reckless</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Exploratory Testing Definition</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
In 29119-2 (clause 4.9) describes this type of testing as spontaneous and unscripted. Then the authors couldn’t imagine a case where investigation, deliberation, preparation and even scripting might be needed to perform testing; a case where the result/s of that test might dictate the next steps. This, in itself, is not unusual as I think many people equate “exploratory testing” with something to do with little preparation rather than using results to guide future tests. I have prepared and guided this type of approach in the past. Therefore the “definition” in the standard is erroneous and unhelpful - either to me as a tester, test strategist or test communicator.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Standard, Dictionary or Constraint?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
By attempting to define (describe) all types of testing (I won’t say approaches as I don’t think that was considered) then this limits my potential testing. If I discover, develop or talk about a new (or variant) of a technique or way of testing then I am (per default) non-compliant to the standard. So the standard is not a guide to good, appropriate or effective ways to test. <span style="color: #ff2600;">Erroneous</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In the draft of part 4, 29119-4, the conformance part states that techniques used that are not described in 29119-4 should be described. This seems to be a way of not capturing everything in the standard - and is a way of avoiding debatable definitions and muddled language.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
It seems to me that the standard is really attempting to be an encyclopaedia of testing - but when I see muddled language, claims without support, circular reasoning and what appears to be untested models it makes faith (and trust) in definitions a little difficult. An encyclopaedia is not the same as a standard - so I think the intention behind the standard (and the result) is muddled.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Informative?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
29119-1 is informative. This means (according to 29119-1, 2) that conformance (agreement or compliance) is not needed. This means it is optional or that it is open to interpretation. One consequence is that having this as an input to part 2 - a generic model - means that the model is open to interpretation. Another consequence is that it’s like a poor dictionary - at best a guide to current usage, at worst misleading. <span style="color: #ff2600;">Superficial</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Process?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
I think there’s probably some confusion about what “process” means. It’s what happens (according to Mirriam-Websters dictionary and the OED). How does that fit into a process model? A model of things that happen? Ok, and this is a generic model. To produce what? That’s not defined. So it’s trying to describe a generic set of actions that happens in software testing without any description of an output. Why would you do that? I can hypothesise that (i) you might perform good testing already (which your organisation might regard as efficient and effective), then 29119 is of no use and may even be detrimental; (ii) you’re a start-up, new org or org without any knowledge of how to approach testing - then 29119 might be used to produce some documentation and activities - but as there is no construct validity in 29119 that may be a totally inappropriate approach also. So what use are generic models with generic activities? <span style="color: #ff2600;">Misleading and lacking evidence of validity</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Repeatable?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
There is a claim that businesses have an interest in developing and using repeatable, effective and efficient processes (29119-1. 5.2.) It seems natural to want your activities (processes) effective and efficient. But repeatable - in which scenarios is repeatable desirable for the way testing is carried out? Does this mean repeatable test cases in scenarios where test scripts are re-executed as some safety net (e.g. in a continuous integration loop)? Fine. But test analysis, planning, design and reporting. Should this be done in a repeatable way? The case isn’t made. <span style="color: #ff2600;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Scripted?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
In 29119-1, 5.6.6, it describes advantages and disadvantages of scripted and unscripted testing. It is claimed that scripting (in this case, a tester following a script) makes the testing more repeatable. Here, “repeatable” is debatable to me - I’ve seen people following a script that don’t do what the script says - there may be some extra step inserted or some extra observation made or some step that is hopped-over, but the script isn’t strictly followed. So, I think these advantages and disadvantages haven’t been challenged or compared with real experiences. <span style="color: #ff2600;">Erroneous</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>- Test policies and strategies</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
In 29119-1, 5.2, it claims that where formal organisational test policies and strategies are absent then the testing there is “typically” less effective and efficient. This is an interesting claim - and could read by some that this must be in place. There are 2-3 problems with this claim:</div>
<div style="font-family: Helvetica; font-size: 12px;">
1. The connection (or correlation) demonstrated in cases where such “documents” are in place and the performance of the organisation/company being effective and efficient is not demonstrated. (I.e. there is no study of “effective and efficient” testing or “effective and efficient test organisations” and what might affect their output).</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. There is no study or evidence to show that the presence of such a formal item (it probably is present even where not formally present - part of the test culture) directs the testing in an organisation - cases where it is present and has no effect, or where it is absent and the testing is deemed “efficient and effective” anyway. </div>
<div style="font-family: Helvetica; font-size: 12px;">
3. Where such a formal item is in place - it is not clear if that comes afterwards - i.e. is not the cause of “effective and efficient” testing, but a byproduct (related to #1)</div>
<div style="color: #ff2600; font-family: Helvetica; font-size: 12px;">
No evidence, rhetoric<span style="color: black;">.</span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Examination of Claims and Language </i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
Some might think I’m being “picky” when I read 29119 - or being nit-picky. Actually, this is a tester reading unsubstantiated claims - or rather noticing claims that are unsubstantiated. That’s not being picky, it’s calling out shoddy work, highlighting issues and raising a warning flag. It’s basic understanding of cause and effect - and being able to distinguish between them. Why is this important? Because if one reads 29119 and is not able to understand the claim, what it implies and doesn’t say, then there’s a strong risk that it is followed by rote. And as the output of the model is not described, following it without understanding its pitfalls is potentially harmful.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;">A generic model?</span></div>
<div style="font-family: Helvetica; font-size: 12px;">
It is not stated but I suspect 29119 is trying to be an encyclopaedia of testing, presenting a menu from which to select which activities to do and perhaps a way to structure it. However, the output is not defined - the aim of the testing - and an assessment of which may help or hinder those results. The process models have no demonstrated validity - meaning it is open to interpretation what they will produce and, more seriously, what readers of 29119 think they might produce, how they might be applied and how to observe if the models are relevant or have any contribution to an organisation’s testing effectiveness or efficiency. Therefore, the generic nature of process models and the association with an idea of conformance is really dangerous. One might conclude that 29119 (in it’s current form) is a clear and present danger to an organisation’s testing if a question of conformance is raised.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Summary</b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
In this post I’ve mainly focussed on parts related to 29119-1 & 29119-2. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="color: #ff2600; font-family: Helvetica; font-size: 12px;">
<span style="color: black;">The </span>lack of output description and consideration<span style="color: black;"> (of the process model) is serious.</span></div>
<div style="font-family: Helvetica; font-size: 12px;">
The <span style="color: #ff2600;">lack of process model validity</span> - any evidence of validity or applicability (or not) of generic models of activities - is serious. It verges on being dangerous and misleading.</div>
<div style="color: #ff2600; font-family: Helvetica; font-size: 12px;">
<span style="color: black;">The </span>muddled language <span style="color: black;">is serious.</span></div>
<div style="font-family: Helvetica; font-size: 12px;">
The <span style="color: #ff2600;">lack of apparent rigour</span> in something it is trying to describe - whether model, definitions process model applicability - is serious.</div>
<div style="font-family: Helvetica; font-size: 12px;">
The notion of conformance which my reading of 29119 means is not possible - part due to the way part 1 is defined as informative, part due to the lack of rigour in the models, part due to the apparent waterfall approach to modelling, part due to the muddled language. This means that 29119 can not be used or applied in a repeatable, efficient or effective way - means that it would be misleading to claim conformance to 29119. <span style="color: #ff2600;">Claiming conformance is a potential warning sign</span>…</div>
<div style="font-family: Helvetica; font-size: 12px;">
The amount of <span style="color: #ff2600;">rhetoric and unsupported claims</span> are potentially confusing to the reader.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I get the impression that the result (29119) is due to a number of factors</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
1. It’s a result of the underlying reasoning and motivation for 29119 not being clear, ref [8]</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. It’s a factor of the working group’s interpretation of said motivations and reasoning - and maybe not being able to clarify or communicate that</div>
<div style="font-family: Helvetica; font-size: 12px;">
3. It’s a result of some underlying assumptions (or beliefs) that the working group haven’t declared</div>
<div style="font-family: Helvetica; font-size: 12px;">
4. It’s possible that the underlying beliefs were not visible in the working group or had different interpretations (because they were not visible)</div>
<div style="font-family: Helvetica; font-size: 12px;">
5. It’s a result of ignorance in processes, how to observe processes and make hypotheses based on observation.</div>
<div style="font-family: Helvetica; font-size: 12px;">
6. It’s a result of ignorance about model validity, experimental design and limitations of these.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>The result? An Example</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
When I read 29119 I get the impression that it’s like Royce’s paper, ref [9], was read to page 2 - and crucially not page 3 or the conclusion - because 29119 seems to model what later became known as waterfall, and ignored any iterative corrections.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Royce warned of the dangers with the model - the type of model displayed in 29119 - over forty years ago. Why is there a belief that the type of model in 29119 “works”? My guess would be that it’s a result of poor observation amongst the other reasons above.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
For example</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
1. Project X (in company Y, under circumstances Z) “worked” and we followed Royce’s-page2-model (waterfall), ref [9]</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. Project A (in company B, under circumstances C) “worked” and we produced lots of documentation</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Someone could draw a conclusion that it’s the documentation that produced the good result in A, and others that it was a waterfall model that produced a good result in X. Someone else could conclude that combining “documentation” and “waterfall” would produce a good result in a new project. Of course, this “hypothesis” is dangerous and reckless. It’s not even certain (or there may be a lot of doubt) that the reasoning for X & A “working” was correct, and it’s very probable that we can’t, couldn’t or didn’t observe the most important factors that contributed to their results (meaning the understanding of Y & B, or Z & C was not what we think it is). Projects X & A might’ve worked in spite of “waterfall” & “documentation”.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
This is the reason why generalisation, and not taking close study of underlying factors of “processes”, and not establishing validity of a model is dangerous. Not being able to connect an observation to a symptom and cause is dangerous. Therefore, I think the reasoning (or absence of reasoning), support or conclusions in 29119 is dangerous. There is no evidence that this model “works”, or will produce anything that a customer can use.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I suspect the intention of 29119 was based on experiences that have worked for the contributors, but the case for internal and external model validity is not made. <u>So, advocating those models outside of the people that have used them is not supported.</u></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>Locked-in Planning and Feedback?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
Fred Brooks, chapter 11 in ref [7], wrote in 1974 about the need to plan to throw one away - that’s about understanding the limitation of planning and process models, and not locking into a waterfall model. Cosgrove, ref Brooks chap11, [7], in 1971 claimed that programmers (and organisations) deliver satisfaction of a user need, and that this perception of need changes as software is developed and tested. There is no clear consideration of user need, satisfaction or change in 29119 - i.e. there is little connection to software development in 29119, especially agile, incremental or interative ways of working.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>Contract?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
I get the impression it’s a contractual model - if you structure your testing along these lines and your stakeholders sign-off on this then you’re “protected” from blame. The model is not directed at producing something that a customer will be happy with, but rather something that a customer potentially signs-off on something before any prototype is produced.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
It seems to me that there is no result-orientation - it’s about standardising locked-in paper trails early rather than working software. There is no guidance on how to square that circle with agile approaches. In fact, if you work in an agile shop and someone starts talking about the process model from 29119 and how to adopt it, I’d be worried - that’s a “bad smell”.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>Practice, and Consolidation in a Standard?</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
There is no evidence of study or observation of test processes in practice or support of claims to say “X” is the factor that contributes to makes this process effective and efficient. This would require a social science type of study of people working (process), of organisational structures, project set-up and outcomes. And it seems to be missing even on the smallest of scales - which would be a case study to support a claim of a specific process.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I get the impression that the working group didn’t make (or search for) any study of factors that contribute to “effective and efficient” testing or testing results. To use the terminology of 29119-1, this is “error guessing”. However, as there is no assessment of “errors” (problems and pitfalls to avoid) I think of it as just plain guessing. <span style="color: #ff2600;">Rhetoric, guesswork and superstition.</span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I can’t work out why no one thought of this in the 6 years 29119 was under development - because if they had then I <i>might not</i> be having these questions now.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>And finally…</i></div>
<div style="font-family: Helvetica; font-size: 12px;">
I can borrow from “Yes, Minister”, ref [5], when I think about the connection between a process model and reality as conveyed in 29119 and in the style of clarity I get from 29119-> </div>
<blockquote class="tr_bq">
“the precise correlation between the information … communicated and the facts, insofar as they can be determined and demonstrated, is such as to cause epistemological problems, of sufficient magnitude as to lay upon the logical and semantic resources of the English language a heavier burden than they can reasonably be expected to bear.” </blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
I.e no evidence of the validity in the claims made. So yes, the overwhelming impressions I take from 29119 are of <span style="color: #ff2600;">unsubstantiated claims and rhetoric</span>. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>References</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
[1] 29119-1: ISO/IEC/IEEE 29119-1:2013 Software and systems engineering -- Software testing -- Part 1: Concepts and definitions</div>
<div style="font-family: Helvetica; font-size: 12px;">
[2] 29119-2: ISO/IEC/IEEE 29119-2:2013 Software and systems engineering -- Software testing -- Part 2: Test processes</div>
<div style="font-family: Helvetica; font-size: 12px;">
[3] 29119-3: ISO/IEC/IEEE 29119-3:2013 Software and systems engineering -- Software testing -- Part 3: Test documentation</div>
<div style="font-family: Helvetica; font-size: 12px;">
[4] 29119-4: ISO/IEC/IEEE DIS 29119-4.2 Software and systems engineering -- Software testing -- Part 4: Test techniques</div>
<div style="font-family: Helvetica; font-size: 12px;">
[5] <a href="http://en.wikiquote.org/wiki/Yes,_Minister#Episode_Eight:_The_Tangled_Web">http://en.wikiquote.org/wiki/Yes,_Minister#Episode_Eight:_The_Tangled_Web</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[6] Test Planning Process <a href="http://www.softwaretestingstandard.org/part2.php">http://www.softwaretestingstandard.org/part2.php</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[7] The Mythical Man-Month: Essays on Software Engineering [F.P.Brooks, 1975]</div>
<div style="font-family: Helvetica; font-size: 12px;">
[8] The Tester’s Headache: ISO 29119 Questions: Part 1 <a href="http://testers-headache.blogspot.com/2014/08/iso-29119-questions-part-1.html">http://testers-headache.blogspot.com/2014/08/iso-29119-questions-part-1.html</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[9] Managing the Development of Large Software Systems [Winston Royce, 1970] <a href="http://leadinganswers.typepad.com/leading_answers/files/original_waterfall_paper_winston_royce.pdf">http://leadinganswers.typepad.com/leading_answers/files/original_waterfall_paper_winston_royce.pdf</a></div>
<div>
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com1tag:blogger.com,1999:blog-5948141587422980338.post-74208235765932147502014-08-31T20:17:00.000+02:002014-08-31T20:17:22.759+02:00ISO 29119 Questions: Part 1<div style="font-family: Helvetica; font-size: 12px;">
<u>Reason & Motivations?</u></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
There is an ISO software testing standard (ISO/IEC/IEEE 29119). Currently parts 1-3 are published, and parts 4-5 are in draft. I have read parts 1-3 and the draft of part 4 and am and have been forming several questions. Some questions might be regarded as problems, some as issues and others as a need for clarification.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I will use a series of 3 posts to document. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 1: Looks at the reasoning and motivation for the standard - based on the information publicly available.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 2: Looks at some of the content.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Part 3: Looks at the output and usage of the standard and some implications.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Note, in this post I’ll only refer to publicly available versions and parts of the standard - the part I am considering here is the introduction - that is viewable in the preview part of the IEC webstore, see references.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Why?</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Whenever someone comes to me with a new idea - I usually hear about all the reasons why it’s a great idea. What isn’t usually obvious is to say why the change is needed - or what problem is being solved.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>29119</b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
And so I had the same question in my head with 29119. I wanted to know the reasons and motivations behind it - especially ones that might support its usage or adoption.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
According to 6.1.4 of the ISO Guide for Standards, ref [3], the introduction should state:</div>
<blockquote class="tr_bq">
<span style="color: #444444;"><i>The introduction is a conditional preliminary element used, if required, to give specific information or commentary about the technical content of the document, </i><span style="text-decoration: underline;"><b><i>and about the reasons prompting its preparation.</i></b></span></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
So, there should be reasons in the introduction of 29119. Looking at the introduction, there are three potential reasons given.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Reason 1?</i></b></span></div>
<blockquote class="tr_bq">
<i><span style="color: #444444;">The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally-agreed to set of standards for software testing that can be used by any organization when performing any form of software testing</span> [see refs 1 & 3]</i></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
Looking more closely at that:</div>
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“internationally”</i>: ISO implies this already - so this part is redundant</li>
<li><i>“agreed”</i>: the drafting of a standard requires consensus/agreement of 75% of the drafting members, so this is also redundant when the standard is published.</li>
</ul>
<br />
<blockquote class="tr_bq">
<i>Note: One could ask questions about the representativeness of those drafting 29119, but that’s not a question I’m looking at here.</i></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
So <i>“internationally-agreed”</i> is redundant.</div>
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“set of standards for software testing”</i> -> seems to be a duplication of “software testing”, redundant.</li>
</ul>
<br />
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“can be used”</i>: possible - this probably relates to how the standard can be tailored (more about this in part 2 of this post). </li>
<li><i>“by any organisation”</i>: quite a bold claim, but linked with the previous point on tailoring this could be read as any company can tailor the standard.</li>
</ul>
<div style="font-family: Helvetica; font-size: 12px;">
So - “can be used by any organisation” -> could be generously interpreted as “can be tailored”. But then this is still problematical. </div>
<div style="font-family: Helvetica; font-size: 12px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Standards application and conformance are usually full compliance, partial (or tailored) compliance or non-compliance. Therefore, the fact that you can declare conformance or not goes with the territory of a standard. Therefore, this is redundant information - in terms of reasoning for a standard. </div>
<div style="font-family: Helvetica; font-size: 12px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
An alternative reading could be that the standard is “useable” by any organisation. Again, this is redundant information - either the standard is adopted and conformance declared against it, or not. And why would effort be placed on a non-useable standard? Therefore, this is redundant information.</div>
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“when performing any type of software testing”</i>: again another bold claim. But part 1 of the standard “defines” the types of testing it is talking about.. The “performing” part is also implied as why else would you use the standard if not for software testing? So these parts are also redundant - by implication if you’re following the standard you’re following the definitions of the standard. (more about the definitions in part 2 of this post).</li>
</ul>
<div style="font-family: Helvetica; font-size: 12px;">
So, the paragraph becomes:</div>
<blockquote class="tr_bq">
<span style="color: #666666;"><i>The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an </i><span style="text-decoration: line-through;"><i>internationally-agreed</i></span><i> to set of standards for </i><span style="text-decoration: line-through;"><i>software testing</i></span><i> </i><span style="text-decoration: line-through;"><i>that can be used by any organization when performing any form of software testing</i></span></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
Correcting the grammar for the parts that are stripped out would look like:</div>
<blockquote class="tr_bq">
<i><span style="color: #666666;">The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define [a] … set of standards</span></i></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Reason 1: Verdict?</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
So, when you strip out the redundant parts of the statement the purpose of the standard is to create a standard. That, in grammar, is called a <span style="color: red;">tautology</span> -> redundant information. That’s not much of a purpose - it still does not tell me why the it was produced.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Reason 2?</i></b></span></div>
<blockquote class="tr_bq">
<span style="color: #444444; font-family: Helvetica; font-size: 12px;"><i>“</i></span><i><span style="color: #666666;">This series of international standards can support testing in many different contexts.</span></i><span style="font-family: Helvetica; font-size: 12px;"><i><span style="color: #444444;">“ </span>[see ref 1]</i></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“can”</i> - I think this claim is a moot point - I suspect the reason is that, “if you claim conformance then it’s supporting testing” - so it’s not actually a reason <b><u>/for/</u></b> the standard. It’s like saying, “if you follow the standard then you are standard-compliant”.</li>
</ul>
<br />
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“support testing”</i> - well as part 1 is attempting to define testing and part 2 is attempting to define a process then in a way it could be claimed to support testing. On the other hand, it could be read that if you perform testing in accordance with parts 1-3 then the standard is supporting your testing - actually this is wrong, because it has defined what is “your testing” and potentially “what is excluded” - in this sense it’s not supporting at all. </li>
</ul>
<br />
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Reason 2: Verdict</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
My reading of this sentence is, “if you follow the standard then the standard will support your testing”. Unfortunately, this is a <span style="color: red;">circular argument</span>, ref [4], i.e. not really a supporting reason for the standard at all.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Reason 3?</i></b></span></div>
<blockquote class="tr_bq">
<span style="color: #444444; font-family: Helvetica; font-size: 12px;"><i>“</i></span><span style="color: #666666;"><i>Together, this series of international standards aims to provide stakeholders with the ability to manage and perform software testing in any organization.</i><span style="font-family: Helvetica; font-size: 12px;"><i>”</i></span></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“stakeholders”</i> - this is ambiguous who is meant. I’ll be generous and assume the user of the standard is meant here.</li>
</ul>
<br />
<div style="font-family: Helvetica; font-size: 12px;">
</div>
<ul>
<li><i>“ability to manage … software testing in any organisation”</i></li>
<li><i>“ability to … perform software testing in any organisation”</i></li>
</ul>
<div style="font-family: Helvetica; font-size: 12px;">
Interesting. This could be interpreted as anyone - absolutely anyone - by following this standard, can perform (and/or manage) software testing in any organisation. In fact, I’m not sure how to interpret it in a different way - there is no guidance (clarification) in the text.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Ok - I have seen (quite a few) people in my time that (i) can’t manage software testing and (ii) are pretty mediocre at software testing. In my judgement I have a hard time understanding how this standard would give the ability for “anyone” to become “testing managers” or “good testers” - except by creating a lowest common denominator. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I can almost hear the explanation, <span style="text-decoration: underline;"><i>“Find the least able person and we’ll calibrate to that person.”</i></span> Really?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I have met and worked with - and I’m sure others will have similar experience - people (managers and testers) in various companies that are “going through the motions” - they tend to work /within/ a box (set of definitions or practices) and are not the ones who think or challenge assumptions (i.e. the ones who <i>“think on their feet”</i>). These are the people that spend more effort defining boundaries than communicating - they are plodders. If the aim of the standard is to produce a set of test managers or testers that either (a) think inside the box, (b) don’t think about or question what they’re doing, (c) plod along; then it is not something that can be associated with good (or efficient) work.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>A Possible Reason 3 Verdict</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The statement is telling - and probably says more than it really should.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Come on! If the aim of 29119 is to <span style="color: red;">set the bar so low that anyone can <span style="text-decoration: underline;"><b><i>look</i></b></span> as though they are performing good work</span> - BECAUSE they appear to be following 29119 - then that Mr Regulator, Mr Customer, Mr Potential-Stakeholder, I repeat, that is a <u><i>“bad smell”</i></u> - it’s a sign the company claiming conformance hasn’t got their eye on the ball! In a sense - you’d want to be extremely careful of anyone claiming conformance…</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Introduction verdict?</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
According to 6.1.4, ref [2], I didn’t see any reason for the standard.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The skills I used to analyse the text were reasoning, logical and critical thinking - basic skills in any good testers toolkit - it comes into play (in my experience) when discussing requirements, understanding needs from stakeholders and reacting to and understanding conflicting needs in a project situation. Or even trying to understand standards. Actually, it’s difficult to do well and takes practice and concentration.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I had the impression that a number of testing experts were involved in the drafting and review of the standard. I think the resultant communication in the introduction has room for improvement - and doesn’t bode well for the rest of the documents. I’ll leave it to the reader to judge if the people involved in drafting the introduction did a good job of explaining the reasons and motivation for 29119.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Other Sources</b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Looking elsewhere for reasoning for the standard. Ok, let’s try some other sources.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Web #1</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
If I look at the “softwartestingstandards” web page, ref [5], I see this:</div>
<blockquote class="tr_bq">
<span style="color: #666666;"><span style="font-family: Helvetica;"><i>“</i></span><i>By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.</i><span style="font-family: Helvetica;"><i>”</i></span></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
Ok, the part about “<span style="color: #515151; font-family: 'Trebuchet MS';"><i>adopting the only internationally-recognised and agreed standards for software testing</i></span>” is covered above - it’s the same logic - and reduces to a redundant phrase (says nothing in terms of motivation).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="color: #515151; font-family: 'Trebuchet MS'; font-size: 12px;">
<span style="color: black; font-family: Helvetica;">The next part is interesting though, “</span><i>which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world</i><span style="color: black; font-family: Helvetica;">”.</span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I’m interested that “high-quality approach to testing” is used - this is not stated in 29119, either as a motivation for using the standard or as a result of using the standard. As far as I can see there is no evidence (whether as case-study, or something else) to support this claim. Therefore, it is <span style="color: red;">rhetoric</span> - an attempt to influence without any evidence (proof).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Web #2</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
A Presentation given at the BCS, page 8 ref [6], states the motivation for 29119 as:</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
• Demand for existing 'standards’</div>
<div style="font-family: Helvetica; font-size: 12px;">
• Conflicts in current definitions and processes</div>
<div style="font-family: Helvetica; font-size: 12px;">
• Gaps in the current standards provision</div>
<div style="font-family: Helvetica; font-size: 12px;">
• A Baseline for the Testing Discipline</div>
<div style="font-family: Helvetica; font-size: 12px;">
• Current industry practice is lacking</div>
<div style="font-family: Helvetica; font-size: 12px;">
• Buyers unclear on what is 'good test practice'</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>“Demand for existing 'standards’” - </i>it’s not stated (or referenced) in the presentation where or what this demand is, therefore it’s an unsupported claim. <span style="color: red;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
“<i>Conflicts in current definitions and processes</i>” - 29119 is replacing some standard, so there is scope to believe that this replacement is reducing conflict between definitions and processes. However, the case for why this needs to happen is not made - it’s not demonstrated (in this presentation or elsewhere) what this conflict looks like or what problems it causes. Therefore, this seems to be a pretty weak argument. In terms of support in the presentation this claim is unsupported. Therefore in the scope of the presentation it is <span style="color: red;">rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
“<i>Gaps in the current standards provision</i>” - it’s not stated (or referenced) in the presentation where or what these gaps are, or if or why they are relevant to being a motivation for a new standard. <span style="color: red;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<i>“A Baseline for the Testing Discipline”</i> <i>- </i>it’s not stated (or referenced) in the presentation where or what this need is, therefore it’s an unsupported claim. <span style="color: red;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
“<i>Current industry practice is lacking</i>” - I could probably agree to this (but maybe for completely different reasons!) Without more information it’s not clear what this means - and isn’t stated (or referenced) in the presentation. Therefore, the claim is unclear and appears to lack any supporting evidence. <span style="color: red;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
“<i>Buyers unclear on what is 'good test practice'</i>” - I like the idea of presenting and distinguishing good test practices. However, the argument about buyers is not supported in the presentation (or referenced) with evidence. Therefore it’s an unsupported claim. <span style="color: red;">Rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b><i>Original Proposal for the Standard & ISO Directive #1</i></b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
According to Annex C of ISO Directive 1, ref [7], a proposal for a new standard just have a justification. Specifically:</div>
<blockquote class="tr_bq">
<span style="color: #666666;"><span style="font-family: Helvetica; font-size: 12px;">“</span><b><i>C.3.2</i></b><i> The documentation justifying new work in ISO and IEC shall make a substantial case for the market relevance of the proposal.</i><b><i>C.3.3</i></b><i> The documentation justifying new work in ISO and IEC shall provide solid information as a foundation for informed ISO or IEC national body voting.</i><span style="font-family: Helvetica; font-size: 12px;">”</span></span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
Now to documentation proposing the work for the new standard, ref [8], produced in 2007:</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The market requirement section states:</div>
<blockquote class="tr_bq">
<span style="color: #666666;">“<i>The national standards on which the new international standard is to be based are widely used both as the basis for commercial contracts and international qualification schemes. However, their national origin often results in them being ignored by potential users in some countries. At present there are a number of gaps (and overlaps) in the coverage they provide. A coherent set of international standards for the complete life cycle is required.</i>”</span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
The first two sentences state that some national standards are used, but not widely. The third sentence states there are gaps/overlaps between existing standards. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The forth sentence is interesting: “<i>A coherent set of international standards for the complete life cycle is required.</i>” But where is the case made according to C.3.2 & C.3.3? According to point C.3.3 there is <span style="color: red;">no evidence to support this point</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Ok, no help there.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In the “purpose and justification” section it does state:</div>
<blockquote class="tr_bq">
<span style="color: #666666;">“<i>The purpose is to produce an integrated set of international standards to cover the software testing</i><i>process throughout the development and maintenance of a software product or system.</i>”</span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
and</div>
<blockquote class="tr_bq">
<span style="color: #666666;">“<i>In overall terms, the purpose of the project is to unify and integrate the currently fragmented corpus of normative literature regarding testing that is currently offered by three distinct standards-makers: BSI, IEEE, and ISO/IEC JTC 1/SC 7. The result of the project will be a consistent, unified treatment adopted by all three organizations.</i>”</span></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
These two purposes seem complimentary, but there are problems. </div>
<div style="font-family: Helvetica; font-size: 12px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In the first says that “the software testing process” will be covered - but this seems undefined. The part 1 of 29119 outlines concepts and definitions, doesn’t it? Well there’s a problem there - that I’ll cover in part 2 of this post. But to summarise here - no parts 1-3 do not define “the software testing process” - part 1 is informative, meaning it has examples and not definitions - there is nothing to stop someone else creating their own definition. Part 2 has problems with conformance meaning that it can’t (and doesn’t) define THE software testing process. More details in part 2 of this post.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
So this purpose might be real, but it’s also unrealistic. <span style="color: red;">Unjustified</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The second purpose - to integrate information from different sources (BSI, IEEE & ISO) might seem reasonable - but this is more of a paper exercise than anything to do with standardising. There is no justification why this should be done or what problem is caused to the users of those individual sources. So, this is an <span style="color: red;">unjustified purpose</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>Summary</b></span></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I wanted to understand the motivations and justification behind 29119. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I looked in the introduction of 29119-1, where it is supposed to be stated according to ISO’s directives. I didn’t find anything that didn’t reduce to <span style="color: red;">tautology</span> and <span style="color: red;">rhetoric</span>. Now, I don’t know personally any of the members of the workgroup that produced 29119. But they are supposed to be experts in their field, and yet they produced an introduction to describe the motivation of 29119 that does not stand up to scrutiny. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Did they all have an off-day - that lasted 6 years? Groupthink? Or what was it?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I looked elsewhere - presentations given to the BCS and the softwaretestingstandards site. There were a lot of <span style="color: red;">claims without supporting evidence or references</span>. This is also known as <span style="color: red;">rhetoric</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Then I searched in the original proposal for the workgroup that would produce the standard. There, I found claims <span style="color: red;">without evidence</span>, some were <span style="color: red;">unrealistic</span> and some were almost a <span style="color: red;">wish list</span>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
After reading this I wondered about the people that reviewed the proposal. I’m sure there’s a bunch of intelligent people that looked at this and yet they seemed to have missed a lot. Was it some form of groupthink, I don’t know.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
Well, I've been looking for reasoning and motivations for 29119 that will stand up to scrutiny - so far without success. In the next part I'll look at some of the content of 29119 in more detail.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><b>References</b></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
[1] IEC/ISO/IEEE 29119-1 preview <a href="http://webstore.iec.ch/preview/info_isoiecieee29119-1{ed1.0}en.pdf">http://webstore.iec.ch/preview/info_isoiecieee29119-1{ed1.0}en.pdf</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[2] ISO/IEC Directives Part 2 <a href="http://isotc.iso.org/livelink/livelink?func=ll&objId=4230456&objAction=browse&sort=subtype">ISO/IEC </a></div>
<div style="font-family: Helvetica; font-size: 12px;">
<a href="http://isotc.iso.org/livelink/livelink?func=ll&objId=4230456&objAction=browse&sort=subtype">Directives Part 2</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[3] The Art of Software Testing Standards <a href="http://www.softwaretestpro.com/Item/5895/The-Art-of-Software-Testing-Standards">http://www.softwaretestpro.com/Item/5895/The-Art-of-Software-Testing-Standards</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[4] Wikipedia: Circular Reasoning <a href="http://en.wikipedia.org/wiki/Circular_reasoning">http://en.wikipedia.org/wiki/Circular_reasoning</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[5] Webpage: ISO/IEC/IEEE 29119 Software Testing Standard <a href="http://www.softwaretestingstandard.org/index.php">http://www.softwaretestingstandard.org/index.php</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[6] BCS - British Computer Society, page 8: <a href="http://www.bcs.org/upload/pdf/sreid-120913.pdf">http://www.bcs.org/upload/pdf/sreid-120913.pdf</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[7] ISO/IEC Directives, Part 1 <a href="http://www.iso.org/sites/directives/directives.html">http://www.iso.org/sites/directives/directives.html</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[8] ISO/IEC JTC1/SC7 3701</div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com1tag:blogger.com,1999:blog-5948141587422980338.post-55904193305692947492014-05-29T12:24:00.000+02:002014-05-29T12:24:27.377+02:00Standards, Straw Man Arguments and SuperstitionI listened to a webinar in November 2013, ref [1], about the testing standard, "ISO 29119".<br />
<br />
At the time I made some notes on the recording, but there was one item that particularly struck me as slightly intimidatory. However I happily assigned it to my mental trash bin and just remembered the meta-labels, "Straw man, scare tactic".<br />
<br />
<div>
I had forgotten about the recording... until today when I saw this on twitter from <a href="https://twitter.com/james_christie">James Christie</a>:</div>
<div>
<br /></div>
<div>
<table style="background-color: white;"><tbody>
<tr><td height="48" width="48"><img height="48" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAYBUlEQVRoBZWaeWwc93XHv7s7ey+Xy1OkeImkbsnW7UM+4jhJnaRIWtdxHCAFggRFijYx0BQt2gZB4z/SJijSNkBbtH8EBRqgRerWSescVmzJTiRbtiTroG6LFEXxEM8ludz77ufNiohhuAEy1HCHszPze+f3fd8becR29MQPttakb3mkD3s8niY79+tu9TpPkEfcr2q1plqlLnk9srOqc979rHOKo5pH3lJdo2fe1AP37tae3Xvd72ulvMS9vrJXlVxey5kljS8saDlfUTEc1WK5psGhDenJZProiZ9P/vn3/umrNzyu8HWdYZW41+t1BWishQC/xraugN1Sq6FAtXFzzVXKjhG8jgI81ocChcWUNkYDunfbFnm9gbsKllQpluUpclGppJXMssbn5nRjcl61RJtev3hNDx7eo3pTXMdeG1/rjrcecjwe33fqtXLcLFdnAdvs0/Z1hdbPu1/+il/r1zUeYzavI1iN37aZAvijVldTuEk7trSqIxqUzytVZNp65Pj48HG1tyqvj2srJVULBc1OTmv8yqhSVY9Ov3lGBz/0QdUr5XgsVP+Ot1qrPr6+sC1jx7abQu/dzLLr177f9+tGaNzfuNtj0rPXOPB68DA/HS0taolHOF0R0aKy31El4FOF47qHEEIRrylVqyifziqzmtbUxBTeqSi5mFQ+k5ePK3yqP+7F9cF3L2zL2t//n4Dvd74h6rt/37W5+xwzRmOvI1QAM0eDAVWLeVXxiDcUUjAelycQQGqfkF918qBaRfhcTuVCGS+hINfn0hl58EJyIUkoei3Pgs66td8t2LuP18V6v3Pr373f5/r1HqzuRg8fJId8/O3l06LeQzjVS1VVCZkqieqY4EWOC0XV8+x8HwhFlOdv1csqFwvK1kt4Ywb7d3MOP5TLXEicVjGa7baRDfw2/LBwsk8LncbfjfPrf6+fs+t/mUMmvD3T9pqFooWkCWtP4letXEdYvFEoqZ7JqlTIuvFfKRVUxBPeSFiBIJ7xh5VJrSm3llKw7iiEgOWSVwtTi3LKKfl95Eo2mwYxqu5iVRaqsbsr8flLoVm5cdIO7m4mjsnWUKbKMyxHfrnZ9yZ047pGLqAIZyvlkkrs1VLZRZswORAkrOr5vCKRgAIk8nJyQTfGbipPIic2tPEUnl/NK51Oamn6lpYnLytcW5Xjd6GTZcy9xJoJYerYSh6++1XbekK/95r18LHzJr4LB9QCqwEeU5j4tjphPjLgqSynVMwVVMXS5dWkm6jXbtzU2PSC1giTiaUl5UjoWqWgGl5qicW4Z0Z1dqdWKauUzykQjlkQuAsadpvbyfD3TWZXMNdDDSvfFdFOv8+GV0g49xoOI6EAIUTYenxmUxIzrfn527p8foR8yGrbjs36+cnTevn1U1rKl1VHLoOlSCCI4nW1NjUrhsdqaZIaxZx6rYRmZL8rekNg12Lu36ZHIwTeK5mdt908ZvXCBHy35d97vRcP2KMMhVaXFtXc0qUshjv+2ms6c/KspoFJD8n8+a7Paya5ohQFbWNvH3lZVS6bVSwcVAVD+INhlXJZ12ORaLMpYAI03Ota3pLONhZzhXQVsXN3FXmPPutCv/uzkTvuU9z7Gk9srFEs5DXUu1GjIzcUjfslx6uHHn5IqV0p1Z2qNmzcoP2HDmh0+o4ePrRHESvdhM9COqerNyfJiZyVcinkV4n4cxx/RMFQDFhyELjhh5qBMbsHk5nlzLpVQxYOrYCYKxtRZh7D+uwW0a6a7j12f0Nnu8d2znCnFR9oBMVpeNOAbt26rkP3HyIxC5pCuG3DmxRtTSgY7tDRnx5X1OPXA3t2qpJd1blr5IQPVAJKjaf4Y8069vZlOWEHZdCyXi25wtSBCxOmaulV89+Vw6poQzmjBm7AuZ4gbBDMCJgpW8WbHgMC+YFP6iQ/XmXQoMr3KEoCV+FB4XhMEf7eOjSgpaWkert7NLihR3dujyveMogpC0pEw5oZG5OzY1gt0Th5U1E2xbN8eM0KG0Z7+8pFOQurNbV6OenBAyxn/8yoVvobArinUA5B2V1ugzkt7NxtPaRQwD20emE5YfBiW51ngyT2bEOdHN6rh+Pwh4yCAUf9Hd0o5dXo9ctaWlnQ0J69qgfq6u7aoJk7Mzry6i/U392l6yhTrxaQ3afphSXdSa2qhGGcl07eViS8iIX9bgiZAn6/j8QHI+qVRoIinM/NE0IF7SzefZR984rfdxd6EdjkNgzPp1eVTiVdy5s3/N4gOB9RvLVVsUBIs3NlPXFgEDTBY1CEpak7+uELL+jJZ57WpdOXdOHtC/rFG29pBbTyU9R85y5qQ0sT6YKXMVCKKr20Bq1AGafZIYagrh5ivmRlvVom3s3bJihegS5WLICxvf14HdAA6K1WKlyEzbGo+aJCMlqk5zKrGjl3Sqsri9AG6BqnHZLAh5dDzc0a3DSsjW3tunTunD51/07t7ojrlVeOaGJqTr5AXGEK4tiNqypYbwBPSkIx8pmcDj/yoFLLS7p2a9YNzxqB5jUP7O00oWuqUFwsOf0mCIJViNUcVfD6xKTWSLoaQlhCNyWaQI8YcEbaE+ed7e3QAJ+mZuc1OT2jJDFdzi/JH0BrPBQk4QIYAzOosJbU7fGa+jf26n9ePqWWalFbP/kIBqtpNVvUt/72O/rgY3u19+AW5etZzWVoZKaXFU206vzVqwrAh2wrV6AQrG0k0WkpZHAVCuCJMIJHMbdDeKyVHR05dkIXb08rz8WYXH6EcdC6FA4oRDyHuL57c79Cjl+jY5PKgd1elPERjqG7HqPf4BwhSEjWqb7d7a2EVwp8COjkpVF9/OED2nXPTvV092to5z2qBNfcpI3SgRVLJsMpza5wPR1cBgMWCOEKFdljfYOHfuLpPfc+V8uX5AOfQ2U0RIhsqaJjZy/o/DujxFmAxR035v0+iBZ7FWW9VZKSWJuj6CwvrhFJCIlQRkfcULM8YTHLC+zhbvF4u/bseUCFokflsk9hyFokRCL3t2vLUI96eroVCLTKV/WrOdKkpbl5FQs5bdmxVbc5rvDkNMx0aW1NxSIexljObDZJvBetCsA3gkoHEpqkcUiH2ykUIR5GrJNoFuFCGYtpn5sk1AYS2wGb6Yq42zgOcEmi4VkqNNdbuwX9FfHv+GPau+9BiBwKEp4JlKmRkC+9eR30adYzTzyodj+eqwIKVR8NT007aTd33XMPoZTHoGdVJHTSeRhrmWYHoxh0O/VQTW0x+ngkKwV6dXWqoHeWMnryqS9ocPN+GuycMitzWDivEyePs3hVwQppi/ZF8sXA0TiTKy+ecukFghncWnkzxCoTOvsO3a94vBNoXFYMPhP315Vay2rs1opuTV1Xmhx49nOfVWcz3i0HtIIRHZJ4lt75Z8dP4kUHq5dd4StWVwhTL7nj9MTaFAmySDispqEDupa+pU/dNwzJghki1dRsWt2dA+rrjGtxNa+Z8WvqqjlayGVU9REmVsT4BHxdYelWgFoEJ5RsqxFq7Rv61d09qOnZFZyIlckhjwdDQKEdGpa6r1nP/+ykltfK+sKnH3Pza/TSFc3PJlkzozQhHYaBFtfSynFc5ZkApAs2vs88eP9zJRaM9A9ohsrbNtCv3//iU9q+fUCRaETLqyVdGBnX+K15NTe366mP/6YODu+kYnbqysQEPThowE5vjfVtRyGsbiqxAmHlhy48Rn4Q08uraiOJHULL5wXfkaIGEmXzqypz7+zSMg38jBYXV5XBI01MIiJtbcqSmwsQwGXQcDGVJRdsjUZp9LZTtAqU9fDANk3Mr2jrYL8by4VSUXv3btZff+NZPfbIfuXoTydnFzWWzGgqQp5UQ+rsvkeRxBA9ba/6tuzSxqEhdfT0KdLcCRUJYqWg+oe2qq9vhyKxDoWYQmQySQUJPbNgicqagk6rRhMThI/5gzpzZUyWkUUsPTc/p8nJKTwKW2iOuF6t4p9KNUyOGTIA7Zn8CpomCBcgiYTs7+snJHw6CcV9/vkXgKycPvu7T6qtrUmJRFyH7tsHgeJmvxfSRZJDbx04uwn4Gx97Sh954rdZqAkBSPhaCNrcLj/XZUETaxPzNOlTs7MQSJQAWjs7212UM1zHrloqFnV8ZESRrg63cm/rH1RfguNEO/R6WLv3Hlb/1gMKYbiChxqUASXa4SNLqZI2dPaopS2hFPz79sSsBgeGFMBDnR0JxbFAWwcwuHurPvrhh5XNVbS8ktHN8Un91Tf/Bq4yoX68V6aq9/QPK9HSodu3R3Xs+HElM1X1bdrGPQUKYETTU1NqaYca4IUq/Yhb/d2WlAIViEClZ7XjwcMqLc5reXxaS8lV8i2iQ4cPSU39mlrK0m7e0tTUpJxCKK5Ez7CuT6TV0zegWMSvMbr+Mv1qX1+vQljtnaujQG1ZO7YPKdEco6aV9IMX/kvLqTyW6USRFW3ZMqAnn/4ttbe0MQKRMoTG9773XX3/+9/XG2+8qv3EcUvLRjkRQoGkv3rtmtqhFAtM3owEukSFg4g3pEWgfXxmXp/77DO6du68umYX+CKOIagDlagS3VF192/X7SlayrmsR31lv+aTy9q9Z7s7HRi9Porgjnp7u91EO3/uiuKM8zYNdqtjQycJ6NGWrcN6++xll4RZ1ezr71E2k1IVAnbkx8c0j2ATE2ONhPYV6XOnoR09FKYioRSA35QVhSaHQmll1papCWC7oTMELcLzlpZSKNukfR94zCVwyytprH5H84uMVgoBECykjlY6srElHjg2pXBTTP0DzFrYent61dreTHwmlKMvfevNsxocHOT7Ppep2sN33LNHDz78OCE0i7UL+sCjj+rxxx5nGJXXGz8/rbnqvBahvQEmbiXGKMXCGiHj0SoNvI8kNuJYopoGQybsHNcUaBkrCsWi8lbCeDXNHAi8J1cKIFysJaY9e7dpahI+xZ4CclMglRPZfL9OXZ3Q/Qf368hPX9fOnZu1dftmtbVEXUb6ytETSoH5Bzd1qLdvA8hR1i9ef0MnT41QSEJYZYpOagtWAwSI6SAd0h8++wfE+R196ctfJH7vkKQwWLC7ShgaY/VBR2pUbONOERjqY7s/rFDEUaKtRVt3bteZE2+plM1peXlNPRtJclpKg+UAqLZtqE1d7QGtkVeLS3k5/V19xNklveO7ClRFNXb5mh5+9DCh0qPjx9/S0aPHte/gvSRoLwvEbIyhffu2Qg0cHX35TQZmVPLWDrUS+8ZqR0Yu68cvvqyhoc1awAOUBJfJ+phAl6AAYQpmlemWtaI1LNvX26Xnvv7HeAfFoBKBADSDMLp09qIK0Aa7tkolDzLN8BmxoupbHjYnHKKBgdiHDnfqof2fhilWSL6C8gy6otGUzrz+Gi3ejDb1+hiB9+rICz/Wqz95WYObhrR5y7DuO/gAeRLTyOUL2tAb1dDwJh7upXougT4zeumnr4D5GTdBqW1U4o14FCqMEn6mbzX4UJW+IhqC9zRBFqkc5h1LhL07tgjGh6cY+Bo7po00kkhBcK9pDBukixdH5GxsybkV1/qAAvBZhYdY/7mldw/P2EUB81JsKupJ7NDk1Lyu37yic+dOq+WVVp0+fZZ8oW5AN69evUae9Om++x5AyZ168cX/1T//y98jZAVBwuoAqstwqDxWtSQ29lFiEmeCG5/yMVKhuLoksI16s2XzkEv2CsWSmmIhK+puGOE796dAbp568zQRAdOzolwP+mjuPRQfGm/ivAbz8xCzhRLwSh3wdAdgh4P6iLObMl/S22cuMTUYpHq2aHb8un7yoxfgOCEo8YC2b73XraI5coeHaSOjkk0DwxqnttRrzHTwdDgYpC7Apyy+kdygFAxCOCyOh4LAbZHZaQlPeGJB8o0r8LAxUDPYykqSKg2MEm5QByZlRmNBgSICT4/f0tydOQ0MDykWbwLqAlBZSBoxm1+5A3OtaUc/9Hj7fmWwxNRcEsJ2QGfOXtLZ02/QUl7U1MyMW9ENHQP0walU2oVIi/sqxrH6srq6jNAmuk08bGxjMY5r7IO/LYH8wLnXJYvmAhS1D35u0ymOjY3KCQKfPs5ifBWwfIELoolmbacqt3d2ucmaw9UByv4io7wmZvm2SLFQAb8zlPtmDQXaVM5OqjjYBP2O6eTpy8CkRwEIj5fKWyLRV5l7lkxwGpgSb12iGMYIYIVzzPgR2CbZJh38xpSiQlvS2ICBb+56x1AMQdkuXbqg2fmbcnJUzTCTD3fwQeORaO1WV0cj4WzinMfdNuT1EFpNTY33f4YOS7wpsUFYGMZqcT442CU/07LmtlZIW0jnL75D/hRJ2iAVOIG1V6HEEdebJeLaEjxAGNn5hlDWW7AhqznCj5xl1g+Cdq5/DAFcRYzBSiMXL7tswbuSRTg/xQNNAwxQ/cEmleD7xkZT6TXiPas1WrjkclJZjktgd4oB0/kLIxC+k+7i8XizVhBk5NII1Pscr7Zy9BAJNfMSz/KpZ+NGV9ASRM2odhwv2rOsmhrLtRcZDbuaAxpHDolehLJ46R+YNbiV2ui6hZGtPzZ2Ey/Tg//rf76qyYUcQtvUzK60DsqSl7eFVKZIJAaD5I3hyhqL+DS3sEz3ZF6hsWe4ygool9bE5CLJ6WNYFSexCBvGIbae+yxa0kRLMzx/wa245tkwjYx9GkQaVFrg2+SD5ZlS0F9TGJcoZotZJnJQZ0RtTAthrYsLi8A1XSJyOvOpkBbW6upqqykM7lqi1mGoeay1upJ1X+tY5pQo6zb8yhP7lyl2WZK3DNwuJlM6N3Jel6/eRDGYJd6z78wYRULF3gXM3LlDdd9HzM5yD4Yhxv0ULZv92HS7QE445IrZ3r6znyiG+/f/+G/A4S196tNPaXiggz6C3porpqjyKSYbxSy9+ly2psUM0641BlsIV7a3J8R00FyHhqt0QOcvXAGVFlCGEKCgAHRqbqW/nVvWWb7LYm0beBnXN2+VaEZsfrl95y4dOHgIyLSpBiMViKDNQq2tBNPwlAMwMEqnN7bK7CqAsew9chh4pDjrH/7uH/WnX/kLvX76Ip0Z4IGSEyBkBu/Yi0AnzfuDQqiFeWSYkFkjQfLM36skWZp2Mqfv/tsPdYvZ/TO/8wm1Lqwqxfjl9h2mY3CbNIK/xYMD4HaZJCtggBCT7h5wv729U929PdADr+agw8lkVu2t9NXJRS3xFqYJ9GOyCFdaISxX1dvfhWggD4pYpbVw3b19EPQr6fLFc/qTP/ozPfsVusMPfkDXJu6gTMl9R+bc99CjyjIgOjexqE88skNLt0dUTK/IF0poPj3HW0EfHNyjWbh/BxX55u1bNNrLJGEFaMy4MOilsKRgpPF4K2QurihThw4GstFoVEmELVCM0gwJCoWqBvp7deXqdV7yNTMFBCwYRY6OXdfBg9vxivnF4ohxDR+7dm1TOMAEEFRKkz/f/Ppf6tiRw5qhzbQ3lnBEOX2h5sq5EyPOxo4mDW0jCSNMJDJe3Rl7R5euXKVVDBMGe11rLkKFp+aympwraCWZZlzEKyBWLBLDgXDEJXxZEnww0UIlnndp8/zcomZnZqElJZeaW8fVtaFdczPTELJmd/Z66Z0xPY3ABQxpbSUtsPvCvndom2LNLUxIIG2cK9MpHXvpRRWAZ6Pm/oCv4kyfPDF/9ez1nuvM40doqJ/+zCflK/HfAAYf0YZcQrMlK0plnRnPanpyUtO3Jnn1yfupLEUFlDFuX6FoffzRhzQ+fot6MKTR0ZtYOwv97lGMFxH5wjixv+CG1xz/eYORD2jj0fT8ArmU0JXRcd2aT2sNSLW2c4XpxTITirXlFYo+bMnQ06g4xY32gGUbnjqwb/+883tf+Njzg4O9X379jdP+U68e0TYoc5my74d82RwyXWh1sd8eYtPltq44luMNOkrZ7Ci5PKd7d+10K/bU1G2odZur6Eef+IjWeM7Kagoq4EORqIv/0+TP6po1+BA6EK+US+vKhQv60he/DGDwTpiGKMtunzYwyxKCsVAQmG9wJmtnrTDEmqLlr331a887u3YMfXt4aGDrjaunDuwf7Gh/aHCjc/QHP9KNaxNaBk1KYLhx/SLxa4hR85dZnJfOtHsHDuzDWrP0tk28rPuJtm3b6raSzRS2CCFlo0MjZGsIFoIsWnLmeLHnZy5UZajc3BwHTfh/EBCz5NwCsd6gzA4V2jApQDlmsA3VL/Iyg+rMQDcYCFQO7T+09I1vPnf24MH93/4/IHCWcBy4Be0AAAAASUVORK5CYII=" style="border: 2px solid white;" width="48" /></td><td><b>james_christie</b><br />
@michaelbolton What happens when the press asks your CEO, "did you follow standards?" when there's been a fiasco?<br />
<a href="https://twitter.com/james_christie/status/471645498278150144">2014-05-28 15:33</a></td></tr>
</tbody></table>
</div>
<blockquote class="tr_bq">
Note: These were <b><u>not</u></b> James' words.</blockquote>
<div>
<b>Gut feeling: Straw man argument!</b></div>
<div>
<br /></div>
<div>
Why?</div>
<div>
<br /></div>
<div>
1. Why would the press be asking about a software testing standard when there is no evidence of the testing standard "working" or preventing similar "fiasco's"? It seems equally plausible for the press to ask if the omens in the CEO's tea leaves were followed!</div>
<div>
<br /></div>
<div>
2. If you have a CEO, that's worth his salt<sub><span style="color: red;"><b>1</b></span></sub>, and there has been a fiasco (presumably a software catastrophe), then I'd be very surprised if the thing that's highest on his mind is some internal witch-hunt.</div>
<div>
<br /></div>
<div>
He's probably gonna:</div>
<blockquote class="tr_bq">
1. Put people on fixing the issue<br />
2. Put people/teams on liaising with the customer (if it's one big and, understandably, angry one)<br />
3. Put people on understanding what went wrong<br />
4. Understand if the company did what they thought was right / good enough<br />
<br />
I'd be surprised if a company that used technically competent people would have the/a root cause being that a template (or group of templates / standard) was not followed - especially if there is no empirical study (evidence) to demonstrate how the standard (template) <u>might</u> avert such a fiasco.<sub><span style="color: red;"><b>3</b></span></sub></blockquote>
<blockquote class="tr_bq">
<span style="color: red;">If a catastrophic failure has been found to be caused<b><sub>2</sub></b> by not following a template then I would love to hear about it.</span></blockquote>
I've been in those discussions with responsible managers and execs - and (maybe this is my good fortune of dealing with competent managers)<sub><span style="color: red;"><b>4</b></span></sub> they're more interested in that the technical influential/responsible people do what is right / good enough for the needs of the business, rather than be concerned about being fully or partially compliant to a particular template.<sub><b><span style="color: red;">5</span></b></sub><br />
<div>
<br /></div>
<div>
<div>
Yes, I'm not naive, I know some people, projects and company cultures follow some template as an ass-covering exercise. I've been there.<sub><b><span style="color: red;">1</span></b></sub></div>
<div>
<br /></div>
<div>
So... The straw man argument is constructing a need to explain whether or not a particular template is used, when the need for that template is not established.</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Reasoned argument?</b></div>
<div>
<br /></div>
<div>
Ok, so it was time for me to revisit the webinar, where at timestamp ~33-34mins, one hears:</div>
<blockquote class="tr_bq">
<i>"If you are still doubtful about using these standards, then I have a question for you. </i></blockquote>
<blockquote class="tr_bq">
<i>Can you afford not to? </i></blockquote>
<blockquote class="tr_bq">
<i>Imagine you're responsible for the testing of an important application. It could even be business critical, or safety critical, and something goes noticeably wrong with the application in use. </i></blockquote>
<blockquote class="tr_bq">
<i>Even if really good testing could’ve missed the bug, how easy will you find it to explain to the business your testing missed the bug and no, your test processes do not comply with international testing standards; that they are the ones we've used for years and no, you don't have them fully documented and justified. </i></blockquote>
<blockquote class="tr_bq">
<i>So, can you afford not to use them?"</i></blockquote>
<div>
<div>
<br /></div>
<div>
First impression: You should follow a template.... just in case.</div>
<div>
<br /></div>
<div>
Second impression: Slightly sinister tone. </div>
<div>
<br /></div>
<div>
Ok, let's take a closer look. Closer inspection:</div>
<div>
<br /></div>
<div>
1. The first two sentences are plain rhetoric, trying to establish doubt in the listener - doubt due to not following a standard. This is "rhetoric" because there's no argument/justification about why a standard would help you avoid this situation.</div>
<div>
<br />
2. "<i>if really good testing could’ve missed the bug</i>" - good testing can miss bugs. This is not necessarily anything to do with a process. Good processes can miss bugs.<br />
<br />
3. "<i>how easy will you find it to explain to the business your testing missed the bug and no, your test processes do not comply with international testing standards</i>" - this statement is intending to sow doubt. But the international testing standard has no evidence of being useful, appropriate or a proven track record.</div>
<div>
<br />
I could go on and pick it apart, displaying the parts which are negative evaluations, argument precursors etc, but you get the picture. It says very little more than an innuendo of repercussions if you don't use the standard.<br />
<br />
<b>In Summary</b><br />
<br />
<ul>
<li>If someone tries to scare you, or generate doubt and worry, ask to see the evidence, the track record, the money. Look out for straw man arguments, misleading and off-the-point comments and allusions to superstition or unjustified belief.</li>
<li>Successful CEO's don't usually look for scape goats in their staff, don't usually follow checklists without a good business reason. </li>
<li>For me the jury is still partially out on the new software testing standard - I will publish some thoughts on its content at a later stage. But, just because it's been labelled a standard does not make it comparable to other ISO or IEEE standards where interoperability or conformance to a standardised <u>result</u> is usually the goal - here the goal seemed to be to create a standard (job done! Is that the business case? WTF! )</li>
</ul>
<br />
<b>Notes</b><br />
<b>1. </b>If you have a CEO (or responsible manager) that doesn't trust the technical competence of the organisation then either (1) change the organisation, or (2) change organisation. <b>And Yes! I have done both in the past!</b><br />
<br />
<b>2. </b>Note, if a template is intended to replace competence then this should be declared also. (This puts an additional burden on the template, guidelines and usage of the template.)<br />
<br />
<b>3. </b>The existence of a standard, template or checklist does not stop mistakes in following such a checklist<br />
<br />
<b>4. </b>Competent managers (in my experience) usually do not chase paper trails to a template, and usually trust their staff. I'll give some tips on how to deal with managers that are tempted by reliance on checklists over competent staff in a separate post.<br />
<br />
<b>5. </b>Businesses tend to adopt (or use) the standards that make business sense, rather than adopting a standard purely because there is a standard.<br />
<br />
<b>References</b><br />
[1] Eurostar webinar: <a href="http://www.eurostarconferences.com/community/member/webinar-archive/webinar-75-iso-29119---the-new-set-of-international-standards-on-software-testing">Webinar 75: ISO 29119 - The New Set of International Standards on Software Testing</a></div>
</div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com3tag:blogger.com,1999:blog-5948141587422980338.post-62473562897031412162014-04-21T12:33:00.000+02:002014-04-21T12:35:19.887+02:00On Test Results and Decisions about Test Results<div style="font-family: Helvetica; font-size: 12px; text-align: left;">
<i>A Test Result != Result of a Test</i></div>
<div style="font-family: Helvetica; font-size: 12px; text-align: left;">
<i>An Observation != Result of an Observation</i></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px; text-align: left;">
<br /></div>
<blockquote class="tr_bq">
It’s not the test result that matters, but the decision about the test result!!</blockquote>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Pass-Fail-Dooesn’t Start-Inconclusive-Can’t Execute</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
How you come to these results (and there are more) is interesting. But from a testing viewpoint what you do, what actions you take based on such results is <span style="text-decoration: underline;"><b>very</b></span> interesting.</div>
<blockquote class="tr_bq">
<i>“If a tree falls in a forest and no one hears it, does it make a noise?”</i></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
Or</div>
<blockquote class="tr_bq">
<i>“If a test result is obtained and not used or considered, was it worth it?”</i></blockquote>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Used?</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Did it confirm an expectation?</div>
<div style="font-family: Helvetica; font-size: 12px;">
Did it contribute to a body of evidence that a feature, product or system is behaving within bounds?</div>
<div style="font-family: Helvetica; font-size: 12px;">
Did it help understand risks with the system?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
If the answer is no, then it might be time to take a good long hard look at what you’re doing…</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Body of Evidence / Evidence</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
All test observations and test results are not equal. They contribute to the picture of a product in different ways. But that picture is not necessarily a paint-by-numbers book. It’s not something that you can necessarily think I’ve ticked all the boxes, I’m finished.</div>
<blockquote class="tr_bq">
Note: In many cases, Testing is not painting by numbers! </blockquote>
<blockquote class="tr_bq">
<i>Unless you’re a doctor finding no pulse and rigor mortis has set in!</i></blockquote>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
It’s about making sense of the observations.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<u><i>The 1% Problem Soundbite</i></u></div>
<div style="font-family: Helvetica; font-size: 12px;">
Suppose 1% of your tests fail. Suppose you’ve seen a problem that 1% of your customers will experience.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<u><i>The 50% Problem Soundbite</i></u></div>
<div style="font-family: Helvetica; font-size: 12px;">
Suppose 50% of your tests fail. Suppose you’ve seen a problem that 50% of your customers will experience.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Based on this information is it possible to say anything about the product?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
No. <i>But you might have something that says, “need more information”.</i></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
These are what I think of as context-free reports.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
From those soundbites, you don’t know anything about the nature of the problems, product, market that the product might be used in, circumstances for usage, etc, etc.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Suppose the 1% problem is a corner case - not allowing installation in a geographical location if some other feature is active - that might affect how the product is launched in that market. Suppose it’s something “cosmetic” that might annoy 1%, but not prevent the product being used. These two different types of observations (results) might produce totally different results.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The 50% case - is this a break in legacy, some feature that customers are using and needs to be operated differently; or is it a new feature interacting differently with existing features (e.g. flagged by some automated scripts) that hasn’t been launched yet and might be only for trial/“friendly users”. Again these observations (of essentially first order feedback, after Weinberg, ref [3]) might have totally different results.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Decisions and Supporting Data</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
1. Are you a tester that decides when testing is finished and a product can ship?</div>
<div style="font-family: Helvetica; font-size: 12px;">
2. Are you a tester that tries to give your boss, project manager, stakeholder a good account of the testing done on a product? Might you even give some insight and feedback about what that testing might mean?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Ok, assuming you’re in the second category…</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
What does your test observation tell you?</div>
<div style="font-family: Helvetica; font-size: 12px;">
What do your test observations tell you?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I’m reminded of a model I’ve helped use in the past about understanding test results, ref [1]. But now I’m looking at the flip-side, not what a “pass” means but what a “not-pass” might mean. A simple version of a test result / observation might look like:</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG90hdYifttMIesTqGudWiHkcff-1lyFMVqYHLLQgSBwLe4JIA9SFQZr38zexLqKZ0I7W3hDKlV1hd3B_l8WO6O_P8Jzoqx6FaZmeVYQvPMLdmUvQGiTkeq3_cSNSI275iBQL-LRUPerdM/s1600/TestResult-Observation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG90hdYifttMIesTqGudWiHkcff-1lyFMVqYHLLQgSBwLe4JIA9SFQZr38zexLqKZ0I7W3hDKlV1hd3B_l8WO6O_P8Jzoqx6FaZmeVYQvPMLdmUvQGiTkeq3_cSNSI275iBQL-LRUPerdM/s1600/TestResult-Observation.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Note, this is a typical interpretation when a test result is deemed to be of the form OK / NOK (including inconclusive, don’t know etc.) The implication of this is that when the desired test result (OK, pass) is obtained then that “test case” is done, i.e. it is static and is unchanged by environment or circumstance. This might be typical in conformance testing, where responses within a desired range is expected, usually when the test subject has a safety component (more on this in another post).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
But, if the results are not black-and-white or can be open to interpretation (as in the 1% problem soundbite) then a different model might be useful.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKe4rIhoPe4JFljJYlV4561IHo4GnH8-nmj-3BzUFCtJqRAhVqHagKK0WKze4XH3WH1vwOFytIesRP8I5CKvBzFUQxMQ-leRFx67sfZIryBJ0AeZm4wSv5cgQXHcujrtadLKDaRaSTHi9G/s1600/TestResultDecisions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKe4rIhoPe4JFljJYlV4561IHo4GnH8-nmj-3BzUFCtJqRAhVqHagKK0WKze4XH3WH1vwOFytIesRP8I5CKvBzFUQxMQ-leRFx67sfZIryBJ0AeZm4wSv5cgQXHcujrtadLKDaRaSTHi9G/s1600/TestResultDecisions.png" height="261" width="400" /></a></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
This model emphasises the project, product and testing elements to consider - these may change over time (within the same project), between product versions and between testing cycles (or sprints). I’ve drawn the line between product owner and product decision as a thick black line to highlight that this is the deciding input.</div>
<div style="font-family: Helvetica; font-size: 12px;">
More on testing and silent evidence can be found in ref [2].</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
A larger version of this picture can be found <a href="https://drive.google.com/file/d/0BxK-QYKaKwoPQzk1NlR0cjQxaWs/edit?usp=sharing">here</a>.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>References</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
[1] Testing Chain of Inference: A Model <a href="http://testers-headache.blogspot.com/2012/08/testing-chain-of-inference-model.html">http://testers-headache.blogspot.com/2012/08/testing-chain-of-inference-model.html</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[2] Silent Evidence in Testing <a href="https://draft.blogger.com/goog_1766236999">http://</a><a href="http://testers-headache.blogspot.com/2012/03/silent-evidence-in-testing.html%E2%80%8E" style="font-family: Arial;">testers-headache.blogspot.com/2012/03/silent-evidence-in-testing.html<span style="color: grey;"></span></a></div>
<div style="font-family: Arial; font-size: 12px;">
[3] Quality Software Management: Vol 2 (Weinberg; 1993, Dorset House)</div>
<div>
<span style="color: grey;"><br /></span></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-41757975036808703752014-04-20T00:14:00.000+02:002014-04-21T14:24:21.119+02:00On Thinking about Heuristic Discovery<div style="font-family: Helvetica; font-size: 12px;">
How do you spot new heuristics?</div>
<div style="font-family: Helvetica; font-size: 12px;">
How do you identify new heuristics?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
These were questions that were “touched on” during parts of SWET6* (a small part in the open season of James Bach’s discussion and during a car journey from SWET6).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>“Pause & Reflect” / “System 2 Re-Insertion”</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
During the open season of James Bach’s topic on documenting heuristics from a test activity I remarked that I’d spotted an heuristic that he didn’t appear to notice. It was the activity of pause and reflection: putting down the work for a while, revisiting, correcting and re-working. This can cycle through after bouncing ideas around with colleagues or getting their feedback, followed by then further ‘pause and reflection’ cycles until the result was deemed good enough. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Reflection</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
Due to the nature of how the activity and report had been made it incurred a series of pauses (interruptions or breaks) which then (I assert) meant that some of the parameters of the context (or frame through which you look at the work) has to be remembered, reviewed or picked-up again - this can mean that the frame gets slightly altered, i.e. “you look at it with a fresh pair of eyes” and, hey presto, see something different or new.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<div>
I do not claim to be the first to spot the power of this activity - but I think of it as re-activating the System 2 mode of thought (according to Kahneman, ref [1] this is more deliberate and takes conscious effort to use). The action of putting the work (report) aside for a time and then revisiting means that some of the context is forgotten and so has to be remembered when the work is picked up again. This is the re-analysis in the system 2 mode of thought.</div>
</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Noticing New Heuristics</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The pause and reflection about the observations is very important. It gives the basis for new pattern recognition. Either you notice something different that you don’t recognise or that is a little different. Typically this might be a procedure that you followed, for example:</div>
<ul>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">I tend to find problems of type X when I do A, B & C, </li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">I tend to find race condition problems when I alter (shorten and lengthen) the timing between certain test steps.</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">I tend to notice new/different patterns when I pause and reflect on a set of actions and compare with previous times I did some similar action.</li>
</ul>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Blink Comparator / Trawl and Compare</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The activity of reflection and noticing new patterns is something I think of as a blink comparator test. A blink comparator test is something astronomers used to use to find new stars. A reference pattern is used to compare with a current observation. It’s like trawling through previous experience (observations) and comparing with the latest experience. In the example above it might be:</div>
<ul>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">When I find problems of type X (in the past) what was the same/similar to the current actions (A, B & C)?</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">When I have found race conditions in the past were there any timing differences in the actions I made in the test steps? (When I change the timing between steps - for the same types of steps - I find race conditions more often.)</li>
<li style="font-family: Helvetica; font-size: 12px; margin: 0px;">When I notice new patterns is there some key step that is common? (Pause and reflect,)</li>
</ul>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Visualization</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Sometime in January <a href="https://twitter.com/YorkyAbroad/status/421283387434082304/photo/1">I tried to sketch this connection</a> between Pause & Reflect and how I noticed new heuristics. This is my latest version that I think reflects the process I tend to follow:</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBYzdP5B5hRrokhbXCX8X02vKTXpk-QdEQJ-PDdNTen5DjT0QEaZ8YhDnWHguYYO0W4EddgFRljkMvpmXMtK7G7vPVXRJpNSrtCf1d127Awu5B_Xwl6LnLsZ_PW-fSWhW1nkliAj22KB9G/s1600/Heuristic+Discovery.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBYzdP5B5hRrokhbXCX8X02vKTXpk-QdEQJ-PDdNTen5DjT0QEaZ8YhDnWHguYYO0W4EddgFRljkMvpmXMtK7G7vPVXRJpNSrtCf1d127Awu5B_Xwl6LnLsZ_PW-fSWhW1nkliAj22KB9G/s1600/Heuristic+Discovery.png" height="320" width="400" /></a></div>
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
A larger version of this model can be viewed <a href="https://docs.google.com/file/d/0BxK-QYKaKwoPbEE2bnByM0NfaEk/edit?pli=1">here</a>.<br />
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<i><u>"Pause & Reflect Heuristic"</u></i></div>
<div style="font-family: Helvetica; font-size: 12px;">
The model is best read entering the “Pause & Reflect Heuristic” box. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Here observations of actions (or reviewing a test report) is done - with a pause & reflection in-between - so the two “Solution / Result” clouds may be different. The “Evaluate” cloud is where a difference is noticed. In some cases this feeds back as re-work, in other cases it causes a question which then feeds into the “Proto-pattern” cloud. </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br />
<u><i>Blink Comparator Test / Trawl and Compare</i></u></div>
<div style="font-family: Helvetica; font-size: 12px;">
This is the step where a search of the difference (comparator item) is made. The search might be a re-analysis of similar experiences to understand what caused the different result. The result / assessment might be that it was a random action, or something outside my control, that caused the difference**. At other times it might be, “step X was made when the system was in state Y” - that might be enough to give me a new useful rule of thumb (heuristic).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
The next step would be to search if it was something in use elsewhere or known already. If new, then classify (name and describe) and publish (talk about or discuss it).</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br />
<i><u>Question, Test, Evaluate</u></i></div>
<div style="font-family: Helvetica; font-size: 12px;">
But the work doesn’t stop there. When using this heuristic new observations about it’s use should be gathered. Does the heuristic work as before? Is there a new (maybe more subtle) pattern or action that makes it useful. This might result in a new heuristic, a refined heuristic or a restriction to fewer applications. This is the “Question, Test, Evaluate” yellow box. It is analogous to the scientific method where the heuristic is the object (hypothesis) tested.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Testing the model</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I’m currently collecting new observations of noticing new patterns and trying to construct ways I can test this model.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
In the meantime, comments and suggestions are very welcome.</div>
<div>
<br />
<div style="font-family: Helvetica; font-size: 12px;">
<b>References</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
[1] Thinking, Fast and Slow [Kahneman; 2012; Farrar]</div>
</div>
<div>
<br /></div>
<div>
<div style="font-family: Helvetica; font-size: 12px;">
*SWET6 was at Hönö Hotell, Öckerö, Sweden, 19-20 October 2013, Attendees: Martin Jansson, Steve Öberg, Saam Koroorian, Mikael Jönsson, Anders Bjelkfelt, Marcus Möllenborg, Klaus Nohlås, Simon Morley, Henrik Emilsson, James Bach</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
**Note, a “different result” might be, when I tackled the problem now I got a different result than previously. What made the difference? This can applies to systems of people interaction too.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
</div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com1tag:blogger.com,1999:blog-5948141587422980338.post-30182793855123730542014-04-19T11:57:00.000+02:002014-04-19T12:06:51.696+02:00Simplistic Views, Complex Subtexts and Whaa?<div style="font-family: Helvetica; font-size: 12px;">
Recently I read an opinion piece, ref [1], from a Technical Evangelist (more on that label later). My first impression of it was a muddled message. Which point was the piece trying to make?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
So, I made a quick close analysis of the text and posted some questions, which basically amounted to, “do you have data/evidence to back this up” and “can you clarify what you mean”, ref [2]. I’ll be happy if I get a response.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Close Analysis</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Close analysis (after ref [3]) is a form of critical analysis where the text is dissected into argument markers, assuring and guarding terms, discounting and evaluative terms and rhetorical devices. The aim is to analyse a text sympathetically - i.e. to try and make the person’s argument as good as possible - to get to the real meaning of what they are trying to say.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
This is my usual first step where I need to pay close attention to text. In the case above I found it unclear and bordering on incoherent hand-waving. Sometimes where the people are responsive then there might be a discussion that starts with, “can I ask some questions?” (ref context-free questions, ref [4])</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
Yes, it’s time to stop, think and ask questions - what is the message, where is the supporting analysis or evidence.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
I will probably be doing this more and more. I’m currently making a close inspection of Taylor’s Principles of Scientific Management. I’ve also recently had reason to look at Program Test Methods and I’ll be revisiting that to provide a commentary of it. I started looking at these because they either get referenced in several places or are the basis for further work. I may end up revisiting more of my software testing library and other literature (as I have done with “regression testing” in wikipedia and the ISTQB glossary, ref [5]). </div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>Some useful Heuristics for Potential Spotting Problematical Articles</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Evidence or Data?</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
If a claim is made is there a reference to some analysis that you can read or how that conclusion was discovered?</div>
<div style="font-family: Helvetica; font-size: 12px;">
Beware of data being retrofitted (cherry picked) to support a claim.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Do the claims appear to be anecdotal or have a single source?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Meaning</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
Is the meaning clear?</div>
<div style="font-family: Helvetica; font-size: 12px;">
Does the conclusion follow from the argument? </div>
<div style="font-family: Helvetica; font-size: 12px;">
- These two are the main parts of close analysis.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Why was the piece written? Was it an opinion or were they trying to influence opinion?</div>
<div style="font-family: Helvetica; font-size: 12px;">
How was the reaction to the piece? Peer review, open comments, discussion or other commentary?</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<span style="text-decoration: underline;"><i>Labels</i></span></div>
<div style="font-family: Helvetica; font-size: 12px;">
The labels people use are important. Evangelist has many connotations (to me) of preaching someone else’s thoughts, words and ideas. Ok, so if you don’t think for yourself should I go to the source?</div>
<div style="font-family: Helvetica; font-size: 12px;">
BTW, the same (to me) applies when I hear someone describe themselves as a “passionate <whatever>” - I wonder does that mean they’re enthusiastic, irrational emotional or irascible.</div>
<div style="font-family: Helvetica; font-size: 12px;">
Labels like expert and authority should be distinguished whether it is self-appointed or peer-recognised.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>And finally….</b></div>
<div style="font-family: Helvetica; font-size: 12px;">
I know this won’t stop articles and books where “<a href="http://oxfamblogs.org/fp2p/what-brits-say-v-what-they-mean-handy-de-coding-device/">I only have a few minor comments</a>”, but it’s important to distinguish a simplistic or nonsensical view from a complex subtext or just plain BS.</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
If you don’t understand it or where a claim is made without evidence, call it out!</div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
<b>References</b></div>
<div style="font-family: Helvetica; font-size: 12px; min-height: 14px;">
<br /></div>
<div style="font-family: Helvetica; font-size: 12px;">
[1] LinkedIn “The Hard Truth about Software Testing” <a href="http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing">http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing</a></div>
<div style="font-family: Helvetica; font-size: 12px;">
[2] Comment: LinkedIn “The Hard Truth about Software Testing” <a href="http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing?goback=%2Empd2_*1_*1_*1_*1_*1_*1_20140411172902*546939713*5the*5hard*5truth*5about*5software*5testing&anchorTime=1397426609731&deepLinkCommentId=5861232018901209088&trk=mp-detais-tw-cmnt">http://www.linkedin.com/today/post/article/20140411172902-46939713-the-hard-truth-about-software-testing?goback=%2Empd2_*1_*1_*1_*1_*1_*1_20140411172902*546939713*5the*5hard*5truth*5about*5software*5testing&anchorTime=1397426609731&deepLinkCommentId=5861232018901209088&trk=mp-detais-tw-cmnt</a> </div>
<div style="font-family: Helvetica; font-size: 12px;">
[3] Understanding Arguments, 8th edition, chapter 4 [Sinnott-Armstrong, Fogelin; 2009, Wadworth]</div>
<div style="font-family: Helvetica; font-size: 12px;">
[4] Exploring Requirements: Quality Before Design (Gause, Weinberg; 1989, Dorset House)</div>
<div style="font-family: Helvetica; font-size: 12px;">
[5] The Tester’s Headache: Another Regression Test Trap <a href="http://testers-headache.blogspot.se/2013/03/another-regression-test-trap.html">http://testers-headache.blogspot.se/2013/03/another-regression-test-trap.html</a> </div>
<div>
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-41192370494131934872014-01-03T00:40:00.000+01:002014-01-03T00:40:35.187+01:00Some SWET6 notes<span style="font-family: Arial, Helvetica, sans-serif;">A while ago, but here are some fragments…..</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Martin Jansson arranged SWET6 at Hönö* - good job Maritn!</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">The theme of the weekend was a test-related topic that interested us from the past year.</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">We gathered, a mix of experienced/less-experienced testers, experienced/less-experienced peer conferencer attendees - a number of us knowing each other well - a good mix.</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">Talk 1 - Simon - Online Coaching </span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">I started with a some experiences of my online test coaching - I had been doing during the year - and raising up some of the patterns I had noticed. Some of these were about the dynamics of the coaching - whether the session was a “challenge” or tutorial or pair-work. I reflected on how I had seen behaviours change between sessions, partly based (my hypothesis) on the previous session (a feedback loop).</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">The open season covered a number of threads:</span><br />
<br />
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Student-Coach relationship</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Growing coaching sessions to have more students</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Differences between coaching and mentoring</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">How did the coach affect the start-up time for the student</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Teaching testing is a great learning experience</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">How do you know when you are done coaching a student?</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Discussion around “tricking” students and ethics</span></li>
</ul>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The topic was a tricky one to tackle in a general form, but produced a wide range of interesting threads and the discussion was fruitful. On this last thread James evolved a new suggestion (model) to the ethics problem - something that deserves a separate write-up along with coaching in general. </span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Several in the group weren’t familiar with how a coaching session works, and I took up an example of a coaching session I’d done with James - which he was willing to demonstrate for the rest of the group - this, combined with the debrief, was a good demo of how a coaching session looks like “up close” - including the learnings and power of using an apparently “simple topic”.</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">Talk 2 - Henrik - On a short/intense test period he’d been involved in during the year.</span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Henrik picked a topic where he’d returned to “grass roots testing” - not teaching, not talking about it, but actually doing it. And that’s one mark of a good tester - getting into the details of a test discussion, with “why I did it this way and not that…”</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">The discussion was about searching for the “important bugs”, the strategy and risk considerations that went it this. Some of my comments/reflections during this talk were:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">Q: What was the value of the information in the report?<br />C: Bugs don’t matter, it’s the information about the product that matters.<br />Q: How do you estimate how many unimportant things that you don’t know?</span></blockquote>
<br />
<b><span style="font-family: Arial, Helvetica, sans-serif;">Puzzle</span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">We had a short gap between the end of the 2nd discussion and the scheduled end for the day, so we threw it open to puzzles. James gave one that we weren’t familiar with. </span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">I enjoy being fooled by puzzles (or falling into the traps) - it’s good ammunition when examining what assumptions you made and why. Good personal retrospective material, quite simply!</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">Lightning Talks </span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">A number of different of lightning talks on different topics - I didn’t take notes for this section, so other participants feel free to add in the comment section.</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">My own talk was a brief walk through some of the research/meanderings I’ve been doing in the last year connected with reframing testing as a hypothesis/experiment-driven activity - leading through the works of Gigerenzer (risk and number illiteracy) and the prosecutor’s fallacy, DiCarlo (How to be a really good pain in the ass), Collins (Rethinking Expertise) and Popper (Logic of Scientific Discovery)</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">Talk 3 - James - A Testing Challenge and subsequent report</span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">James talked about an experience of a test challenge he had gotten earlier in the year, which had been quite involved and whose reporting had been drawn out. He had subsequently decided to turn the report into an exemplar of the processes he had gone through, techniques and analyse applied, re-visit, re-work and documentation of mistakes and thoughts to show the many non-apparent activities that go into producing the final report.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">A very interesting taster of the challenge, formal report and extended report.</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Some comments/reflections I had at the time:</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">C: The FORM of the reporting AND the testing contributed to the result, i.e. The pauses AND reflection (and maybe discussion/consultation with colleagues) contributed to the quality of the work. </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;"><i>At the time I declared this as an heuristic: The "pause and reflect" (and, optionally, consult) heuristic; or the “System 2 Insertion heuristic” (in deference to Kahneman & Torkel Klingberg)</i> </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">Q: Did you use all testing techniques you know about? </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;"><i>At the time I commented that a heuristic (I used) for noticing heuristics or patterns was “blink comparator testing”.</i></span></blockquote>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">Checkout</span></b><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">All were happy, had spent a weekend surrounded by sharp minds, had learnt something and achieved the goals of the weekend. I personally had material to further expand on…. Quite simply a good weekend!</span><br />
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="min-height: 14px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<b><span style="font-family: Arial, Helvetica, sans-serif;">*SWT6 details</span></b><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Date:</b> 19-20 October 2013</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Place:</b> Hönö Hotell, Öckerö, Sweden</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Attendees:</b> Martin Jansson, Steve Öberg, Saam Koroorian, Mikael Jönsson, Anders Bjelkfelt, Marcus Möllenborg, Klaus Nohlås, Simon Morley, Henrik Emilsson, James Bach</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVj4N4kFlR2jH7J_u8X6eizlUpkGN_wP4LbsE-OipPJzAA9cd350FTjG-K5y9Bs4WXdi-JdUt2oqMBr5iwRKqaIokulcgRrw7Xi9SdVVBnqOkApozPy_2lEf-DpKdF9hqHfBDpuYrPxBG2/s1600/600_302253362.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVj4N4kFlR2jH7J_u8X6eizlUpkGN_wP4LbsE-OipPJzAA9cd350FTjG-K5y9Bs4WXdi-JdUt2oqMBr5iwRKqaIokulcgRrw7Xi9SdVVBnqOkApozPy_2lEf-DpKdF9hqHfBDpuYrPxBG2/s1600/600_302253362.jpeg" height="300" width="400" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<div>
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-89606353085953673502013-09-07T15:13:00.000+02:002013-09-07T15:18:10.125+02:00On Being Replaceable, Role Traps and Adding ValueI said to one of my managers several years ago that one my main goals towards the team and organisation was to make myself replaceable.<br />
<br />
He nearly fell off his chair....<br />
<br />
As I was a leader within the organisation - if I become replaceable what does it mean for the position I have or the value of the work I'm doing? Oh, I say this to most managers (partly as a test :) )<br />
<br />
It is important to distinguish how people are viewed within teams, projects and organisations. In this context there are two main views:<br />
<ol>
<li>What they bring in terms of personal skill, knowledge and capability.</li>
<li>What they bring in terms of operational responsibility or ability.</li>
</ol>
<b>Traps</b><br />
<blockquote class="tr_bq" style="text-align: center;">
Where teams and organisations have roles or specialized people (e.g. testers) then there is a risk of attaching someone's behaviour to that of the role.</blockquote>
<blockquote class="tr_bq" style="text-align: center;">
Or the other way around, </blockquote>
<blockquote class="tr_bq" style="text-align: center;">
The skill of a role is often thought of in terms of a person/s that perform it.</blockquote>
<blockquote class="tr_bq" style="text-align: center;">
(or have performed it). </blockquote>
<blockquote class="tr_bq" style="text-align: center;">
Or:</blockquote>
<blockquote class="tr_bq" style="text-align: center;">
Experience is implicitly labelled as an exemplar. </blockquote>
<b>Example</b><br />
In the past I have sat in a coordination meeting with a team and asked an incisive question such as, "Customer X have feature Y adapted/changed in their network and so your development of feature Z should also look into interactions between features Y & Z". This means that a potential risk is reduced - or at least some investigation is started to reduce a potential risk.<br />
<br />
Now, if I miss a similar meeting with another team, there might be such a question not asked that might mean such a risk is not caught or it is thought about later, meaning some additional work. <i>(Note: These coordination meetings might be some form of reference/project or expert aid to the teams.)</i><br />
<blockquote class="tr_bq">
<b><i>Question: Who was the target audience for my question? </i></b></blockquote>
Many people would think it was the team doing development of feature Z. My target audience is <u><b><i>everyone</i></b></u> though - it's not just about the question, it's also why it is asked (some might call that the context) - in this case to help all realize something they hadn't seen beforehand.<br />
<br />
I don't ever see myself as the person who asks questions others don't - but more as someone who might help others ask better/different questions next time. If I haven't passed on some of that capability then I've failed.<br />
<br />
<b>Adding Value</b><br />
There are two main ways I add to the team, project or organisation.<br />
<br />
<ul>
<li>I point out the differences between what I am bringing as me, and what I am bringing as "performing role X". They might overlap at times, and not at others - and it's important to help others with that difference.</li>
<li>I ask "why?" a lot. Not to be a pain (even if that's how it might sometimes be seen), but to help understanding.</li>
</ul>
<div>
But, how is this "adding value"? It's highlighting behaviours that others can adopt that are not "owned" by a role.</div>
<div>
<br /></div>
<br />
<b>The "Why?" Question</b><br />
One of the most common questions I ask is "why?". It's a sign of wanting to learn what someone else is thinking. It's a sign of wanting to flush out, and clarify, assumptions. <b><i>The "why" question is one of the cogs of dialogue and understanding.</i></b><br />
<br />
So, if I am the only one who is looking out for dialogue and understanding then there might be a problem in the organisation. However, in my experience, someone has attached the "why" question or "the types of question that I ask" to the role I had - so it becomes "role ABC asks those type of questions".<br />
<br />
That's where I need to remind people around me that these are not my questions, I don't get territorial about such questions, and that if I'm not around and the question pops into your head, go ahead and ask it.<br />
<br />
<b>Learning Organisations</b><br />
Organisations, teams or projects that want to grow and learn must be very careful about roles -> sure, if someone is the designated decision maker let them make it. Until that point, there's usually plenty that can be done without the decision-maker - including asking questions.<br />
<br />
Sometimes people (teams) need to be given permission to think for themselves - strange as that may sound.<br />
<br />
<b>Lessons</b><br />
<blockquote class="tr_bq">
<div style="text-align: left;">
</div>
<ul>
<li>It's not what skills you bring to the table, it's what you leave behind for others after you've gone.</li>
<li>It's not what attitude you bring to the party, it's the positive change in attitude you leave behind that's important. <i>(Sometimes, that means more people are prepared to ask, "why?")</i> </li>
</ul>
</blockquote>
<b>Doubt</b><br />
A typical question I get about achieving replaceability is: don't you do yourself out of a job, or no one needs you after you've improved others?<br />
<br />
In a team, or organisation, that has a constant ambition of improvement that is never a problem - there is always a new problem to work on. As Weinberg said (I think), when the most important problem gets solved, problem #2 gets a promotion. Sure, it's a different problem and may take you out of your comfort zone, but ultimately, that's how you improve.<br />
<br />
<div style="text-align: center;">
<b><i>Some people treat knowledge as power and hold onto it. Unfortunately, those are the folks that can become one-trick ponies or get bypassed by progress...</i></b></div>
<br />
<b>And finally...</b><br />
So, to me, being replaceable is positive - it means I've added value - it means I've given others a tool for thinking more clearly - it means I can carry on improving, learning and adding value.<br />
<br />
<br />
<span style="color: red;">Are you adding value? </span><br />
<span style="color: red;">Are you leaving something on the table for others when you move on?</span><br />
<br />Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com8tag:blogger.com,1999:blog-5948141587422980338.post-71315127138953615552013-08-27T23:14:00.001+02:002013-08-27T23:14:31.728+02:00Joining the ISST<div class="p1">
Some weeks ago I was asked to join the ISST (<a href="http://commonsensetesting.org/">http://commonsensetesting.org</a>) as a founding member. The group was presented as a force for the spread of context-driven testing* - it's mission and objectives were something I saw as a force for good. (* I've never called myself a context-driven tester - and some may think I have my own cdt principles - anyway….)</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Why Join?</i></b></blockquote>
<div class="p1">
I didn't know who else was in the organization more than the board - it wasn't sold to me as though, "<i>Hey Dave Merenghi is joining so what about you?</i>" - so why did I join?</div>
<div class="p2">
<br /></div>
<div class="p1">
I knew the board, I liked the mission - and I knew that these were probably guys that could achieve that mission - that was a differentiator (for me).</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>No brainer...</i></b></blockquote>
<div class="p1">
Yes, working for these causes was a no-brainer (for me), but as in most things I do, I occasionally let my tester brain off the leash. I asked questions about the mission, key differentiators for the organization, fees (their use and what a prospective member might get for it), how members and founding members could contribute and organizational structure</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Good enough...</i></b></blockquote>
<div class="p1">
I got answers - and answers that satisfied me. But the key part for me - I could be in this organization and so help drive the direction of it. </div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Got a question?</i></b></blockquote>
<div class="p1">
I think the board has been very open about answering questions from members and potential members. Need an answer? Try the contact page: <a href="http://commonsensetesting.org/contact/">http://commonsensetesting.org/contact/</a></div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Free lunch...</i></b></blockquote>
<div class="p1">
The fee? Well, it's a non-profit organization and - as I understand it - the fee will be used to facilitate work attached to the mission and objectives. I know the board, so it's maybe easier for me to accept that than some that don't know them. I've yet to see a real free lunch though - there's always payment (or an agenda/politics/favour) somewhere. I joined and paid (technically, paid first) so I can be on the journey.</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Positive, not negative...</i></b></blockquote>
<div class="p1">
Do you want to win the lottery - well you have to be in it to win it. Do you want to engage and work for something that can be hugely positive - I'm in - I'm not going to bash anything non-cdt, I'm going to work for good testing - rooted in value advocacy for testing skill.</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Knowledge work -> original problems...</i></b></blockquote>
<div class="p1">
I want to be onboard for the journey - the route is not mapped out -> just like most original problems. </div>
<div class="p2">
<br /></div>
<div class="p1">
The board, founding members and new members (I've seen via twitter so far) comprise a great set of minds - I want to work with those testers in tackling some of the key problems that testers have today -> whether that is highlighting the importance of tester skill, helping communicate the value of testing to higher execs and so relate tester value to business value (as well as vice versa) [BTW, who else is working on that today?] Or why not introducing a bit of common sense along the way…. </div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Reframe...</i></b></blockquote>
<div class="p1">
My take on one outcome <span class="s1"><b>I</b></span> can envisage -> Reframing the testing problem, reframing the business problem and see how testing and testers can help the business - an exec might call that "good common sense".</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Commit...</i></b></blockquote>
<br />
<div class="p1">
In the next year, I want to commit to the cause, be part of the energy that everyone joining brings. The organization needs the energy of the members to help achieve its mission and objectives. I'm in. </div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Crystal ball time...</i></b></blockquote>
<div class="p1">
If this grouping hasn't achieved anything in one year's time then it could be called a failure, but right now, I'm confident with the people on board - and there's room for other intelligent testers that are maybe not part of it, yet - it won't be.</div>
<blockquote class="tr_bq" style="text-align: center;">
<b><i>Already joined?</i></b></blockquote>
<div class="p1">
If you're part of the journey too, I'd be interested in your take.</div>
<div class="p1">
<br /></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0tag:blogger.com,1999:blog-5948141587422980338.post-74505054242814343292013-05-28T20:37:00.000+02:002013-05-28T20:37:43.618+02:00Examining the Context-Driven PrinciplesAt Let's Test 2013 James Bach had a keynote about the Context Driven Testing principles, some of their origin, usage and implications of them. An interesting piece of trivia that James revealed was that the basis of the principles were written down very quickly, with some refinement afterwards.<br />
<br />
He discussed a little about the implicit meanings behind the 7 principles, but the slides were skipped through so I didn't see the details. However, by this point at least one question occurred to me.<br />
<blockquote class="tr_bq">
My question in the open season went along the lines, <i>"The principles were mentioned as a litmus test to gauge if someone was context driven. However, adopting a scientific approach (ie one of searching for information and not settling on an assumption), there may well be more than 7. How do we add number 8?"</i></blockquote>
An underlying question was (for me):<br />
<br />
<ol>
<li>If the principles were written so (comparatively) quickly why haven't they been challenged from testers claiming to follow or adhere to them?</li>
<li>Indeed, isn't this a good exercise for a tester - context-driven or otherwise?</li>
</ol>
<br />
Therefore, I thought I would take a look at them - as a (comparatively) quick exercise. James did mention that Markus Gärtner had extended the principles as an exercise previously. I remember seeing a discussion on the software-testing yahoo list some time ago and checked it out - and sure enough I'd discussed the possible extensions there, and thought it was a useful topic to revisit.<br />
<br />
<b>The 7 published principles, ref [1], </b><br />
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="white-space: pre;"> </span>1.<span class="Apple-tab-span" style="white-space: pre;"> </span>The value of any practice depends on its context.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>2.<span class="Apple-tab-span" style="white-space: pre;"> </span>There are good practices in context, but there are no best practices.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>3.<span class="Apple-tab-span" style="white-space: pre;"> </span>People, working together, are the most important part of any project’s context.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>4.<span class="Apple-tab-span" style="white-space: pre;"> </span>Projects unfold over time in ways that are often not predictable.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>5.<span class="Apple-tab-span" style="white-space: pre;"> </span>The product is a solution. If the problem isn’t solved, the product doesn’t work.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>6.<span class="Apple-tab-span" style="white-space: pre;"> </span>Good software testing is a challenging intellectual process.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>7.<span class="Apple-tab-span" style="white-space: pre;"> </span>Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.</blockquote>
<b>My Approach</b><br />
In assessing the principles I used two approaches, (1) Critical/Logical consistency - do the propositions support conclusions or implications?; (2) Are the principles testable (falsifiable)?<br />
<br />
<b>Assessment</b><br />
#1 [<i>Unchanged</i>] I tried to extend this one, but I think it is rhetorically good - succinct and a good start to the principles. This can not be tested but is inferred by induction.<br />
<br />
#2 [<i>Remove</i>] I could think of removing/replacing this one. It is a derivation of #1. The essential content here is that "Practices evolve over time" - I thought of making this my #2, partly to remove the distraction of "best practices" (that would be a derivation of these principles), and partly to emphasize the evolution of practices. However, I finally settled on having no #2.<br />
<br />
#3 [<i>Rephrase</i>] The principle is an assertion. For this one I want to emphasize that the project, the processes and people form a system and that they change over time. The importance here is to illustrate that the system is not static, it is complex because of people (and their emotions) and as such, working with such a system is not a trivial activity. Therefore I would re-phrase.<br />
<br />
#4 [<i>Remove</i>] This is a derivation / implication of #3, an application of the principles.<br />
<br />
#5 [<i>Unchanged</i>] Again, succinct and difficult to refute.<br />
<br />
#6 [<i>Unchanged</i>] An assertion that where good software testing happens there is a high intellectual demand.<br />
<br />
#7 [<i>Remove</i>] This is unnecessarily wordy and sticks out as too different from the previous principles. It also drifts into certainty and seems like an untestable principle, and so I would remove it.<br />
<br />
<b>Addition</b><br />
#8 Here I want to tie the system mentioned in the rephrased #3 and good software testing.<br />
<br />
<b>Result</b><br />
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="white-space: pre;"> </span>1.<span class="Apple-tab-span" style="white-space: pre;"> </span>The value of any practice depends on its context.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>2.<span class="Apple-tab-span" style="white-space: pre;"> </span>-<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>3.<span class="Apple-tab-span" style="white-space: pre;"> </span>Projects and people form part of the system that work together, where products and practices evolve over time<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>4.<span class="Apple-tab-span" style="white-space: pre;"> </span>-<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>5.<span class="Apple-tab-span" style="white-space: pre;"> </span>The product is a solution. If the problem isn’t solved, the product doesn’t work.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>6.<span class="Apple-tab-span" style="white-space: pre;"> </span>Good software testing is a challenging intellectual process.<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>7.<span class="Apple-tab-span" style="white-space: pre;"> </span>-<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>8.<span class="Apple-tab-span" style="white-space: pre;"> </span>Good software testing encapsulates the system of project, people and practices that work on building the product.</blockquote>
<div>
So, I ended up with 5 principles. The application of these would then produce variants and clarifications. For instance, the best practice item is derived from #1. Understanding systems of people, emotions and time is derived from these and many, many more.<br />
<br />
I stopped there, but if I spent more time could probably refine them somewhat. But they are good enough for this exercise. And as James suggested - if you started from scratch, you might well not end up with the original 7 principles.<br />
<br />
So, how would your interpretation look?<br />
<br />
<br />
<b>Reference</b><br />
[1] <a href="http://context-driven-testing.com/">http://context-driven-testing.com/</a></div>
Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com9tag:blogger.com,1999:blog-5948141587422980338.post-38515387757512891842013-05-10T17:45:00.000+02:002013-05-10T18:02:15.434+02:00Peer Conferences -> Conferring -> LearningThis week I held my first internal (in-house/in-shop) peer conference. This is a new concept to many at my shop and I'd been promoting it via short promo talks, e-mail and some gentle prodding.<br />
<br />
<b>Concept</b><br />
For those that have never participated in a peer conference the concept is a cross between a roundtable discussion, an in-depth interrogation and an experience report. After a short presentation, statement or question, then an open session of moderated discussion explores the topic. In my experience, the open session is the real examination and distillation of learning ideas and experiences and it's there that many learnings and insights come forward.<br />
<br />
The format is run in different formats including non-time boxed, ref [5], and time-boxed variants, refs [6] & [7].<br />
<div>
<br /></div>
<b>Concept in academia</b><br />
I recently discovered some academic support for this type of learning, ref [1], which looks at learning and communication by posing questions and answering them. This can involve the talk-aloud protocol (which is a description and discussion about the experience) - this is one way to help the group get to the real cognitive processes behind the experience.<br />
<br />
If the questioning is done in real-time (whilst someone is making a decision or performing a task) then this may be termed the think-aloud protocol. In an open season sometimes the questions turn to "how did you chose that", or, "why did you make that decision", then there is a chance that we relive the think-aloud concept. These are powerful because they can help to show up unintended misunderstandings - either in the reporter or the audience - and so enhance learning.<br />
<br />
The talk- and think-aloud protocols, ref [3], help all to discover the implicit ideas behind the experience and for all to build a mental model of the key turning points in the experience. This is important for use and understanding.<br />
<br />
This is a way of making hidden knowledge explicit, and Ericsson and Simon, ref [1], refer to the work of Lazarsfeld, ref [2]. This talks about the differences (1) in response to the way a question is phrased and also (2) about tacit assumptions.<br />
<br />
An example of (1) is given as, "why did you buy this book" - where the response will depend on where the emphasis is - on "buy" the response might contrast to borrowing the book, on "this" the answer might be about the author, on "book" it might be about opportunity cost, contrasting with a restaurant or cinema visit.<br />
<br />
Lazarsfeld's example of Tacit Assumption is from Chesterton's "The Invisible Man", ref [4]:<br />
<blockquote class="tr_bq">
<i>"Have you ever noticed this--that people never answer what you say? They answer what you mean--or what they think you mean. Suppose one lady says to another in a country house, 'Is anybody staying with you?' the lady doesn't answer 'Yes; the butler, the three footmen, the parlourmaid, and so on,' though the parlourmaid may be in the room, or the butler behind her chair. She says 'There is nobody staying with us,' meaning nobody of the sort you mean. But suppose a doctor inquiring into an epidemic asks, 'Who is staying in the house?' then the lady will remember the butler, the parlourmaid, and the rest. All language is used like that; you never get a question answered literally, even when you get it answered truly."</i></blockquote>
Note, Lazarsfeld also remarks that people consider quality in products differently!<br />
<br />
<b>So what?</b><br />
The lesson from these sources is that many and varied questions get to the root of the experience and reasoning! Responses will be interpreted differently by different members of the audience. Therefore, it becomes part of their interest to ask and clarify from their perspective.<br />
<br />
When you look at it this way the experience report and discussion becomes so much richer and fuller than a written report could hope to achieve. The learning becomes tangible and personal.<br />
<br />
A key experience of peer conferences, for me, is that skills (or key decisions) are able to be distilled from the experience report. This means that those skills (or decisions), once isolated, can be investigated and used in practice - once something is recognized it becomes observable and possible to use in future experiments (practice).<br />
<br />
<b>So how did it go?</b><br />
We had a small group (7 of us) which was very good as the concept was new to all, excluding myself and ran according to the LEWT format. The theme was testing skills and we dedicated an afternoon to it.<br />
<br />
In the discussion part of my experience report (an experience about usage of combinatorial analysis for initial test idea reduction) we were able to diverge and find "new" uses for combinatorial analysis. We discussed a usage to analyze legacy test suites - a white spot analysis - now although this was something we already do, we hadn't recognized it as that. That means we can do it (potentially) differently by tweaking input parameters to produce different levels of analysis.<br />
<br />
I was very pleased to be able to demonstrate this learning / insight "live", as it reinforced the power of the peer conference concept -> ie we discovered it "live".<br />
<br />
We finished the afternoon happy and energized with the format and committing to be ambassadors for others. Job done!<br />
<br />
<b>Next steps</b><br />
I started this internal conference as a monthly series of short talks and some deeper discussion. In June there will be another occurrence and I'll also try a variant joining a session in two countries with HD video comms - basically I'm spreading it around the world!<br />
<br />
<b>Upcoming</b><br />
In just over a week there will be a peer conference prior to Let's Test, followed by the eagerly-anticipated <a href="http://lets-test.com/">Let's Test</a> conference. Those will be experience and learning-rich environments. Pretty cool!<br />
<br />
<b>Reference</b><br />
[1] Protocol Analysis: Verbal Reports as Data (1993; Ericsson, K.A., and H.A. Simon. ; Revised edition. Cambridge, MA: MIT Press.)<br />
[2] The Art of Asking Why in Marketing Research (1935; Lazarsfeld) <a href="http://www.jstor.org/discover/10.2307/4291274">http://www.jstor.org/discover/10.2307/4291274</a><br />
[3] Wikipedia: The Think Aloud Protocol <a href="http://en.wikipedia.org/wiki/Think_aloud_protocol">http://en.wikipedia.org/wiki/Think_aloud_protocol</a><br />
[4] G. K. Chesterton, The Invisible Man: <a href="http://www.readbookonline.net/readOnLine/15496/">http://www.readbookonline.net/readOnLine/15496/</a><br />
<br />
<b>Peer Conference References</b><br />
<b><br /></b>
[5] <b>SWET: Swedish Workshop on Exploratory Testing</b><br />
<a href="http://thetesteye.com/blog/2010/10/swet1-fragments/">http://thetesteye.com/blog/2010/10/swet1-fragments/</a><br />
<a href="http://testers-headache.blogspot.se/2011/04/swet2-serious-testing-talk-by-serious.html">http://testers-headache.blogspot.se/2011/04/swet2-serious-testing-talk-by-serious.html</a><br />
<a href="http://thetesteye.com/blog/2011/11/notes-from-swet3/">http://thetesteye.com/blog/2011/11/notes-from-swet3/</a><br />
<a href="http://blog.johanjonasson.com/?p=547">http://blog.johanjonasson.com/?p=547</a><br />
<a href="http://testers-headache.blogspot.com/2013/04/some-notes-from-swet5-small-and-intense.html">http://testers-headache.blogspot.com/2013/04/some-notes-from-swet5-small-and-intense.html</a><br />
<a href="http://www.satisfice.com/blog/archives/527">http://www.satisfice.com/blog/archives/527</a><br />
<br />
[6] <b>DEWT: Dutch Exploratory Workshop on Testing</b><br />
<a href="http://dewt.wordpress.com/">http://dewt.wordpress.com/</a><br />
<br />
[7] <b>LEWT: London Exploratory Workshop on Testing</b><br />
<a href="http://www.workroom-productions.com/LEWT.html">http://www.workroom-productions.com/LEWT.html</a><br />
<br />Simon Morleyhttp://www.blogger.com/profile/10629592766073538811noreply@blogger.com0