Some Software Development & Testing Challenges 2016

So it's 2016 and I have been reflecting on some of the challenges I see for Software Development, with emphasis on Software Testing.

Continuous Integration

Everyone knows what this is right? The concept has been around a while* - everyone has been there and done that, if you read the hype. But who is innovating?

A lot has been written about CI and its place in support of testing... Or has it?

Some Challenges

1. Massive parallel script execution against the same target drives a re-think on test framework design, modelling and creation - impacting data modelling and needs for flexibility in frameworks and harnesses
• This is a move away from single isolated and independent "tests" on a stateful application. It will trigger a change in test script approaches. Where is the work on that? Pointers gratefully received...
• I have seen some research on "multi-session testing of stateful services", but more is needed.
2. CI script execution and studies showing the effectiveness of dynamic test script selection strategies for team or testing support
• I see that as a rule-driven approach to setting a series of checks on commits, e.g. (1) which checks cover my updated code-base (execute and result), (2) which whitespots in my codebase are there now (report)
• Where are the studies or experience reports, where is the work?
3. There are socio-technical challenges with CI use and implementation. Technology is the easy part, the soci-technical part comes in when organisational issues and preferences distort the technology choices. This might range from "we have always done it this way" to "the language or framework of choice here is X so everyone must adapt to that".
• CI is a development approach, and is distinct from testing. It's like an extension to compiler checks**. Thinking around selecting and adding to those "compiler checks" needs to be dynamic. Experience reports, empirical studies for this?
• There is a danger that "testing" is driven into a TDD/Acceptance Test-only mode.
• I would like to see more research on organisational and soci-technical challenges around software development...
4. Are people really going all-in on cloud and virtualization technologies to solve their CI related bottlenecks? Mmmm...

Software Testing

Some Challenges

1. Detachment from Software Development
• This can be seen in various forms
1. Distillation down to process on "testing" only - the ISO 29119 papers are a classic example of this. This is the "reductionist" approach to a wicked organisational problem - not very sophisticated and with a high risk of solving the wrong problem.
2. Other examples are some/many software testing only books - yes, it can be good to highlight the testing and testers role and special challenges there, but until you start from software development as a whole (systems thinking approach) then there is a high risk that you are making a local optimisation. Another reductionist approach, liable to solving the wrong problem.
• So, software testing focus -> good; software testing interplay and interaction and CONNECTION to the creative process -> better.
2. Mis-understanding of the software testing challenge - how to link a creative process (software creation and positive and confirmatory tests and checks) to a challenging process (testing for issues, highlighting problems, testing for risks)
• Many organisations focus on confirmatory tests - a proxy for customer Acceptance Tests - as an MVP (minimum viable process), i.e. a proxy "get out of gaol card". See Myers [2] example of testing in an XP approach is an example here.
• Myers [2] first wrote about the psychology of software testing. However, Martin et al [4] make the case for reframing this as an organisational approach/problem. Rooksby et al [5] observe the cooperative nature of software testing.
• More studies on satisficing the organisational needs please!!
3. Lack of soci-technical studies and research into software testing and its part in software development. Rooksby & Martin et al [4] & [5] performed ethnographic studies of software testing to highlight its cooperative and satisficing nature. This called for further research
• Sommerville et al [6]:
• "An over-simplification that has hindered software engineering for many years is that it is possible to consider technical software-intensive systems that are intended to support work in an organization in isolation. The so-called ‘system requirements’ are the interface between the technical system and the wider socio-technical system yet we know that requirements are inevitably incomplete, incorrect and out of date."

The sooner we stop treating software development, and especially testing, in reductionist approaches, consider the socio-technical aspects - especially for large and complex systems - the better. And, today, most systems are inevitably complex.

Got any pointers to recent advances? I'm all ears...

References

[1] Li, Chou, 2009, IEEE; A Combinatorial Approach to Multi-session Testing of Stateful Web Services

[2] Myers, 2011, 3rd ed; The Art of Software Testing

[3] Rethinking Experiments in a Socio-Technical Perspective: The Case of Software Engineering

[4] Martin, Rooksby, Rouncefield, Sommerville, 2007, IEEE; ‘Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company

[5] Rooksby, Rouncefield, Sommerville, , Journal of CSCW; Testing in the Wild: The Social and Organisational Dimensions of Real World Practice

[6] Sommerville, Cliff, Calinescu, Keen, Kelly, Kwiatkowska, McDermid, Paige, 2011, Communications of the ACM; Large Scale Complex Systems Engineering

*I led the architecture work on a multi-stage CI system in ¨2010

**yes, a big simplification.

Blogs, Darwin and the Testing Eco-System

#softwaretesting

A bit of divergent thinking...

I started about by thinking about testing blogs recently, progressed through evolution and ended up thinking about the testing eco-system (or natural habitat) - or really the testing evolutionary tree.

Blogging

I wondered what testers get out of blogging and in a "rare" moment where my glass was only half full (as opposed to half-empty) I could only come up with the positives. Here.

I like and agree with both James and Matthew's comments. Blogging is definitely one way of getting a body of knowledge out there (or views and opinions) - true sometimes they are or have to be bland, as James says.

But it was Matthew's reference to a sponge that triggered my Darwinian thinking...

Evolutionary trail...

There is room for sponges in the testing eco-system - some people have to start somewhere. Jumping on a Darwinian-theme: it's quite ok for someone to absorb information and get a few ideas under their belt.

James wrote an interesting post where he was saying that those methods/techniques will always be someone elses methods/techniques - it won't be the testers (that has sponged/absorbed them) until they develop them further and make them their own.

Hence Darwin - this is the evolution of the sponge into something more advanced (not sure what the next step up the evolutionary scale is - biology was never a hot topic for me). But the important thing is that they do "evolve" and don't lie stagnant...

Following the evolutionary analogy, I guess the spammers are lower down on the scale, maybe the hackers are more advanced. The sniping commentators are probably on another evolutionary side-branch and evolution will ultimately take care of them.

Maybe the evolution analogy is over-doing it -> people learn, develop and progress (hopefully.)

Phylogenetic Tree

However, all this thinking about evolution made me wonder about an evolutionary tree for software testing.

I really like to think of the testing eco-system/habitat as something comparable to an evolutionary tree - there are many branches, many ideas that have developed, transformed and evolved.

This is something to chart the different strands for test techniques and test approaches - one tree for each view. This is something I'll continue to work on.

Suggestions for the test evolutionary tree?