Everyone knows what this is right? The concept has been around a while* - everyone has been there and done that, if you read the hype. But who is innovating?
A lot has been written about CI and its place in support of testing... Or has it?
- Massive parallel script execution against the same target drives a re-think on test framework design, modelling and creation - impacting data modelling and needs for flexibility in frameworks and harnesses
- This is a move away from single isolated and independent "tests" on a stateful application. It will trigger a change in test script approaches. Where is the work on that? Pointers gratefully received...
- I have seen some research on "multi-session testing of stateful services", but more is needed.
- I see that as a rule-driven approach to setting a series of checks on commits, e.g. (1) which checks cover my updated code-base (execute and result), (2) which whitespots in my codebase are there now (report)
- Where are the studies or experience reports, where is the work?
- CI is a development approach, and is distinct from testing. It's like an extension to compiler checks**. Thinking around selecting and adding to those "compiler checks" needs to be dynamic. Experience reports, empirical studies for this?
- There is a danger that "testing" is driven into a TDD/Acceptance Test-only mode.
- I would like to see more research on organisational and soci-technical challenges around software development...
- Detachment from Software Development
- This can be seen in various forms
- Distillation down to process on "testing" only - the ISO 29119 papers are a classic example of this. This is the "reductionist" approach to a wicked organisational problem - not very sophisticated and with a high risk of solving the wrong problem.
- Other examples are some/many software testing only books - yes, it can be good to highlight the testing and testers role and special challenges there, but until you start from software development as a whole (systems thinking approach) then there is a high risk that you are making a local optimisation. Another reductionist approach, liable to solving the wrong problem.
- So, software testing focus -> good; software testing interplay and interaction and CONNECTION to the creative process -> better.
- Many organisations focus on confirmatory tests - a proxy for customer Acceptance Tests - as an MVP (minimum viable process), i.e. a proxy "get out of gaol card". See Myers  example of testing in an XP approach is an example here.
- Myers  first wrote about the psychology of software testing. However, Martin et al  make the case for reframing this as an organisational approach/problem. Rooksby et al  observe the cooperative nature of software testing.
- More studies on satisficing the organisational needs please!!
- Sommerville et al :
- "An over-simplification that has hindered software engineering for many years is that it is possible to consider technical software-intensive systems that are intended to support work in an organization in isolation. The so-called ‘system requirements’ are the interface between the technical system and the wider socio-technical system yet we know that requirements are inevitably incomplete, incorrect and out of date."
Got any pointers to recent advances? I'm all ears...
 Li, Chou, 2009, IEEE; A Combinatorial Approach to Multi-session Testing of Stateful Web Services
 Myers, 2011, 3rd ed; The Art of Software Testing
 Rethinking Experiments in a Socio-Technical Perspective: The Case of Software Engineering
 Martin, Rooksby, Rouncefield, Sommerville, 2007, IEEE; ‘Good’ Organisational Reasons for ‘Bad’ Software Testing: An Ethnographic Study of Testing in a Small Software Company
 Rooksby, Rouncefield, Sommerville, , Journal of CSCW; Testing in the Wild: The Social and Organisational Dimensions of Real World Practice
 Sommerville, Cliff, Calinescu, Keen, Kelly, Kwiatkowska, McDermid, Paige, 2011, Communications of the ACM; Large Scale Complex Systems Engineering
*I led the architecture work on a multi-stage CI system in ¨2010
**yes, a big simplification.