There is a view of exploratory testing that believes it is spontaneous (as in combustion) - one just turns up at a keyboard (or other lab/test equipment) and you "just do it"!
Nice, if you can get it.
But what if your system under test is not trivial - in terms of data set-up, configuration, simulated subsystems, user profiles, behaviour or network elements? Does this mean that you can't have an exploratory approach?
Well, of course you can.
Think of it as going on a walk/hike/trek in an area you haven't covered before. To actually get to the "start" of the hike you might use a series of public or scheduled transport (a pre-determined schedule) to get to the start point.
I emphasize the uncertainty around the "start" as you might move this depending on how you defined where the trip should start.
You might decide to start your walk when stepping off of the bus - the bus is a pre-determined script to get you to a point where you will start - but perhaps you notice something odd on the journey; maybe the bus takes a route that isn't signposted as your destination - is it a pre-determined route or a diversion, will the journey take longer ... The questions start.
Using a script to get to your starting point doesn't mean that you can't ask questions - remember, this is testing with your eyes wide open as opposed to eyes wide shut. You finally get to your start point - where the trek can begin.
Just like an exploratory approach - there might be a whole series of set-up and configuration to get the system into a state ready to start working with your test ideas. It might be that you need to construct a simulation of a network element to start testing in the desired area. All of this may take some time to achieve - a whole series of hoops and hurdles.
In some of the systems that I work with this is exactly what we do - it's about a certain design (simulation, config and test ideas) - getting to the starting point where we can start working with our test ideas.
Sometimes the test ideas yield more investigative tracks than others - but that is exactly where we "intend" the exploration/testing to start. But like any (good) tester we notice things on the way, we raise questions about our simulation assumptions, about the problem/test space in our target area - and we learn about the product (along the way and in our target areas) - we learn about our test ideas, we learn about our test ideas, if we're missing something vital in our simulation data, configuration or how the whole might relate to a real end-user.
That was our walk in the woods, and the scripted element is very much a part of the trek/exploration. The scripted part is setting the conditions for the exploration - an enabler - whether I am questioning the set-up as we go is a matter of preference, time, priorities and feedback from previous testing.
Do you observe or get curious on the way to your walk in the woods?
Hi Simon,
ReplyDeleteExtending the analogy somewhat, once you get to your starting point you have a 'finish' in view but as you walk you notice different things, obstructions get in the way, new questions arise. These all shape the route you will take to get to the finish. On subsequent walks you might vary your route slightly between the two places.
A good tester will follow as many route variations as is practical and/or possible each time s/he executes a test plan.
Hi Stephen,
ReplyDeleteYes, varying the route is a good extension of the analogy.
But, I wouldn't rule out following the same route and making new or different observations along the way - I think the analogy just holds together :)
Or even repeat observations - I can think of the hubble telescope - it was pointing at one piece of the night sky for 3 months before the astronomers noticed a new system.
Thanks for the comments.
Hi Simon,
ReplyDeleteFollowing up on your last question:
When I am on my way to the walk in the woods I am very curious as to discover new paths I wanna visit later. I would sit and stare out through the window of the bus, observing and analyzing the landscape for good hiking possibilities. (One of my hobbies is hiking in the mountains so I often look for promising hikes with a blend of steep hills and nice outlooks...)
Of course, if I travel the same trip over and over again just to take me to the recreation area, I will eventually go from curious to blase. Then I would probably be sitting dreamily and begin to think about other bus trips I wanna take instead of this "boring" one.
But up until that, I will have observed all interesting hiking possibilities that I can think of.
Cheers,
Henrik
Hi Henrik,
ReplyDeleteYes, there's always a risk of becoming blase.
The example I was thinking of was of following the same route and identifying the different trees (for example) along the route, the next time it might be collecting information on the different animals trails or habitats - there might be an aspect to repeat as the seasons change (ie different environmental/external conditions affect your observations).
Doing the walk/trek after a rain fall will highlight contours of the trek in terms of where water gathers, after a hard frost you can see which areas are more "insulated" (the sun doesn't reach during the day). I discovered something similar once in a national park that I frequent - there was a stretch that still had hard frost from the winter in late April (near Stockholm, not the north :) ).
It's an interesting discussion - and shows that there is (nearly) always something new to discover. But of course repeating something has to have a purpose.
I was thinking of RST's 10 reasons to repeat a test scenario/case a while ago, and trying to use the hubble example to create an 11th reason, but I think it's covered.
Hi Simon,
ReplyDeleteI love nature and I love walking. I find I tire of a park or walk when I take it every day.
I find if I vary the parks, and walks then when I return to the park, its still fresh and new to me, showing me its wondrous depth and complexity in a new and inspiring light.
Good thoughts!
ReplyDeleteI didn't think of those things, but I guess that my exploration would have been somewhat similar. I am an explorer at heart.
But now we touch on an interesting subject:
When do you become blasé; and does that mean that you have considered that the information available isn't interesting anymore?
(
Other related questions:
How much do you want to care about things?
When do you stop feeding your brain with information?
When does that information become useless; and how do you decide that it is useless?
How do you prioritize between things to focus on?
)
If we go back to the test situation you described. In theory, there are so many opportunities of things to think about in the setup part. But then you have your real life poking for attention.
The loans have to be paid; your kids are getting the flu; the car need winter tyres and there are no slots available at the workshop until next week; your boss had an ugly tone in the last email he sent you; etc.
It is indeed hard to stay focused; and as soon as something becomes a bit routinely, your consciousness becomes active and your brain starts to wander.
And as you say, repeating something has to have a purpose (at least in order to keep you focused).
Hi Anne-Marie,
ReplyDeleteYes, agree. Variety is the spice of life...
Changing routes and returning after a while always has the /chance/ of giving new insights (if you're aware/alert to them).
This is probably related to the focus/de-focus heuristic.
Hi Henrik,
ReplyDeleteYes, becoming blase, dis-interested or losing focus is a risk we have in any aspect of daily life - not just testing.
From a testing perspective there is the angle of how we relate what we're doing to the objective of the testing activity. Then there is a perspective of how we tie that into a "stopping heuristic" - so that we know we are still adding value (or still being curious).
Your other questions are very relevant. Just as I might have a "bad hair" day, I might have other distractions to my daily work.
In terms of test set-up - where it might be quite long or complex - I want to find forms of "fast and frugal heuristics" (I've recently been reading "Gut Feelings") that can help "check" that I haven't missed anything major. (I'm exploring these heuristics in other uses also.)
I like your additional questions - I think it's something we should explore more!
Thanks Simon for your thoughtful reply!
ReplyDeletePerhaps the answer to those questions is one part of Exploratory Testing 2.0 that Rikard has mentioned? :-)
We in The Test Eye will definitely have a look at what Exploratory Testing 2.0 might include. You are most welcome in that discussion!
Another question regarding your analogy:
You elaborated around how the different environmental/external conditions affect your observations during the hike/bus trip.
What kind of external effects do you see that is affecting your observations for your scripted part (the setup)?
Hi Henrik,
ReplyDeleteYes, I'll gladly get involved in discussions with Exploratory Testing 2.0.
On the external/environmental conditions aspect - maybe an AFS server starts running slow (not intrinsically part of your test env but part of an associated network) - then you could get elements of the script that are affected - maybe response times run slower, maybe some commands time out (and when repeated they are ok).
Or maybe, a machine in use has a lot of cached data - how do you get that into the same condition as last time?
I'm not saying suspect everything - but sometimes there are aspects that are sitting "under the hood" - and even if we don't check-off all of those pieces, we don't forget about them either.
I was encouraged by your first paragraph:
ReplyDelete"There is a view of exploratory testing that believes it is spontaneous (as in combustion) - one just turns up at a keyboard (or other lab/test equipment) and you "just do it"!
After that, a little disappointed.
As I see it, testing is about learning. We experiment with the program to learn new quality-related information.
Some people emphasize exploratory execution (sit at the keyboard and pull tests out of their psyche) far beyond what I think is wise. I think a significant portion of exploratory testing involves investigating the intent, risks, market and market expectations, environment, and communications responsibilities of the software under test. Many of my best ideas for tests come when I away from the keyboard (or if I am at the keyboard, when I am at an information site or working with a competing product or a collaborating product).
Starting from a script might take me to a useful place sometimes, but I think we need to more consciously trade off our time investments. How effective is following a script or following it with slight variation as a learning activity, compared to alternatives?
i do like the 'Walk in the woods' idea, especially since I find walks in the woods very enjoyable.
ReplyDeleteAs you pointed out, the scenery always changes with season , weather etc. And even though I may intend to walk exactly the same trail as last time I rarely end up doing that.
I might spot an interesting flower off the trail and go have a look at it (might even bring out my Flora for a comparison...). And over there I might find a narrow deer trail I have never seen before and follow that instead.
At least for me the trick is to take many short walks rather than a few long ones so that I do not get tired and lose interest. Exhaustive testing is not possible, but exhausting testers is.
Hi Cem,
ReplyDeleteThere were two threads in this post - (i) Using some pre-defined set-up as an enabler to an exploratory session, i.e. there can be a certain amount of pre-meditated design and thought before you start exploring, (ii) The pre-defined set-up and exploration should not exclude observation and feedback.
The emphasis here was that this activity is anything but "just sit down and do it", that it can involve a deal of pre-planning, deliberation and set-up and that whilst doing this the importance of keeping ones "eyes wide open" is vital - even if we're re-tracing some steps to get to a state where we want to begin the new session.
So, I think what you can read here is that continuous observation and feedback are important during the preparation and feedback, couple that with evaluation and I think we have what you'd call learning.
I make no claim about the effectiveness or otherwise of using the same script to achieve the start-up state for the target session.
I agree with your point that there is a general need for cogitation - that review and analysis after testing and even allowing the sub-concious to engage - where new ideas can spring forth. I'd started exploring elements of this angle before, here. But I think more discussion around this topic is needed - I'll take that as a challenge, thanks!
Thanks for the comments.
Hi Christin,
ReplyDeleteYes, many short walks is a good - increasing potential for feedback and reflection and observations of changing conditions.
I might also want to think about when to include a long walk too - or recommend it to someone else who would get less exhausted than me :)
Hi Simon,
ReplyDeleteI really like your "walk in the woods" analogy, there are so many possibilities to extend it even more.
Like you shouldn't walk in the woods without anyone knowing where you are or where you are heading.
Or it is really good to bring a friend since you always see the nature in different ways.
I haven't seen this blog before but now you got a new "follower".
BR
Magnus
Hej Magnus,
ReplyDeleteWelcome!
That's a great point - walking with someone else is valuable (just like testing with someone else adds a new/different dimension). Yes, and to help avoid getting hopelessly lost - letting people know about the area you're heading for (target areas for test) is very useful (if you get so engrossed that you're stuck then a helping hand is always welcome - even just to help you de-focus.)
I try to have this blog geared towards discussion & dialogue (rather than me spouting opinions :) ), so you're very welcome to add comments & questions and insights (like you've just done now)!
Thanks!