I had a brief and interesting twitter exchange with Michael Bolton last week:
My Frame
First I'll give the background of where I'm coming from when I think of "scripted testing".
I started thinking about the checking vs testing distinction a while back (here, here and here). If you read them (rough and ready as they were) you'll see that my ideas about checks are wrapped into testing - i.e. you can't use checks without testing - testing being the information gathering exercise where you use the information gathered to help determine which other type of information to gather - that (to me) is one of the aspects of "good testing".
Note on Earlier Posts on Testing vs Checking
Those posts were a little rough - peer review would definitely tighten them up.It's actually interesting looking back on these posts (for me) as I can see areas where I've developed and refined some of my thinking. But in essence the check is an artifact and the test is the activity
So it was in that frame that I then triggered the question to Michael above.
Testing?
Am I turning scripts into tests? Maybe.As Michael pointed out (which was quite a helpful clarification!) - I'm using them as an input - and as I referred to in the posts - I can't use a check without thinking about it's use beforehand and it's results afterwards - so in that sense it's very much a part of my testing.
Who Cares?
Some people - but definitely not everybody!
I've noticed a tendency to dismiss scripts or to detach them from testing. Well, as Michael also helpfully pointed out - there are aspects of scripting in most testing (whether it's a mission, goal or hunch) and that most good testing can't be constrained by a script.
And Finally?
For me:
- Scripted Testing = Having a script wrapped up in testing, an input into the information gathering exercise.
- Scripted Execution = The script on it's own. The activity is constrained by the script. Mmm, what can you do with that on it's own? Not much.
Have you noticed when you've been constrained by the script? Did you let the script constrain you?
I remember lurking and reading that Twitter conversation. Thanks for posting this. Interesting things to consider.
ReplyDeleteI use various models as material for test execution. I used to create models than extract test cases - sometimes even scripted procedures. Nowadays I have a test idea, use the model as background and do a test session based on that. When I write test cases I find they constrain me and also it steals time from real testing. So test design and test models are valuable activities. For me lately test cases or scripts is not.
ReplyDeleteBy the way - what is the purpose of writing a script if you are not planing on letting it constrain you? I mean either you write a script and follow it or you don´t write the script. Why write a script and then just use it as a guideline? Then there are better ways of working. See you on the bus 17.47. --Tobbe
Great article Simon - I like the differences you are making between scripting and checking. It causes some confusion.
ReplyDeleteI use scripts to aid my testing and to help guide me. Once I have finished testing I have a set of steps which then become a check. If I follow the steps again and again afterwards I will not learn anything new about the system but it can tell me if something has been changed.
Your blog post has given clarification to a hot topic - keep up the great work.
Tobbe,
ReplyDeleteI think we're thinking past each other here. I'm considering the case where a script is used (maybe in automation or a configuration capacity) - so the script is inheritted (see the twitter conversation above).
In this scenario, I may decide to let the configuration script do its job (I may test for this at the end) or if I no longer want to trust it I might use it as an input to my test.
Even if I make the decision to not test the script (ie proceed after it has configured the system) then that choice is a testing decision (for me) - I think about how the scripts are used (and for what purpose) before they're used and then I may evaluate the information they throw up - all of this is informing my decision about what to do next.
My point (and the point in the previous posts referenced) is that I can only think of using scripts as input to my testing - I see no point in them without any evaluation. I think we probably agree on that.
Thanks for the comment - I agree with your points about test design btw.