There has recently been a debate over metrics when applied to software testing. Stories of usage and examples have run the whole range from the good, the bad, the ugly to the incomprehensible.
Linda Wilkinson has bravely decided to start a series on metrics and her viewpoints, here. I anticipate a good discussion - if there is any negative feedback I'm sure it'll be given in a professional way - rather than just being unfriendly feedback.
My latest metric
The latest metric that I have used is the pedometer - or step counter. It shows me how many steps I have taken and total elapsed time with a conversion to distance and calories used.
But that's just raw data - I need a certain history and environment in which to interpret the data.
It doesn't tell me about the terain I was walking in.Was it hilly - so some steps counting double for the exertion?Was I walking fast/slow?Was I walking fast and taking lots of breaks or did I do the whole thing in one go?Was I carrying any baggage?Was I pushing or pulling something?How comfortable were my shoes?
Test Metrics
If using them, then understand the data - what it's representing, what it's an instance of and question what it's not telling you.
Know and work with examples of what might be deducable from the data.
Know and work with examples of what cannot be deduced from the data.
Know how to deal with data that's missing - sometimes nothing more than to acknowledge it - but that's an important step in itself.
Extensions?
Some would say that the answer to my pedometer problem is to get a better pedometer or ultimately some sort of human-tachometer. However, I just want a simple comparison - something that gives me some background data. It's just data until I can set it in a context with some meaning and I'm happy to do that step.
Even a super-duper-human-tachometer couldn't tell me about my motivation on a given day. The final story always needs the narrative!
Have you thought about the problems and limitations of your metrics?
Pedometers tell you how many steps you have taken. If that's all you are trying to measure, then a pedometer is a perfectly appropriate tool.
ReplyDeleteBut you haven't indicated what you are actually trying to represent with this measure.
- Are you looking for a metric that represents your progress towards good health for the day?
- Are you looking for a metric that represents the energy expenditure for the day?
- Are you looking for a metric that represents your weight loss for the day?
Any of the above will be less accurately represented by the numbers on your pedometer.
When deciding on a test metric tool, decide what you want to measure first. Then decide what you want that measure to represent. Last, consider if that simple measure is capable of adequately representing the attribute that really interests you.
Too often, we use an easy-to-gather measure (such as number of steps) to represent a difficult-to-quantify attribute (such as overall health). That's often where we go astray.
Hi Joe,
ReplyDeleteThanks for the comments. You've almost pre-empted a follow-up post where I look an example in a bit more detail.
Yes, absolutely true about what you want to measure first. This is a given isn't it?
No.
Too many times I've seen measures collected - maybe because they've always been collected and might not be relevant anymore - without a plan about want "is" being measured and why.
Statistics can be dangerous. Just like a "little" knowledge.
Statistics on their own are equally dangerous. I usually present the interpretation first - if anyone wants to delve into the background then we can dig into the data together - but it's never data on it's own - that leads to people making their own conclusions...
Metrics use has is almost analagous to a testing activity:
Identify requirements (what to measure) -> design (how) -> measure (execution) -> analyse (making meaning & assessing significance) -> present/discuss (reporting & feedback).
Regards,
/S