When To Use Test Automation (part 2)

This post looks at some of the cost considerations for using test automation within a project or organisation.

Costs of Test Automation

There are costs associated with any test automation effort - some are obvious and some are not, depending on ones perspective/experience with automation. Let's start with some of the obvious (in your face costs):

Manpower/resources

It will become increasingly apparent that automation is not a part-time/on-the-side activity. It's true that all SW testers need to be able to handle/script to a greater or lesser extent as the amount of information to process is sometimes huge (for some testers this might just be some manipulation of data in a proprietary tool or an office tool like excel.)

Finding the balance for testers is important. It might be that the test framework is supported by full-time staff and other testers just make use of the API's in their own specific test cases.

Framework?

The idea of whether to use a framework in an automated test environment is very much related to the flavour of the project and the use to which the test automation is used/re-used. This question is something that needs analysis before the test automation effort is committed.

It may be that the framework is so light-weight to be just some basic templates of how the test scripts should work, fitting into a standard execution environment. In this case the framework just needs a one-off start-up cost with limited maintenance afterwards.

On the other side of the scales the framework could be something quite comprehensive enabling test cases/scripts to be written on varying degrees of abstraction. In this case the maintenance and support of a framework is a dedicated/full-time activity.

Depending on where the test effort falls within these scales dictates the amount of effort required up-front on framework development, as well as supporting processes and strategy.

Processes, Strategy & Documentation

This is needed in all automation efforts. The degree to which it requires up-front analysis and investment varies. Ideally, all processes, strategy and documentation updates should be in place before teams are asked to use them. The reality can be that all of these develop and are updated in parallel with team development.

In both cases there is a bigger cost on the first teams using the new/adapted automation strategy. This is the cost associated with an "early adopter".

Training

Either the automation tools/framework are bought-in, developed in-house or some hybrid. Whichever way it falls there will be a training cost associated with it. With any external tool/framework that is brought in there is a potential saving (re-using testers that have experience with those tools) - but there is still a cost as the test cases/scripts have to be "flavoured" for the product/system under test.

Test Organisation vs. Project

When estimating costs of an automation effort the long term needs to be considered. An automation effort (whether it is framework development or new test case development) will have a certain long term cost / maintenance need.

A less obvious factor to bear in mind in relation to test frameworks is that they usually need to be long-lived - it is no use having to re-write the test framework every project or every year, so the long term needs of the framework need analysis - partially covered in the business case analysis.

Business Case for Test Automation

Many talk about the return on investment (ROI) when considering automation business cases. This can be notoriously difficult to estimate in a standardised way. Below are some points to consider when looking at the case for automation - analysis of these will give different answers on the ROI question.

Uses

Consider the uses of automation (see previous post on benefits for some) and which problems (in the test strategy) that automation might try and solve.

Expectations

Automation can get a bad press in terms of its perception to solve many testing needs - this can be a combination of misunderstanding or over expectation. So in any analysis it's important to be realistic and actual for the current test environment - and manage any expectations where necessary.

Underlying Implementation

Consider the need for a framework - do this early - as it could rule out large scale automation early on. It could be that for what's needed it isn't economical to automate on a large scale. Reasons for this could be immaturity on the product/system under test (especially true in early development stages) or limitations in available tools to integrate into a new automation regime.

Costs: Strategy & Effort

Consider and estimate the costs associated with any automation effort - as broad as possible. This should cover documentation, process updates, training, and allocation of dedicated support persons as well as strategy changes. The strategy aspects should take into account what happens with the framework after the coming project and how (or if) the base framework and test suites should be future proof - what's the cost both way (to future-proof or not?)

Summary

• Test automation should be considered an activity in its own right (even if automation is done by non-automation test engineers)
• When looking at aspects to consider with test automation it becomes clear that the use of automation within a project or maintenance unit should be considered as part of the Software Development process in its own right - and not an offshoot of the testing effort.
• Understand the costs of automation.
• The first three points will help determine a better business case for test automation.