|
Maximising productivity through automated testing
In the fast changing and highly competitive business environment,
organisations need to accelerate their software development cycles and reduce
their time-to-market to stay ahead of competition. Test automation provides
the answer to an organisation’s business problems by minimising the effort required
for testing systems, while ensuring uniformity in the testing process, says
Sunil Gupta
Using automation, tests can be run faster,
in a consistent manner and with fewer overheads. Automation is the replacement
or supplementation of manual testing with a suite of test programs. Benefits
include increased software quality, reduced time-to-market, reusable test procedures,
and reduced testing costs.
Automated testing has its own disadvantages
and involves a number of challenges. If not planned carefully, it may lead to
a poor quality of testing. The following factors need to be considered before
adopting automated testing and during the automation process.
Number of external interfaces
Complexity of Automated Testing Number
of external interfaces
The complexity of automated testing is
directly related to the number of interfaces in a system. Whenever a software
is put under test, simulators are used at all its external interfaces in order
to test its valid as well as invalid behaviour. Automated testing is achieved
by writing scripts at these simulators. Therefore, if the software has many
external interfaces, complete automated testing would require lot of effort
in selecting and customising tools at each interface and scripting.
The more interfaces a system under test
(SUT) has, the more complex would be the automated test set-up and cost-effectiveness
of automation would be compromised.
Type of external interface
Every external interface of a software
cannot provide room for automated testing. This is true for software having
interface with ‘firmware’ or ‘device drivers’. In such cases, testing needs
to be done with real standard interfaces and not with the simulators. Such a
situation leads to two types of problems. Firstly, all invalid behaviour becomes
very difficult to test and secondly, complete automation is not possible on
these interfaces because it may be possible that in some invalid scenarios the
driver / firmware gets stuck in a state where human intervention (by giving
a command or triggering external action like resetting the physical interface
etc.) is required to come out of it. Also, if software handles ‘interrupts’
coming out of some hardware interface then this also needs to be tested manually.
Number of releases expected for testing
Productivity of automation µ
Number of testing cycle expected
One of the main objectives of automation
is to reduce and cut down similar types of effort again and again. Thus, automation
is directly related to the number of test iterations expected for the product.
For example, if only one or two releases are expected, then going for complete
automation doesn’t make much sense unless it is achieved through minimal effort.
The figure shown below gives the variation
of automation productivity with test iteration for a given level of automation
Series 1 - Productivity with automation
level 1 (bare minimum automation)
Series 2 - Productivity with automation
level 2 (partial automation)
Series 3 - Productivity with automation
level 4 (complete automation)
It can be inferred from above that productivity
of automation increases with test iteration and automation of any level is not
of much use if the number of test iterations expected is less than three
Maturity of the product
A new product can not be tested in a completely
automated manner. Automated testing assumes certain stability in the product,
which may not be present in a new product or system.
Effort in automation
Every automation activity requires certain
effort. This effort needs to be estimated and evaluated for cost-benefits. By
this we mean that one has to see how much effort is being saved through automation
against the effort needed to achieve this. If automation saves two man-days
effort but requires five man-days to achieve this, then this is not a good solution
at all. The factors described above should be considered before going ahead
with a automation strategy for a product.
In summary, going for automation is not
always fruitful. One needs to evaluate the decision against the factors described
above before making the final call on test automation.
Once the decision regarding automation
is made, the next step is the implementation. However many organisations find
the task of implementing test automation time consuming and cumbersome. Frustrations
creep up when automation goals are not clearly defined, high turnover is the
result of the long time lines required to achieve this. Proper planning and
strategy adopted by the top management can soften the hurdles in the transition
process of movement from manual to automated testing. A well thought out process
is the key to success. The process starts with the selection of test tools,
customisation of tools according to test needs, development and verification
of scripts and implementation of a test management system.
Test tools selection
The selection of a test tool requires understanding
the scope of testing. One should know the expectations from the tools and type
of automation targeted. In general, a test tool should essentially support:
- Scripting interface, which is mandatory for automation.
- Facility to give all combinations of valid and
invalid input
- Facility to control IUT (implementation under
test) and its simulated external interfaces, result comparison and verdict
declaration facility
- Logging facility
- Ability to run scripts in batch mode
Needless to say, while selecting a test
tool, reusability, reliability, and cost are the prime criteria to leverage
the maximum benefit out of the tool being procured/developed or customised.
Customisation of the tool
In today’s customer-driven software market
where quality and time play a crucial role, automated testing has become very
important. Many standard tools are available in the market, which provide generic
support to achieve this. A test manager should select these tools and customise
them according to testing needs, instead developing a new tool. Developing new
tools could be a time-consuming and costly affair and should be developed only
as a last resort. Customisation also should be generic and there should always
be room for easy and quick enhancement.
Development and verification of scripts
Scripting effort includes development and
testing of scripts. This depends on two factors. First, the skill of the persons
involved and secondly the capability of the test tool to provide flexibility
to develop scripts for all valid and invalid scenarios. However, scripting should
always be done in a modular manner so that the same modules can be reused in
different scripts. A modular approach would further lead to less scripting time
and also help in testing and reuse across different releases. All scripts should
be tested with a release prior to the actual test execution so that no problems
are found in scripts during the real testing run.
Implementation of a Test Management System
This is done to facilitate complete automation
in terms of managing the execution of scripts at all the external interfaces
of the SUT and thereby enabling the tester to consolidate the test results at
one place. This is not applicable if complete automation is not targeted. As
an example, the figure below depicts a sample configuration where there are
individual tools (Upper Tester–UT) above the SUT and at the Peer side (Lower
Tester–LT) and also a test manager, which controls the complete testing cycle
automatically. Here UT and LT are the external interfaces to the IUT.
The test manager provides a communication
interface, which enables remote control of the test scripts running at the external
interfaces of an IUT. This helps in the following functions:
- Management of test cases
- Management of test activities with schedule, priority
- Management of test results and log files per
release basis
- Interface with automation tools
- Generation of test reports, statistics, trends
This configuration helps in deciding the
order of test execution and eliminating the need to manually trigger the scripts
at different external interfaces of an IUT and having the complete status of
the test execution at one place.
Automated testing: A complete procedure
While planning the automated testing process,
the activity should be planned as a separate process running in parallel with
the test activity of the product. Milestones of automation activities should
be in sync in the selection of an appropriate test tool, efficient and error-free
scripting and proper time synchronisation of testing and automation activities.
|
TEST ACTIVITY |
AUTOMATION ACTIVITY |
|
Test strategy finalisation |
Finalisation of test automation level targeted anmaking of test automation
strategyd |
|
Test plan preparation |
Customisation of tool, making of basic messages |
|
Final test plan available |
Start making test scripts |
|
Beta release available |
Test scripting completed, testing of scripts started |
|
Final release available for evaluation |
Start testing with automated scripts |
Sunil Gupta is director of Engineering at Hughes
Software Systems. He can be reached at sngupta@hss.hns.com
|
