Software Regression Testing

Attain A Known State And Stay There

It is always preferable to release software in a 'known' state compared to software in an unknown state which may contain surprises for both the developers and the users!

Making changes to software which is in a known state can, and often does, pose a serious threat to that known state. Every developer knows that even the smallest change can render software useless if the implications of the change were not properly understood or if the change was insufficiently tested during and after implementation.

Changes to software are generally unavoidable. Eventually the need arises to amend software, either to correct defects or to modify functionality, and those changes may be required at short notice. It is under these circumstances that the investment in automated regression testing tools can begin to pay back big time.

The Need For Automation

Without automated regression testing tools the emphasis remains on manual testing to re-affirm the 'known' state of software after changes. This may tie up a large number of expensive resources or simply prevent changes from being delivered successfully at short notice. Under such circumstances pressures can lead to the shipping of software which has not been sufficiently re-tested after changes, sometimes with dire consequences.

To remain competitive, software developers must be able to implement changes to software quickly and reliably. Whilst no one doubts that changes can be made quickly, doubts about the reliability of the changes must be dispelled with proof. To support rapid change, testing must therefore be both thorough and quick, leaving little option but to automate the testing process.

Sow And Ye Shall Reap ... Eventually

The decision to use automated regression testing tools is not one to be taken lightly. Acquiring the tools, which may be expensive, is only the beginning. Expect to incur further costs for training and test script creation as well as for maintenance to keep the test scripts in synchronisation with your changing software.

For feature-rich software, the lead time to create test scripts to ensure adequate coverage of both the software's key features and its code base should not be under-estimated.

If adequate resource is not applied in the early stages of development then the test scripts may continually lag behind the software and thereby never offer a full coverage test leaving the state of the software 'unknown'. It is therefore essential that your project schedule reflects both the resource and effort required to create, maintain and execute regression tests to achieve full coverage of the code base.

Ensure Tests Provide Full Coverage

For software with a large code base, ensuring full coverage is not a small undertaking and you should therefore plan to deliver coverage in stages, perhaps choosing to address the most important or commonly used software features first.

To achieve full coverage requires two technical challenges to be overcome. Firstly, there is the challenge of identifying the lines of source code which are executed during testing, and those which are not. Secondly, comes the problem of designing new tests to exercise code paths which are currently not being reached.

If full coverage of the code base is not achieved by testing then the software remains in an unknown state and thus carries an uncertain risk of failure.

Note: Unless source code contains suitable reference comments to enable tracing of code back to functional requirements, it can prove a difficult task to design new tests that ensure coverage of all code paths. Dead code can also be misleading and is best removed.

Design Applications With Testing In Mind

The purpose of regression testing is to exercise all code paths fully and to confirm that the software under test continues to function as expected. The tests thus serve as an 'insurance policy' to aid identification of unexpected status changes during software operation.

To achieve this goal, regression tests must contain regular check points where the current status can be compared with the expected status and any mismatch reported immediately. By taking regular snapshots of screen images and data etc. comparisons of current versus expected status can easily be made and differences detected quickly.

Because not every element of functionality can be linked to visual display elements, it helps if features within the software can be enabled during testing to create additional trace output which can be used in snapshot comparisons. (Debug builds created by conditional compilation switches enabling verbose trace information provide an excellent means of achieving this.)

For some components, it is possible to perform isolated testing via the use of test harness applications while database tools may provide facilities for extracting data, or details of data which has changed, which can then be used in making comparisons.

The greater the number of check point comparisons, the more likely the regression tests are to find differences when they occur. Tests must be designed carefully so as to minimise unnecessary maintenance work on test scripts whenever software changes are made, yet retain their effectiveness at detecting differences.

Failure to detect a difference means that the impact of a change may pass unnoticed leading to potential software failure downstream. Regression tests around software integration points in particular must be constructed to ensure impact on external systems is detected as early as possible.

Note: Comparisons must take into account (or preferably design out) differences between operational and baseline data occuring due to date/time related fields and values.

Schedule With Testing In Mind

Just as logistical issues related to coding tasks require careful planning and sequencing to implement them efficiently, so too do testing tasks.

Changes to functionality should be implemented in stages whenever possible thereby allowing regression tests to be more easily kept in synchronisation with software changes and for new baseline comparisons to be established.

New functionality added to software should always be matched by new regression tests and confirmation that new baseline comparisons have been created and that all new code paths are covered by the tests. Bug fixes on the other hand are likely to require modification to existing tests and baseline comparisons.

Conclusion

Successful integration of automated regression testing tools into the development process offers greater levels of confidence that changes introduced into software at short notice will not go without testing nor will they cause unexpected consequences when the software is later shipped to users.

Automated testing enables thorough testing of software to be conducted both quickly and repeatedly. No longer are expensive resources tied up for long periods of time to repeat tedious manual tests.

The efforts of testers can instead focus on designing and implementing high quality test cases. Once designed and scripted, each test can be repeated quickly on demand in a fraction of the time taken to create the test case.

The benefits of automated regression testing tools however do not come for free and require considerable up-front effort before they can be realised. Without automation, the emphasis remains on tedious and slow manual testing to ensure that all changes are thoroughly re-tested before software is shipped. If you need faster turn-around of changes combined with higher levels of confidence that your changes didn't 'break' things, then perhaps automated regression testing can help you.