3 Regression Testing Best Practices
Though it tends to be a catch-all term, regression testing is most commonly the effort made to verify that a bug fix has achieved its goal of correcting a code defect. An expanded definition includes testing for additional issues that may have been created by that bug fix.
With the broad flow of data between system modules and utilities, it is common for a change, whether a fix or a feature addition, to cause an unexpected problem in system operation that is difficult to trace back to its cause.
Regression Testing is a Key Element of Software QA
Add to this the possibility that two apparently unrelated changes can conflict with each other to cause one of these ‘phantom’ problems and managing regression testing becomes a key element of software QA.
That management should consist of:
• planning and documenting regression test cases
• coordinating them with functional and performance testing, and
• automating regression test suites whenever possible.
Considering those objectives in reverse order is a good way to prioritize the process of attaining them as each step will drive the necessity for the next.
1. Automating Regression Testing
Test automation has become both a cure-all and a whipping boy for QA process improvement. On one hand, it forces consistent and ordered testing efforts and on the other it seems to require endless resources that could be producing revenue generating code. Ironically, these two conflicting aspects make regression testing an ideal QA segment for automation.
Regression tests need to be run for every system code build. When automated, they can be triggered by the code control system every time such a build is generated. This will ensure that nothing has been changed that undoes some previous feature or fix. That is the process order generated by automation.
However, the developer/tester resources who will ensure that the regression tests are maintained along with the system code base must still be devoted to script maintenance. An automated test that no longer reflects current system operation will generate false error reports that will lead to the test simply being ignored and make it a useless waste of time and resources.
2. Sanity Check Coordination
A well maintained code base will have a sanity check to keep watch over all the main system features as they are expanded and defect fixes are implemented. Such a check is the first line of defense against what Microsoft has called the “escape rate” of defects that go undetected in public releases. Sanity checks are another prime candidate for test automation and can be coordinated with regression testing to ensure that both are triggered for all code builds.
That coordination can be used to keep both the sanity check and regression test suite up to date with the latest feature/bug fix compliment reflected in the system code. Regression tests should be generated as the code is introduced into the system and reflected in the sanity check when functionality changes are a result of that introduction.
3. Test Planning in the Age of Agile
The casualties of the Agile development process push for release velocity have unfortunately included system and especially test documentation. Regression testing is not immune to this and the lack of documentation makes test scripts very difficult to maintain. A work around is to make detailed use of the defect tracking system to generate regression test case documents.
Both development and QA can use the defect tracking system to document everything they find that doesn’t work exactly as expected. An essential part of that documentation is a list of steps that consistently reproduce the symptoms of the defect it describes. These steps, along with any required data constructs, form the basis of a regression test case specification.
This use structure for defect tracking allows each test case to be extracted into a regression test plan document that sets the stage for creating the test scripts that will automate the process. The case descriptions should also be included as header comments in the test scripts themselves to ensure that they stay with the code.
Regression Testing is Your First Line of Defense
As with all software QA, a major goal of regression testing is transparency that supports the organization’s understanding of exactly what the code system will and won’t do. The regression test suite is the first line of defense against defect leakage into public releases.
To be effective, it must be scrupulously reviewed and maintained, or it will become a report nuisance and be ignored. Making regression testing an integral part of a vibrant test automation process is the best way to keep it relevant and to assure that it is used to guarantee the integrity of its system.