Importance of Regression testing
Experience has shown the re-emergence of faults is quite common throughout the software development lifecycle. Sometimes it occurs because a fix was lost through poor revision control practices (or simple human error in revision control), or a fix for a problem is "fragile" - i.e. if some other change is made to the program, the fix will no longer work.
Risk 1
- A change intended to fix a defect is not successful
Risk 2
- A code change caused the recurrence of an old defect of the previous build/release
Risk 3
- A code change introduced by a new defect
Regression Testing guidelines
The general guideline is 10-30% per major functional module, depending on the complexity, severity and number of defects found in current and past releases, as these are indicators of code stability.
10% Exploratory testing of functional modules where no identified code changes occurred to provide a minimal check to ensure we didn't overlook the integration point. This testing can be done without creating new test cases.
10% Test coverage for non-invasive code changes such as UI changes.
No Severity 1 or Severity 2 defects were found during test execution.
30% Test coverage for complex code changes such as multiple integration points.
A limited number of Severity 1 or Severity 2 defects in isolated code.
50% Test coverage code changes with multiple Severity 1 and Severity 2 defects that crossed integration points – for instance, one defect is fixed and created two more.
Methods of Regression Testing
Regression testing may be performed manually or through automation, or some combination of both. As regression testing is a rerun of the test suite, automating the entire regression test suite shall save time and result in a better ROI. A feasibility study/assessment on automation of the regression test suite should be conducted to decide whether additional automation can be beneficial.
Test cases intended for Regression Testing will be flagged in qTest by introducing a new custom field – Automation status to track the automation status of the test case.
Regression test cases intended for automation will be flagged by selecting Ready in the Automation Status field in qTest. This status will indicate when the Automation Engineer can review the test case to determine automation readiness.
| Automation Status | Status Definition |
|---|---|
Not Ready |
The status will default to Not Ready. When the test case is in this status, it is incomplete or has not been reviewed by the QE Analyst to determine if it is ready to be automated. |
Ready |
The QE Analyst will set the test case to Ready when it is complete and ready for automation. A test case is ready when it is fully detailed with all test steps, allowing the Automation Engineer to manually execute it and see the expected outcome to automate it. |
Rework Needed |
The QE Analyst will set the test case to Rework Needed when the functional behaviour has changed since the test case was automated, making automation updates necessary. |
Rejected |
The Automation Engineer will set the test case to Rejected when the test case is not sufficiently detailed to allow automation to be performed. A test case in this status requires further work by the QE Analyst to add details regarding test steps and expected outcomes. |
In Progress |
The Automation Engineer will set the test case to In Progress when the test case is being automated. |
Complete |
The Automation Engineer will set the test case to Complete when the test case has been automated and can be run during regression instead of manually executing the test case. |
Not Automatable |
The Automation Engineer will set the test case to Not Automatable when the test case steps do not lend themselves to automation because verification requires a QE Analyst to review the results. |
Regression Testing across STLC
Planning
This phase consists of preparing the Test Strategy, Test Plan and Test Scenarios after an impact analysis of requirements and design specifications. As requirements are elicited, the QE team will capture test scenarios for test case identification later. The resulting test scenarios shall be a combination of new test cases and regression test cases. The impact analysis performed will facilitate the regression selection process for existing functionality.
Modifications to existing applications are identified by analyzing the Specification documents, product backlog and user stories.
New applications that have interdependencies and/or interface with existing functionality
Defect fixes for existing functionality within the project release
Defect fixes for new functionality within the project release
Post-analysis, the test estimation will be performed. The template considers complexity and weighting to create a view of the level of regression required.
Entrance Criteria: Spec walkthrough, Backlog grooming and clarifications
Exit Criteria: Defined Regression Test Suite
Test Design
This phase involves the identification/preparation of test cases for new functionality and design specifications, along with the refinement of the Regression Test Suite. As the Development Team progresses with the design for new functionality and modifications to existing functionality, further clarification will be provided to enhance test planning and design.
During test creation, selective test cases will be flagged for the regression test suite.
New test cases will be designed for defects targeted for inclusion in the project release, and these test cases will also be targeted for the regression test suite.
Refinement of the Regression Test Suite will continue through test design to ensure boundaries based on impacted functionality.
Entry Criteria: Design Phase output created and approved
Exit Criteria: Test case creation complete and Regression Test Suite finalized
Test Execution
A subset of the planned System Test & System Integration test cases shall be used as the basis for the Regression Test Suite. The selection will be based on the criticality of functionality, requirements prioritization and overall business criticality. Regression testing guidelines, as detailed above, shall be followed.
A limited regression will be performed in each test cycle to evaluate code stability.
At the end of the System and Integration Test, a final regression test cycle shall be performed to evaluate code stability.
Entry Criteria:
Unit test results provided to the Quality Engineers
Code deployed to the Test environment
Exit Criteria:
No Severity 1 or 2 defects
No uninspected defects
Business signoff for existing defects
Workarounds documented for existing defects
Test Closure
The Test Closure Phase begins when test case execution is complete. The QE team shall document the application's state, known defects, workarounds and final test metrics.
Review the test case library to ensure all test cases have been updated to reflect the current state of the application at release based on change requests, defect fixes, etc.
Review the regression test case library to validate that the test cases flagged for regression are still valid.
Review regression test case library to flag test cases ready for automation.
Review automation scripts to ensure all scripts have been updated to reflect the current state of the application at release based on change requests, defect fixes, etc.
Produce project-level metrics for all testing performed
Entry Criteria:
All test cases planned for test execution have been executed and passed, or defects documented and accepted by the business
Test cases are up to date, reflecting the current state of the application
Automated scripts are up to date, reflecting the current state of the application
Performance scripts are up to date, reflecting the current state of the application
Exit Criteria:
Approval and signoff have been received for all STLC Phases
Business signoff for release to production received
Final Requirements Traceability Matrix produced and distributed to the project team
Final project test metrics produced and distributed to the project team
Regression Test suite creation workflow
Workflow – Below workflow illustrates the creation of manual and automated regression suites with maintenance flow.
Test suite creation
The activities for Test Suite creation for Regression Testing are outlined in the steps below:
Step 1: Create a high-level mind map for the application
Step 2: Detail each process of the application to the scenario level
Step 3: Identify the critical business functionalities
Step 4: Prioritize the functionalities/scenarios according to criticality
Step 5: Review the prioritized scenarios with the Business team
Step 6: Build the regression suite accordingly
Step 7: Review the regression suite with the Business team
Step 8: Maintain regression test suite/library as an ongoing activity in support of any Customer Care or Release changes
Test Suite Maintenance
Review the mind map during Post Implementation phase of each release and update catering to Business and technical needs.
Review existing regression test suite during Post Implementation Phase and provide recommendations to Business and IT for needed functionalities.
Always maintain regression test library for Integration and System testing functionalities.
Review existing Manual/Automated test scripts for reusability and re-testability and provide recommendations on the functionalities/test Scripts that can be eliminated from the current regression test library.
There is a trade-off between possible redundant testing costs and the costs of deciding which test results can be reused and which manual/automated test scripts can be rerun.
Some manual/automated test Scripts may need to be updated during Post Implementation phase.
Identify regression test functionalities and review with Business and IT to add to the regression test suite.
