UAT Guidelines

Published date: April 15, 2024, Version: 1.0

Purpose

The purpose of this document is to define User Acceptance Test (UAT) methodology in the QE SAFe® Agile framework. The scope of this document is to set a standard for UAT to ensure a consistent and predictable outcome.

Definition

UAT is to deliver the right product by ensuring that applications and services meet business, customer and stakeholder demands by validating end-to-end solutions and business interoperability concerning features/requirements that are planned. UAT must be completed before the software can be released to the market.

Quick Links

QE Approach QE Approach in SAFe® for Agile Requirement Analysis QE Approach for Waterfall

Objectives

To ensure the software can handle real-world tasks and perform up to development specifications.
Boost collaboration between customers and developers to create a holistic product that meets requirements.
Increase the number and type of potential defects found through a new tester perspective.

What should be covered in UAT?

E2E User Journey Testing: Business process/End to End-test cases profiled by the business.
Usability Testing: Test for usability against the specifications.
Analytics-Driven Testing: Business risk profiles, frequent failure points and voice of the customer.
Exploratory Testing / User Free Form Testing: Unscripted and random exploratory scenarios from a user perspective.

How should UAT be done?

The business Team should drive UAT. Business users should get involved in all phases of UAT, such as Strategy, Plan, Design and Execution. Business users should engage right from PI planning and have active involvement throughout Product delivery.
In Agile, the Business Users should perform incremental UAT for each sprint and release UAT before Product release.
UAT should be done in a fully integrated Staging/UAT environment, which should mimic the production environment.

Approach to User Acceptance Testing in Agile

UAT should be planned from Strategy and Planning stage even though UAT happens towards the end of testing.
In Agile, UAT happens incrementally with respect to Sprints or PI releases. UAT execution occurs at the end of the sprint and before the release of a PI of the product.

UAT Planning

High-level strategy inputs on the below key areas are updated in the overall strategy, and a detailed plan is created with more specifics required for execution.

UAT Environment should be like Prod to mimic real-time scenarios and an integrated environment to perform E2E User Journey/ Business process testing.
Project Manager to raise a request for booking UAT environment with environmental requirements.
Quick validation of environment for usage in the Planning phase by BAs or SIT Team.
Finalize the UAT environment and update the Plan/ Global environment plan.

No. of Test Cycles for UAT may vary depending on the complexity of changes, Business Risk, and volume. In most cases, two cycles of UAT are done by covering Business risk coverage-based test cases in Cycle 1 and Exploratory/ Defect retesting in Cycle 2.

Analyze Requirements (Waterfall) / Features and User Stories (Agile) for E2E User Journey / BPT and gather Test data requirements (Pre-requisites and actual).
Test data that can be created with Automated scripts.
Test data can be requested from the TDM team in advance.
Test data that need to be created manually.

UAT team to provide effort for Design and Execution that will be included in the UAT project Plan or Sprint Plan.

Ex: Real users / Business – PO to coordinate and identify the right business users for execution, line up BAs to support etc.

UAT Design

UAT Design is done by Business (Scenarios), Business Analysts (Test Cases / User Stories), and review is done by the Business.

UAT Design activities:

Participate in requirements walkthrough / PI Planning / Sprint refinements to understand requirements in scope
Create UAT Test scenarios as per coverage guidelines mentioned in the next slide
Create Test Scenarios with detailed steps
Review test scenarios with business and incorporate changes
Manual vs Automation mapping – Identify test cases that are covered as part of Automation with the help of SIT team and categorize test cases for manual and automation execution
Upload test cases to qTest and tag them with the Release cycle. Change status to “Ready for UAT”
Create Test Design traceability with Test Case / User Stories – Scenario – Requirement (Functional requirement, User Story, Subtask of User story, etc.)

UAT Test Design Coverage Guidelines:

Test scenarios to cover the complete business process or journey for a change in one of the processes. UAT scenarios should not be feature or system test cases.

Exploratory scenarios are executed on the spot with different ways of navigating and checking user views.

Coverage does not have to be detailed with all probability of test data combinations but key scenarios that cover high business impact and critical areas.

A business Risk Matrix should be created to have high impactful UAT and test, in the order of priority and Risk:

with different types of groups, types of plans and customer combinations (as applicable) for the requirements in the test.

Categorize with Usage frequency, Business Criticality etc. Assign weightage.

Categorize with Changed areas, complex areas, defect-prone areas etc. Assign weightage.
Perform Risk approach with the above areas and assign weightage.
Sort by Risk and assign priority with high and medium to Cycle 1 and Low to Cycle 2.
Assign Exploratory and Defect retesting to cycle 2.

Legal consequences by not fulfilling government requirements, History transactions, Product being used less frequently, High Sales product etc.

Number of change requests, sub-functions within a function, user experience etc.

UAT Execution and Defect Management

Product Owner to confirm UAT team for the product under test and block time for execution. UAT is done in a working session with UAT Testers, BAs, and SMEs. Product Owner coordinates with all stakeholders and provides software details, access and required artifacts and details to UAT testers to proceed with testing.

UAT Execution Activities

UAT testers / Support team quickly validate Environment, Product access in a given environment, Test cases and test data requirements.
The Product Owner provides a product walkthrough and changes to UAT testers with the business team.
Test cases are downloaded from qTest and shared with UAT testers for reference, or access is provided in qTest.
UAT testers create test data wherever possible, or SIT/ BAs support them in providing the Test data.

Cycle 1

UAT testers log in to the working session and perform Cycle 1. Test scenarios/ Test cases with high Business risk and high and medium priority (In dedicated scheduled days for cycle 1 of UAT) are executed.
Discuss and clarify with the support team (PO, BAs and SIT SMEs).
UAT testers to raise issues and triage. PO / BA/ SIT to submit valid defects in Jira with priority and severity for closure.
PO and SIT team to coordinate with the Dev team in triages and track defects to closure.
Post execution, update test case status in qTest.

Cycle 2

UAT testers log in to the working session and perform Cycle 2. Random Test scenarios of Exploratory E2E User Journey in a user perspective are executed, and perform Defect retesting.
Post execution, update test case status in qTest.
SIT team supports executing sanity/smoke tests in UAT environment and impact regression through automated suites and manual combination. SIT team enables UAT testers to stitch together components to automate and execute.

UAT Sign-Off

The business owns the ownership on UAT and provides sign-off for Release Go / No-Go based on the below Acceptance Criteria.

UAT team review the results, Coverage against the Business risk matrix and open defects to ensure the following:
- All high and medium-risk items are executed, and actual results are compared to expected ones.
- Discrepancies in actual vs expected results were identified and documented as defects in Jira.
- All Sev 1 and Sev 2 defects are closed. Other defects are documented with a workaround in Jira for future testing.
- All in-scope product types and all portfolio types are tested successfully.
- Traceability coverage with original requirements and Business Risk Matrix is complete and reviewed.
- Test planning documents (Test Approach, Test Conditions, Test Scripts / User Stories, Test Data, expected results, test scripts mapped to requirements, etc.) are updated as necessary.
- The closure summary report is completed and published.
Once the above Acceptance Criteria are satisfied and signed off, Product Owner should plan for release activities.

Entry and Exit Criteria

UAT Phase	Entry Criteria	Exit Criteria
Strategy and Planning	IT Product Roadmap, Epics and Features End to End architecture documents	Strategy and Release plan updated with UAT UAT Plan Feature Planning User story planning
Design	UAT Plan / Features / User Stories	Test Scenarios / Test Cases for E2E User Journey testing BRM and Test cases execution cycle plan based on BRM
Execution and Defect Management	Environment details Test Scenarios / Test Cases and Test Data Business Risk Matrix Execution plan Software / Product and access 80% of SIT with 90% pass Incremental UAT All test components have been built and migrated to UAT environment Code Freeze – No change being accepted outside of defect resolution for release UAT No open critical defects from SIT There are no Major / Minor defects without a documented workaround in the system found during the Unit / System Integration Test or Application Functional Test Phases relevant to products in scope for E2E User Journey testing. Test conditions, scripts, data procured/created and signed off by relevant stakeholders	Execution Summary of 2 cycles Defect Report with no open showstopper / High defects. Other defects are documented with workarounds and details in Jira Traceability with requirements, Test cases, Defects and BRM
Sign-Off	Execution Summary Defect Report Traceability	Go / No-Go report Sign-Off

UAT Automation

Partner with SIT team to automate E2E User Journey scenarios wherever possible:

SIT team to provide automated test cases in the scope of the release.
Automation team or SIT team to train BAs and business on Test Complete Unified Automation Test Framework.

Automation Design:

Identify E2E User Journey scenarios to be automated.
Coordinate with SIT team for keywords and components of the product under test (Reuse from SIT) and create automated components as required.
Stitch together the components for E2E User Journey testing.
Identify supporting test data and volume - Coordinate with SIT team on executing automated scripts to search for data or execute pre-requisite. Request the TDM team for synthetic data ahead of execution.

Automation Execution: