Quality Gates

Enforcing quality gates through pipelines is essential to a robust software development process. Quality gates help ensure that software artifacts meet certain criteria before they can progress to the next stage of the pipeline.

Approach to Quality Gates

Here's a detailed approach to enforcing quality gates through pipelines:

Define Quality Criteria: Start by defining the quality criteria that need to be met at each stage of the pipeline. These criteria could include code quality standards, test coverage thresholds, security checks, performance benchmarks, or any other relevant factors contributing to the software's overall quality.
Identify Pipeline Stages: Identify the different stages of your pipeline where quality gates should be enforced. Typically, a pipeline consists of stages such as code compilation, unit testing, integration testing, static code analysis, security scanning, and deployment.
Select Appropriate Tools: Choose the right set of tools and technologies that can help automate the enforcement of quality gates. Depending on your specific requirements, you may use tools like Jenkins, GitLab CI/CD or any other continuous integration and delivery (CI/CD) platform. Additionally, you might consider integrating other quality-focused tools like SonarQube for code analysis, Veracode for security scanning, or BlazeMeter for performance testing.
Implement Automated Checks: Integrate automated checks into each stage of the pipeline to enforce the defined quality criteria. For example, in the code compilation stage, you can ensure the code compiles successfully without errors or warnings. In the unit testing stage, you can set minimum code coverage thresholds that must be met. In the security scanning stage, you can perform vulnerability scans and ensure the software is free from known security issues.
Define Gates and Exit Criteria: Establish the specific conditions that must be met for each quality gate to pass. This could include metrics such as minimum test coverage percentage, the maximum allowed number of code vulnerabilities, adherence to coding standards, or meeting specific performance benchmarks. Clearly define the exit criteria for each stage, determining whether the software can proceed to the next stage.
Failure Handling and Notifications: Determine how failures or violations of quality gates should be handled. When a quality gate fails, the pipeline should be halted, and appropriate notifications should be sent to the relevant stakeholders. This ensures that any issues are addressed before proceeding further, maintaining the integrity and quality of the software.
Continuous Monitoring and Reporting: Implement mechanisms to continuously monitor the pipeline and generate reports on the status of quality gates. This allows for visibility into the overall health of the software artifacts at any given stage. Reports can be shared with the development team, quality assurance team, and other stakeholders to facilitate transparency and collaboration.
Iterative Improvement: Quality gate enforcement is an ongoing process, and it's essential to evaluate and improve the criteria and checks continuously. Regularly review the quality criteria, adjust thresholds if needed, and incorporate feedback from the team to optimize the pipeline's effectiveness in ensuring high-quality software.

Following this approach, you can establish a robust system for enforcing quality gates through pipelines. This ensures that software artifacts progress through the pipeline only when they meet the defined quality criteria, reducing the risk of introducing bugs, vulnerabilities, or performance issues into the production environment.

An example of Quality Gates is given below. Teams are expected to define and adapt to their current state, continuously improve and mature it.

Quick Links

Capacity Calculation for PI Starting a New Initiative QE Practices QE Approach

Meet code linting standards: Ensure that the code adheres to the defined coding standards and has no linting issues
100% user requirements coverage: Verify that all requirements have corresponding implementation and test coverage
<5% valid requirement defects: Limit the number of valid defects related to user requirements to less than 5%
90% Code Coverage & Stability Index: Maintain a code coverage of at least 90% to ensure sufficient test coverage. Additionally, measure the stability index to ensure code stability
No Critical Performance Design Defects: Ensure there are no critical design defects related to performance in the code

100% successful pass rate of all unit tests with a 90% code coverage: All unit tests must pass successfully, and the code coverage should be at least 90%
80% automated suite: Ensure that at least 80% of the test suite is automated, reducing manual testing efforts
<5% Valid Design Defects: Limit the number of valid design defects to less than 5%

Successful static analysis scans: Perform static analysis scans on the codebase and ensure no critical issues. All code should pass the static analysis checks
100% code coverage: Achieve 100% code coverage to ensure all code paths are tested
Environment readiness check (health check): Verify that the environment is ready for deployment by performing a health check. This includes checking the availability of required services, dependencies, and configurations

Successful passing of all post-deployment checks: Ensure that all post-deployment checks, including health checks, smoke tests, and critical regression tests, pass successfully
90% of test cases executed and 80% passed: Execute at least 90% of the defined test cases and ensure that at least 80% pass successfully
80% Automated Runs: Automate a significant portion of the test suite, aiming for at least 80% test automation coverage
<10% Valid Code/Build Defects: Limit the number of valid defects related to code or build to less than 10%

Successful passing of all automated functional and integration tests: Ensure that all automated functional and integration tests pass successfully

Successful completion of dynamic code scans: Perform dynamic code scans to identify and address security vulnerabilities, code defects, or performance issues. Ensure that the code passes the dynamic code scans without critical issues.
Environment readiness check: Verify that the deployment environment is fully prepared and meets the necessary requirements for the software to run effectively. This includes checking the availability of resources, configurations, dependencies, and any other prerequisites
Manual functional test completion: Execute manual functional tests to validate the behaviour and usability of the software. These tests should cover critical user workflows and validate that the software meets the expected functional requirements

Successful passing of all post-deployment and smoke tests: Ensure that all post-deployment tests, including smoke tests, pass successfully.

Successful passing of all automated and manual functional E2E and regression tests: Ensure that all automated and manual functional E2E and regression tests pass successfully. These tests validate the end-to-end functionality of the software and check for any regressions
95% test execution and 90% passed: Aim for a test execution rate of at least 95% and ensure that at least 90% of the tests pass successfully. This metric measures the thoroughness and effectiveness of the testing effort
99% E2E business processes validated: Validate at least 99% of the critical end-to-end business processes to ensure the software meets the required business requirements and workflows
Zero Critical and <3% Major defects: Aim for zero critical defects and limit the number of Major defects to less than 3% of the total defects. This ensures that the software is free from critical issues and maintains high quality