Contract tests assert that inter-application messages conform to a shared understanding that is documented and verifiable. By testing against this shared understanding (a contract), it is possible to validate the compatibility of inter-connected systems before they are ever deployed to an integrated environment. This helps to enable more consistent stability of the integrated environments and helps to reduce the impact on other teams that are dependent on integrated environments, by enabling developers to detect breaking changes while working on their local machines and as an automated quality gate before deploying any changes.
Bi-Directional Contract Testing is a “Specification First Design” approach where the design API is constructed in the first place and then the actual implementation of the Provider takes place. It provides the same level of confidence that integrated system messages will be compatible between systems but allows for a more de-coupled workflow. Consumers still write unit tests and generate PACT files to serve as their documented expectations of provider service, however, Providers will instead validate their real service against all the versions of the published OAS spec of that service, and PactFlow will cross-check that both Consumer and Provider sides are compatible based on their results.
Contract Testing Best Practices:
All Providers and Consumers should follow a consistent naming convention to ensure contracts are clear on which services/systems the interaction represents.
A Contract Test should cover connections to other systems/components where logic depends on any specific content in response to ensure the ability to identify breaking changes on the Consumer and Provider sides of any interaction.
Only connections where a consumer uses the response body should have Consumer Contract Tests written, and only for the fields used by the Consumer.
Provider services must be updated to match their OAS spec if they aren’t currently.
Create contract tests for the minimum data you expect from a given provider.
Always use an actual method from the service class to call the mock server in a test.
Use the same mocks for Unit, Component, and Contract tests.
Simply running PACT tests does not perform any compatibility validation. PactFlow does the cross-check once both the Consumer and provider upload their PACT files (and OAS spec).
Can-I-Deploy is the real gate-keeper of ensuring compatibility & stability and should be leveraged to realize the maximum benefits of Contract Testing.
Providers must ensure their real service and their OAS spec are in sync.
This is done through provider-side schema validation in the Bi-Directional Contract Testing approach.
Developers must run their contract tests locally before committing changes to catch issues as early as possible.
Best Practices for Writing Consumer Tests:
The Key Is To Know What NOT To Test
- The real goal with a consumer contract test is to document the expectations that you, as a consumer, have for the given provider
- If there are response types that will go unprocessed by the consumer, or no logic is dependent on the content of a response, then don’t write tests or generate contracts for those
- Don’t test and assert fields in response types that go unused
Only Document Real Expectations
- Consumer teams should only be mocking out the properties they care about in their PACT test, not necessarily the entire response that comes back from the provider.
- Only test and assert the fields that you actually use from the response! Otherwise tests will fail even in cases where the consumer and provider would actually still remain compatible
Aim For The Minimal Set Of Necessary Assertions
- There should be sufficient assertions to ensure that the provider response will not break the consumer
- The scope of the consumer code under test should be kept as small as it can be (these aren’t functional tests nor end to end tests)
- Keep the expectations on the response as loose as they can be while still being compatible for the consumer
Reuse Where Possible
- If there are existing mocks used for unit/component testing, the same mocks should be used to run tests to generate the Consumer contracts, as this will help keep everything up to date with changes
- Use the same mocks for Unit, Contract, and Component testing where ever possible if they already exist
Adoption Expectations
Note: This is only applicable to teams that use or build custom-developed APIs.
| Practice | MVP | MVP+ |
|---|---|---|
Train the team on Contract Testing |
+ |
|
Implement Contract Testing for one service |
+ |
|
Implement Contract Testing for all the services |
|
+ |
Tools:
| Functionality | Tool Name |
|---|---|
Contract Broker |
PactFlow |
CI/CD |
Azure pipelines |
Test code |
Programming language used by the team |
Roles:
| Name | Responsibilities |
|---|---|
|
Create Consumer contract tests against each external API that is used in their codebase Create Provider contract tests against each API that is developed within their codebase Ensure contract tests are configured and sent correctly into PactFlow Run contract tests locally after updating any code related to APIs / API interactions |
|
Create application, environment, team structure(s) in PactFlow to manage the roles/permissions/configurations in PactFlow for their system(s) |
|
Incorporate Contract Testing workflow into CI/CD pipelines to allow Contract Testing to serve as an automated quality gate |
