Tatiana ShabskayaQA Engineer

How the definition of quality evolves with a project


In 2014, Facebook changed their famous “Move Fast and Break Things” motto to “Move Fast With Stable Infrastructure.” As a project keeps getting more mature, initial project goals and focus may change. We should make sure the definition of sufficient quality evolves along with the project.

All of us are interested in doing our job the best we can. But in reality, sometimes you can feel that it is challenging to define what is the best possible quality, given time and budget constraints. You've probably seen this diagram - but it's still relevant today:

The golden triangle of quality - time, scope, cost.

The definition of quality and project goals

What is quality? There are multiple definitions. In general, it is a level of satisfaction by conformance to expectations, both implicit and explicit, by all stakeholders. By stakeholders, we mean everyone who is affected by the product and project state: your customers, your users, your developers, your suppliers - you name it.

The definition of quality should correspond to the project goal and context. It is basic common sense that you won’t approach development and management of a prototype project the same way as you approach a long term or high-risk domain project. And you cannot expect the same level of quality in these two project types.

How to define your test approach

  • First, discuss project business goals, assess project risks, and determine expectations of quality from all stakeholders. Depending on the project goals, you need to understand your stakeholders’ expectations for the project and define what quality means to you.
  • Based on the project context, you should select applicable test activities that help reduce those risks and which meet the definition of quality as well as achieve business goals. Then develop an approach that helps you meet the required level of quality. Also, you should develop a strategy to test this.
  • Discuss and approve test activities along with dev activities. They should be included in the project scope, as they might be quite time consuming.
  • After approval, note them in the project scope and in the Definition of Done.
  • After that, establish a testing process in the team.

But here is the trick: later, as the project evolves, the project goals may change. The basic test strategy and approach should be updated along with the project evolution.

How the definition of quality evolves with the project

It is a common thing that a lot of changes are introduced during the project development. New requirements are uncovered, both functional (new features) and non-functional (e.g. accessibility, performance or localization requirements).

The goal is to have those expectations in mind when developing the test strategy and to share a new vision of quality as it evolves with the team.

For instance, if you introduce new accessibility requirements, it’s not enough to just change the code and test the changes once. It is crucial to add accessibility criterion to the definition of done for all upcoming features moving forward.

Our suggestion is to have regular scheduled test strategy reviews with key project stakeholders (e.g. once a quarter or once half a year) to make sure the definition of quality and test approach correspond the current project goals.

Below, you will find three examples of definitions of quality vs project goals to give you an overview of how the test approach may vary from project to project:

  1. Proof of concept or a prototype
  2. Short-term project (without further support)
  3. Long-term project

Proof of concept or a prototype

Project goals: Understand the value or fitness of the solution. Both time and budget are constraints.

Definition of quality: The product should be just good-enough to understand the value of the solution.

Possible test approach:

  • Manual ad-hoc testing
  • User acceptance testing (UAT)
  • UX testing (if applicable)
  • No time investments in dev support activities (unit tests, peer reviews, etc).

Risks: All project members involved should understand unambiguously that this type of project is not of release quality and was not designed for maintainability. Functional and nonfunctional requirements are not applicable for this type of project. Its only goal is to determine the value of the solution.

Short-term project (without further support)

Project goals: Develop a short-term solution for a specific purpose. Usually, time is the main constraint.

Definition of quality: The product should meet functional and nonfunctional requirements, such as reliability, performance, etc. (if they are applicable). But maintainability may not be in scope.

Possible test approach:

  • Manual and tool-based test techniques are used to make sure the product meets the requirements. The team makes use of exploratory testing and test design techniques.
  • UAT is used to make sure the product solves the business need.
  • The team can practice pair programming, peer code reviews and team design reviews to prevent defects, but the team is not focused on continuous testing and code-base coverage.
  • No time investments in continuous dev support activities (such as unit tests or CI mechanism). A minimum viable test suite may be implemented.
  • A task management system is selected for short-term support (e.g. Trello might be used).
  • Defects are either fixed or rejected quickly.

Risks: All project members involved should understand unambiguously that this type of project is of release quality, but as time is the main constraint, maintainability was not the focus when designing the product, so it may not respond well to change, and further support and development of such a project might be costly.

Long-term project

Project goals: Develop a long-term reliable solution. Usually, the scope is divided into iterations, and cost might become the main constraint. Also, it is expected that the project will undergo a number of changes along the way.

Definition of quality: After each iteration, the product should meet functional and nonfunctional requirements. Also, the design, development and test approach should have maintainability and response to change as a priority, right from the start.

Possible test approach: The team focus is on continuous development, testing and support, so continuous automation testing is used when possible:

  • Automated regression test suites are created to ensure that fast feedback is available after any change is introduced.
  • Automated test suites include unit, integration, end-to-end and acceptance tests.
  • Test coverage requirements might be introduced.
  • Automated contract tests might be included in the suite as well.
  • Some nonfunctional requirements might be covered with continuous automated tests on demand (e.g. continuous performance tests, automated security checks, or Visual regression tests).
  • The team practices pair programming, peer code reviews, team design reviews, etc. to prevent defects, along with continuous automation testing.
  • Manual testing is used to ensure the product meets high-level business goals, is appealing and usable. The team makes use of exploratory testing and test design techniques.
  • Tool-based non-functional testing is used to assure product meets nonfunctional requirements, such as reliability, performance, etc. (if they are defined).
  • A long-term task management system is selected.
  • A strategy of dealing with minor bugs might be defined.

Risks: All project members involved should understand that dev support activities, building automated test suites and thorough design take more time, but only in short-term projects. Introducing changes without a continuous feedback mechanism, such as automated regression test suites, unit tests, or CI might result in waste during long-term projects.

Summary

It is crucial for a project to update the definition of quality and the test approach along with project goals and new requirement discovery. There is a variety of test types and techniques to cover different product expectations, and you should select the ones that suit your project context to make sure all stakeholder expectations are covered at any step in the project lifecycle.