How to prioritize your bug fixes

Any bug is a headache. On the one hand, bug fixing is expensive in terms of both time and money. It takes resources and time that could be allocated to new feature development, pushing your project back. On the other hand, not fixing bugs is risky for the development process, business and even for end-users’ lives in some cases.

It creates obstacles and technical debt. It makes users unhappy and leads to costly outcomes in fulfilling the business need. So what is the proper solution? The trick is in fixing the right stuff. You should balance between bug fixing and implementing new requirements by properly prioritizing your bug fixes.

Matt was really passionate about his project. He strived for a booming, successful first release and worked hard to achieve it. He tried to polish every detail to get the job done. He was not tolerant of any bugs left in the bug tracker at all. The workflow was simple: every bug found was assigned, fixed and tested. New features were delayed until the bugs were fixed, increasing the project time and budget.

That resulted in a release which was 2 years behind schedule. Though the quality was good, the market had moved on, and the released product was already outdated.

Understand the project context

Probably, at some point of the project lifecycle, Matt asked himself if he is moving the project in the right direction and if he really needs to fix every single bug.

If you were to solve the problem similar to Matt’s, you’d start with answering some simple questions:

• Describe the product you are building and the business needs you are trying to fulfill.
• Determine what your business environment is.
• Note whether any financial, health or property risks are involved.
• Consider how you decide what to fix next - be it ad-hoc or systematic - and figure out some criteria.
• See if those criteria correspond to the project goals and risks.

What could possibly go wrong?

If there is no prioritization process at all, you might end up in a situation where you fix every single bug with a “first in-first out” strategy. Not a good idea - it is not a data-driven decision, and the business need and risks are not considered.

Another thing that could go wrong is trying to fix every bug and delay a release until there is nothing left reported. Not a good idea either, as it does not consider cost efficiency and market demand changes.

One more thing that usually goes wrong - sole decision making, which is prone to the Dunning–Kruger effect (a cognitive bias when you have the illusion that you know everything[1]) and scores low on bus factor (a measurement of risk resulting from information not being shared with team members[2]). It also creates a bottleneck if a person making decisions has other responsibilities and the scope is quite large. The last one is not necessarily a bad approach, but better to have it supported by data and be prepared for the risks.

Where prioritization stands in your bug lifecycle

To have a better visualization of what we are trying to improve, imagine your bug lifecycle. Here is a simplified workflow:

1. find a bug
2. report a bug
3. decide when to fix it
4. fix
5. test
6. deploy the fix

In most iterative SDLCs, the decision on when to fix it is made several times. Here is the same workflow, a bit more in detail:

1. find a bug   
2. report a bug  
3. decide if it can wait or whether it should be fixed as soon as possible
4. reproduce
5. decide when to fix it in the long term
6. reproduce
7. plan when to fix it in iteration
8. fix, test, deploy

So, you should actually prioritize a few times when you analyze what implications the bug creates in the long and short term.

Items in your prioritization list

You should understand that a bug report is not necessarily a ticket in your bug or task tracker. It can take multiple forms, depending on the project culture. Here are some of them:

• ticket in help desk
• ticket in bug tracker
• email from client / business team / product owner
• direct message in a chat to some developer
• notes taken during pair testing or any kind of code review
• issues mentioned in a meeting / conference
• issues noted during demo or alpha / beta testing
• a business person coming to the developer’s desk and asking them to fix something as soon as possible
• QA saying “we have a bug here” out loud in a room

Sometimes the initial decision should be made fast, without waiting in the queue for prioritization by a single person, e.g. if the case will have a severe impact on the business. Also, the team should know when to say ‘no’ to unscheduled tasks that can wait and be allowed to make such decisions.

So, with enough data, the whole team should be able to make a good short-term decision on a priority.

Making the initial decision

You should provide your team with enough data to decide if it can wait or whether it should be brought up immediately and fixed as soon as possible. That way, if you are on vacation, they are not blocked.

Here are some points that should be clear and accessible for everyone in the team:

From the business side

• current long-term and short-term business goals  
• sales team goals (any upcoming activities sales team might need help with)

From the customer or end user side

• company approach to customer support (do we promise to fulfill their needs in some specific time, do we have an agreement on bug fixing, etc.)
• customer experience (are some clients facing more issues than others?)
• privileged clients (maybe some customers are more valuable)

From the team and project side

• project risks
• anyone or anything blocked by the issue

From the domain side

• legal regulations
• publisher requirements
• product risks (e.g. life, health, financial, property risks)

Anyone from the team should be able to ask themselves to make an initial decision: if I refuse to fix it right away, how greatly would our business, end users or team be affected? If we leave it as it is for awhile, would there be severe implications?

Select a way to share this information that suits you. For some points, it can be a wiki that is up to date and available for everyone. For other points, it can take a form of a meeting, e.g. a daily status meeting to see if anyone from the team is blocked . It can be a project plan that is available to everyone involved in the project, like a dashboard in the bug tracker with a specific data set.

Explain to your business and support teams that it is not possible to fix every issue right away as it affects the schedule. Come to an agreement that the team has a right to say ‘no’ to unscheduled requests that don’t have severe implications.

When the initial decision is made by you or the team, the bug goes to a shortlist or to a pool.

Prioritization of the bug pool

Someone has to keep an eye on the entire list of known issues, even those items on the back burner. The list should be reviewed and prioritized regularly.

Review and prioritization should be scheduled activities. Make sure you allocate time in the project for this officially, otherwise you might end up with delays in the review process or other tasks assigned to the reviewers.

The review and prioritization of a bug report usually consists of the following:

1. read the bug report and understand the context
2. reproduce
3. update with more data if required
4. analyze implications and risks
5. categorize
6. prioritize
7. take action (assign, reject, close, etc.)

The process can be performed by one person or a triage committee, but in most cases there are usually a few people involved.

Reproduction and information updates are performed before the categorization process and other team members can be responsible for this, e.g. a QA person.

Risk analysis can take multiple forms, but the process is common with other project risk analysis techniques.

Categorization

Define a taxonomy for your bugs. You might want to add a few new fields to your bug report form to categorize items better. Here are some ideas of new fields to add. They will help you  assess risks and find areas to improve:

• product / sub-system / service where the bug was encountered
• feature
• attribute of quality affected (stability, performance, usability, etc.)
• module where the error was introduced
• project stage where the error was introduced (requirements, design, etc.)
• type of defect (e.g. race condition, error handling, etc.)
• root cause (e.g. miscommunication, lack of training, outdated documentation, etc.)

Also, choose attributes by which you will make decisions on priority. There are a few approaches. Some suggest prioritization by severity, frequency and visibility, others - by severity and likelihood. You can choose attributes that matter to you. Here are a few examples:

• severity = blocker, critical, major, minor, trivial
• frequency = occurrence
• visibility = how likely will it catch the eye of an end user
• impact = high risk areas, e.g. authentication process, authorization, payment systems
• effort = how much team effort will it take to fix that issue

Prioritization

The process is quite simple: we take a bug, fill in the bug categories, fill in the attributes we chose for prioritization - and then, based on the values, define the priority. We can use all parameters to make a decision, or a few of them. The following is just an example, and it varies from project to project. Define what suits you by yourself.

In this example, it’s decided to categorize the issues by severity and frequency.

Work format

Meeting vs. review. Depending on the format of communication, it can be a meeting or an asynchronous reviewing process. There are pros and cons for each approach. The meeting requires more coordination effort, while the reviewing process requires discipline, continuous reminders and set deadlines.

Single person vs. committee. A decision that was made single-handedly might not always be correct. One’s biases might affect the prioritization process greatly, so in some cases, actions can be taken to share responsibility and minimize the human factor.

Depending on the number of people involved, it can be a single person decision, or a 2-3 committee decision. If one person is to decide, human factor is involved, whereas a committee decision is more objective, but it takes more time.

Select the format that suits your project best and don’t forget that the key factor here is consistency. Plan this activity ahead and include it in the project plan using automatic reminders to help enforce the activity.

Prioritization in a single iteration

When choosing which bugs to fix in the next iteration, it’s obvious we select high priority ones first - but that’s not the only item to consider.

We should also select items that don’t depend on each other or at least assign counterdependent issues to the same person to avoid conflicts, while considering the required effort and doing items of the highest risk first to test them quickly.

Summary

In his next project, Matt shared prioritization responsibility with the team. He kept everyone updated on the project goals and risks and gave the team a right to say ‘no’ to unreasonable changes. He improved the bug categorization process and defined priority selection criteria based on attributes. He scheduled regular prioritization meetings and involved a cross-functional triage committee in the process. He also prioritized bugs in the same iteration. With a well-defined bug prioritization process, Matt and his team had more time to focus on the items that were driving the project forward and avoided bottlenecks related to bug fixing activities.

The prioritization process is a time-consuming activity, but doing it properly allows you to save valuable project time by fixing the right stuff while avoiding bottlenecks.