Wasteful Activity #2: Playing with defects

What’s a defect? I like this definition.

A defect is anything that threatens the value of the product.”

Before we start, let’s agree that:

  1. we don’t want to have defects that threaten the value of our product
  2. we want to give our customers as much value as possible at all times.

If you don’t agree with 1 and 2 then, don’t waste your time and stop reading now.

Defects are commonly called Bugs
Software defects aka Bugs

Testers are normally associated with finding defects. Some testers get very protective with the defects they find and some developers can be very defensive about the defects they wrote. Customers don’t like defects, developers don’t like defects, product managers don’t like defects, let’s be honest, nobody likes defects besides some testers.

Why would that be? The reason is that the focus of testers is on detecting defects, and that’s what they get paid for in most organisations. If you are a tester and love your defects, you might find this article quite distressing, if you decide to proceed, do so at your own peril.

Let’s be clear from the start: defects are waste. Waste of time in designing defective products, waste of time in coding defective routines, waste of time of detecting them, waste of time in fixing them, waste of time in re-checking them. Even writing this sentence took a good while, now think how much time it takes you to produce, detect, fix, recheck defects.

Our industry has developed a defect coping mechanism that we call defect management. It is based on a workflow of detecting => fixing => retesting. Throughout the years it has become best practice (sic) to have defect management tools and to log and track defects. Defect management approaches are generally cumbersome, slow, costly and tend to piss people off no matter whether you are a tester that gets his defect rejected, you are a developer that gets a by design feature flagged as defect or you are a product manager that needs to spend time prioritising, charting and trending waste.

Another bad characteristic of defects is that they can be easily counted and you will always find a pointy haired manager that decides he is going to shed light on the health of his product and on the efficiency of his test team by counting and drawing colourful waste charts.

But, before we said that defects are waste, why are we logging and tracking waste, creating waste charts seems even more ridiculous, wouldn’t it be easier to try to prevent them?

Oh, if only we could write the right thing first and reduce the amount of defects! I say we can, be patient and read on.

Software development teams have found many ways of being creative playing with defects, see some examples below.

Example 1: Reward waste

Reward for the wrong reason
Reward for the wrong reason

Some years back i was working on a business critical project in one of 5 scrum teams . Let me clarify first, that our scrum implementation was at best poor, we didn’t release every sprint and our definition of done was questionable.

Close to an important release we found ourselves in a situation where we needed to fix a lot of defects before going into production. We had 2 weeks and our teams had collectively around 100 defects to go through. Our CTO was very supportive of the defect killing initiative and he was eager to deliver with zero defects. He put in place a plan that included free food all day and night and some pampering for the developers that needed to focus 100% on defect resolution. Then he decided to give a prize to the team that would fix the highest amount of defects.

I remember feeling frightened of the possible future consequences of this reward. I spoke to the CTO and told him that I would have liked more a prize for the team that produced the lowest amount of defects rather than the one that fixed the most. Our CTO was a smart guy and understood the value proposition of my objection, he changed his approach and spoke to the teams on how not introducing defects in the first place is much more efficient than fixing them after they have been coded. Soon after the release we started applying an approach that focussed on preventing defects rather than fixating on detection. We never had the problem of fixing 100 bugs in 2 weeks again.

A typical defect prioritization meeting
A typical defect prioritization meeting

Example 2: Defects metrics

In my previous waterfall life, I remember when management introduced a performance metric directly linked to bugs. Testers were to be judged on the Defect Detection Index calculated as (Number of Defects detected during testing / Total number of Defects detected including production)*100. An index lower than 90 would mean nobody in the test team would get a bonus. Developers were individually judged on number of code defects found by the testers and business analysts were individually judged on number of requirement defects found by the testers.

The bug prioritisation meetings were battles where development managers argued any bug was a missed requirement, product managers argued that every bug was a coding error or a tester misunderstanding and the test lead (me) was simply shouted at and criticised for allowing his testers to go beyond the requirements and make use of their intellectual functions outside a scripted validation routine.

Going to that meeting was a nightmare, people completely forgot about customers and simply wanted to get their metrics right. The amount of time we wasted arguing and defending our bonuses was astonishing. Our customers were normally unhappy because instead of focusing on value delivery we focussed on playing with defects, what a bunch of losers we were!

Example 3: Defects as non conformance to requirements

Let the nitpicking season start
Let the nitpicking season start

In the same environment as Example 2 testers, in order to keep their Defect Detection Index high used to raise large amounts of minor or non significant “defects” that were in reality non conformance to requirements that generally represented improvements. Testers didn’t care if they were requirements, code defects or even improvements, to them they were money, so they raised them. Improvements were filed as defects as they were in non conformance to requirements. In most of the cases these were considered low severity and hence low priority defects to make the testers happy and had to be filed, reviewed, prioritised and used in trends, metrics and other useless calculations. This activity could easily take 30% of the tester time. Such defects would not only take testers’s time, but would affect developers, product managers, business analysts and eventually clutter the defect management tool. Waste that creates waste, exponentially, how wonderful.

A colourful dump
A colourful dump

 Example 4: Defect charts, trends and other utter nonsense

Every week I had to prepare defect charts for management. These were extracted from our monstrous defect management tool and presented in brightly coloured useless charts. My manager got so excited at the prospect of producing useless information that she started a pet project to create charts that were more colourful than the ones I presented. She used 2 developers for 6 weeks to create this thing that was meant to wow the senior executives. In the process of defining the requirements for wowing the big guys she introduced a few new even more useless charts and consolidated it into an aggregating dashboard. She called it the product quality health dashboard, I secretly called it the dump. Nobody gave a damn about the dashboard, nobody used the numbers for any reason, nobody cared that they could configure it, but my boss was extremely proud of it. A legend says that she got a big raise because of it. If you play with rubbish, then you will start measuring rubbish and eventually you will end up doing data analysis and showing a consolidated view of the rubbish you store in your code.

How can we avoid this?

1. Focus on defect prevention

Many development teams focus on delivering features fast with little consideration for defect prevention. The theory is that testers (whose time is sometimes less expensive than developers) will find the defects that will be fixed later. This approach represents a false economy; rework disrupts developers activities and harms the flow of value being delivered. There are many approaches available to development teams to reduce the amount of rework needed.

Do you want to prevent defects? You can try any combination of the below:

  1. With BDD/ATDD/Specification By example, delivery teams test product owners assumptions through conversations and produce the right feature the first time.
  2. The ability to have fast feedback loops also allows for early removal of defects, automated unit and integration testing can help developers quickly identify potential issues and remove them before they get embedded into a feature.
  3. Tight collaboration between business and delivery teams helps teams be aligned with their real business goal and reduce the amount of unnecessary features. This means less code and as a consequence less defects. Because, your best piece of code is the one you won’t have to write.
  4. Reducing complexity is very powerful in preventing defects, if we are able to break down a complex problem in many simple problems we are likely to reduce the amount of defects we introduce. Simple problems have simple solutions and simple solutions have less defects than complex ones.
  5. Good coding standards like for example limiting the length of a method to a low number of lines, setting limits on cyclomatic complexity, applying good naming conventions to help readability also have a positive impact on the number of defects produced
  6. Code reviews and pair programming greatly help reduce defects
  7. Refactoring at all times also reduces defects in the long run

Moral of the story: If you don’t write defects, you will not have to fix them.

2. Fix defects immediately and burn defect management tools

If like me years back, you are getting tired of filing, categorising, discussing, reporting, ordering defects I have a very quick solution. Fix the defects as soon as you find them.

It is normal for a developer to fix a defect he finds in the code he is writing as soon as he finds it without having to log it, but as soon as the defect is found by a different individual (a tester for example) then apparently we need to start a strict logging process. Why? No idea really. People sometimes say: “if you don’t do root cause analysis you don’t know what you are doing, hence you need to file the defects”, but in reality nobody stops you from doing root cause analysis when you find the defect if you really want. What I am suggesting is that whoever finds a bug walks to a developer responsible for the code and has a conversation. The consequence of that conversation (that in some cases can involve also a product owner) should be let’s fix it now or let’s forget about it forever. Fixing it now, means normally that the developer is fresh on the specific code that needs to be fixed, certainly fresher than in 4 weeks when he won’t even remember he ever wrote that code. Fixing it now means that the issue is gone and we don’t have to worry about it any longer, our customer will be thankful. Forgetting about it forever means that it is not an issue worth fixing, probably it doesn’t threaten the value of the project and the customer won’t care if we don’t fix it. Forgetting about it forever also means that we won’t carry a stinky dead fish in a defect management tool. We won’t have to waste time re-discussing the same dead fish forever in the future and our customers are happy we are not wasting time but working on new features. If you decide to fix it, I’d also recommend you write an automatic test around it, this will make sure that if the issue happens again you’ll know straight away.

I have encountered huge scepticism when suggesting to burn defect management tools and fix just in time. Nobody seems to think this is possible. As a matter of fact my teams were able to do this for the last 5 years and nobody ever said, “I miss Jira and the beautiful bugs”.

Obviously this approach is better suited for co located development teams, I haven’t tried it yet with a geographically distributed team, I suggest you give it a try and let me know how it goes.

Playing with defects waste index:

defects

Epidemic: 90% – The only places that don’t file and manage defects I have ever encountered are the places where I have worked and have changed the process. I have heard of one other place where they do something similar but that’s just about it. The world seems to have a great time in wasting money filing, categorising, reporting, trending waste. The IT industry has been doing this for decades and questioning the approach is generally seen as heresy.

Damaging: 100% – Using defects for people appraisal is one of the worst practices I have ever experienced, the damage can be immense. The customer becomes irrelevant and people focus on gaming the system to their benefit. Logging and managing defects is extremely wasteful as well, it requires time, energy and can among other things, endanger relationships between testers and developers. Trending and deducting release dates from defect density is plain idiotic, when with a little attention to defect prevention defects would be so rare that trends would not exist.

Resistant: 90% – I had to leave one company because I dared doubt the defect management gospel and like an heretic I was virtually burned at the stake. In the second company I tried to remove defect management tools I was successful after 2 years of trying, quite resistant. The third one is the one where people were happy to experiment and as soon as they saw how much waste we were removing it quickly became the new rule. I have had numerous discussions with people on the subject and the general position is that defect management must be done through a tool and following a rigid process. I believe it is a very difficult wasteful practice to eradicate.

Recommended Reading

Lean Software development – An Agile Toolkit (Mary and Tom Poppendieck)

https://mysoftwarequality.wordpress.com/2015/05/06/little-tim-and-the-messy-house/

https://mysoftwarequality.wordpress.com/2013/09/10/how-i-stopped-logging-bugs-and-started-living-happy/

Advertisements

Wasteful Activity #1: Product roadmaps

What’s a product roadmap? After a quick search on the web I found what I believe is a good definition.

“A product roadmap is a tool that provides a strategic guidance to team members, business partners and customers. Just as a map of a city helps you reach your destination, product roadmaps provide organizations with a plan to shape and define their product’s vision.” Source OneDesk

A Product roadmap is not bad in itself, but abusing it’s meaning and usage can create massive amount of waste and sometimes kill the product it is meant to describe. I am going to list 2 concrete examples of how abusing a product roadmap can kill a product.

Example 1: Static product roadmaps

If you think about a real roadmap, you think about something that changes with low frequency, in fact we don’t build new roads every day. Comparing today’s roadmap of Rome with one that is 10 years old we probably would find some inconsistencies due to some new roads, extra buildings, etc. Using the 10 year old map today might be difficult but very likely we would still be able to get around. On the other hand if we use a 2000 years old roadmap of Rome today, we would be definitely in trouble. Besides being able to go to the Roman Forums that still exist today, we would have very little use for it and we would get lost for sure.

Old Rome roadmap
Old roadmap of Rome (not 2000 years old)

Product roadmaps work more or less the same way, with the difference that the frequency of change in features (rather than roads and buildings) is much higher. In my experience I can compare what changed in the roadmap of Rome in 2000 years to what can change on a product roadmap in one single year. If we are aware of this, then we understand that creating a static product roadmap and sticking to it is a waste of time. Why create something that we know very soon won’t reflect reality and use it as a map for our day to day real world decisions? Would you use Rome imperial map for your next trip to the Eternal City? I bet you wouldn’t, but yet most of us accept that we need to use a product roadmap for our day to day prioritisation decisions.

Modern Roadmap of Rome
Modern Roadmap of Rome

The waste with static product roadmaps is twofold. Not only we waste time creating a roadmap, but in some cases we waste huge amount of time and money in defining the various steps of the roadmap in advance. I’m sure you heard things like “Phase 1 is still in progress, but we must start writing the requirements and specifications for phase 3 of project XYZ!” This is what I like to call meta waste, i.e. wasteful activity built on wasteful artifact.

Example 2: Dumb Unaware product roadmaps

Roadmaps sometimes become less  effective when we are trying to go somewhere, we choose the shortest route and we find out that minutes ago an accident caused bumper to bumper traffic. If we knew that traffic was so bad on that route, we would have picked the longer route with no traffic, but we didn’t know. Now we have more information than when we started the trip and can make the decision of reversing and choosing the longest but fastest route. Simple, right?

Believe it or not, I have seen product roadmaps misused to the point that reality is ignored and no matter how bad the traffic, the product direction doesn’t change. In one occasion, I worked on a product, designed to make lending money to customers easier, that started at the height of the housing boom. When in 2008 the economic crisis hit Ireland hard, we had 2 more years product roadmap wastefully laid out and meta wastefully designed in detail in front of us. Management decided that evaluating the new economic outlook and redrawing a new product strategy was not the way to go and stuck to the reassuring product roadmap. I let you guess the consequences on the product and on the company.

The above is an extreme example, but less dramatic examples happen continuously generating incredible amounts of meta wasteful work. Product roadmaps are based on people’s assumptions at a certain time. When we don’t use customers feedback to validate or disprove our assumptions we are running blindfolded carrying a pair of sharp scissors. Sooner or later we are going to have a problem, maybe fall off a cliff and when (assuming we are lucky and still alive) we take the blindfolds off we are surprised by how we got where we are.

 

How can we avoid this?

1. Shorten your roadmap

As mentioned above, product roadmaps are not bad per se, the problems lie in their usage. If you have to work and live with a product roadmap, I suggest you avoid as much as possible to spend time working, refining, grooming the items that are scheduled far in the future. Ideally you are able to plan Just In Time, but if Just In Time doesn’t work for you, try to reduce the future work as much as possible. As a rule of thumb, I would never do any work for something scheduled to happen 2 or more months from now. Eliminate all the meetings that get called on products and features outside the next 2 months. If people complain point them to this blog post to explain your reasons. The great thing is that the world won’t implode, you won’t get fired, your company won’t fail and actually you will not notice any difference, with the added benefit that you have avoided wasteful work for yourself and your team. Try 2 months to start, if you are already doing 2 months try to shorten down to 1 month. I like to work with a maximum of 2 weeks work in the future, the world never imploded for me either.

2. Quantify Waste and compare to missed opportunities

If you are dealing with management that want to follow the product roadmap no matter what and force people to work months in advance you can try by explaining the reasons in this post first, if they don’t budge try the following: when working on something that according to the product roadmap will happen 3 or more months in the future, log all the time you and your team spend on it. When the item gets cancelled or de-prioritised into oblivion, send your boss all the artefacts that your team have created (diagrams, specs, requirements, wikis, code, etc) plus the report of the time your team has spent on that item. Using a feature you are currently releasing, map how earlier it could have been delivered if your efforts were focused on it instead than on the deceased item. Example: “If we didn’t have to work on feature X that we eventually will never do, we could have released feature Y last week instead of next week and we could be realising the profits of it already”.

3. Always be aware

Your ability to adapt and react will depend on you being available to work on changing needs, customer feedback, competitors moves and economic changing conditions.

Constantly validate assumptions by using customers feedback, practice deliberate discovery to reduce guessing. Listen to Eric Ries “Build Measure Learn”. Deviate from the product roadmap before falling off the cliff.

When you become good at this  you will be able to use customer feedback, moving targets, changing conditions and requirements as your real time always up to date roadmap to successful products.

P.S. Please don’t confuse a product roadmap with a vision. A vision is necessary, a vision is a point of arrival, not a roadmap.

Product roadmap waste index:

Waste Index
Waste Index

Epidemic: 80% – Product roadmaps are quite widespread and most of the organization I have witnessed display some level of roadmap abuse. People find comfort in having a roadmap to follow, this make the product roadmap very popular.

Damaging: 95% – Creating product roadmaps is a wasteful activity but it doesn’t take in itself a lot of time. The problem starts when large amount of work are done on parts of the roadmap far in the future. Chances are that either that work is  never used because the company changes priority or in the worst case scenario, the work is completed regardless of the changing conditions.

Resistant: 65% – Product roadmaps are considered by some schools of thought as best practice hence difficult to dismantle from that point of view. On the other hand, explaining how they can be damaging is a simple task and if an organization is willing to learn, it will lose the roadmaps quite easily.

Recommended reading:

Lean Software development – An Agile Toolkit (Mary and Tom Poppendieck)

The lean Startup (Eric Ries)