A few words on bugs and software quality

From time to time I see bugs in the code and I start thinking "really? is it possible that no one noticed that bug before? am i the first person to see this code?". I thought it might be worth writing a little post on what helps me to deal with bugs and software quality in general and what are the common pitfalls in developer's thought process. Although it is not a very extensive post i hope it may inspire some developers to try new approaches :-)

In general it is extremely rare for me to commit bugs that would make it into production (maybe less than one a year? hard to say as it is really rare). To be honest I can't remember a serious issue with my code for last 5 years.

Treat bug reports as math theories needing scientific proof or negation

When I think of bugs I always have the following state of mind:

  • Computers are 100% deterministic and nothing happens by accident (at random).
  • System and underlying software works - If there is a bug it is in my own code.
  • If there is no bug I have to prove that there is none, can't reproduce is not good enough.
  • If there is a bug report in critical part of the system I never dismiss it. This is the risk no one can take.

I think my years on the math faculty caused this strictness but it works for me :) If there is a mystery i have to get to the bottom of it otherwise it keeps nagging me.

Over the years I can think of at least a few times when I thought "no, this is not a bug, it is impossible, I am sure user did something wrong" to find out later that there actually was a problem. It was mostly in first years of my career as with time my sensitivity to bugs grew. It was usually a pretty bad feeling to realize that I missed a bug as I did not investigate it well enough. If I can recommend anything it would be to keep eyes open and always assume that the reported bug is out there (it is just hiding from us).

Bugs never, ever, ever, ever fix themselves

If I ever catch myself thinking "seems like that bug must have fixed itself" i just smack myself on the face :P Yes i am sure I deserved it. Bugs never, ever, ever, ever, EVER fix themselves. I can not emphasize that strongly enough. Whenever little voice in my head starts saying this kind of stuff it is time for reality check. I do not know how to reproduce the bug, I did not manage to fix anything. I give it some more time or talk to QA / support to figure out what has really happened and how to get this issue to manifest itself. If I really, really cant solve the puzzle I never close the bug ticket, I only mark it as "cant reproduce" and usually I bounce ideas around the team to try to make some sense of it before giving up. In general i don't give up on bugs ;-)

I also never say this kind of stuff aloud: "it just started to work again.", "It has fixed itself.", "Some other change must have fixed it." as it is pretty embarrassing. So as soon as I hear this kind of voice in my head I put headphones on and dive right back into the code!

It is almost never a bug in MySQL nor PHP, it is almost certainly my code that is broken

I know it is difficult to accept the fact that our code has bugs but experience has taught me it is almost never a bug in the system nor a device, nor a browser nor a network driver nor Mysql nor PHP nor testing tools i am using.

Whenever I am having hard time finding a bug I go through the 7 steps below or ask for help someone on the team. Sometimes I need a fresh pair of eyes on the problem or just explain it to someone to suddenly see the answer.

  1. Check logs
  2. Check environment
  3. Check configuration
  4. Check data
  5. Check code
  6. Go through the code using debugger (xdebug)
  7. Yes I have not fixed anything yet, go back to step 1

In my experience (almost 10 years of building stuff) I have seen only one bug in mysql, 2 bugs in PHP and maybe 5 bugs in zend framework throughout my career. I have not seen a web app bug caused by linux nor network issue my entire life :-)

It is really important to remember that rule as statistics do not lie here. It is very very very unlikely to find a bug in the underlying codebase. Whenever I catch myself thinking "it must be a bug in PHP or Apache" just slap myself on the face and go back to step 1.

It becomes more likely to find bugs if you use stuff like dodgy frameworks, beta releases etc. Then it may be worth to validate that there is no bug report for the issue you are experiencing. In general "stable" does not mean the same thing to people and a drupal module may be more likely to have bugs if its usage stats are low. Core software used by millions of people usually has too few bugs to find them.

Random changes to code never help nor fix bugs

From time to time people to try different random changes to see if they would fix a bug they are trying to fix. As useful as it might seem to be it is usually a very bad thing to do. From my experience it is a sign of: rush, lack of understanding of the feature, lack of understanding what is broken, exhaustion or carelessness.

It happens extremely rarely but I also catch myself on doing it. Again, first comes a slap, second comes voice of reason "i wont be able to make code any better by changing stuff at random, better leave it for a while and digest the problem or try a different strategic approach".

Automate it

Automation does not help with finding new bugs? Well it does help and it helps a lot.

Building any automated tests promotes change of mindset from developer to tester. When I am writing unit tests or building some selenium or jmeter test suites i try to break stuff. When i start thinking like a tester i try to detach myself from coding and try to prove to myself that I am as good at testing as I am at coding. If I cant find the bug who can? If I wont find it will someone else be able to?

Some people say testing is boring but I think it can be quite rewarding and exciting. I think of testing phase as a hunt. I am out to find bugs and i don't stop till i find a few as there are always bugs in new code. Secondary goal is to find all bugs that are possible to find so that QA would have hard time finding anything :) I love it when QAs find bugs in my code as this is a proof they are awesome! they got my back.

The more obvious benefit of automation is that it is faster to repeat it. In case you change stuff regression testing comes easy.

Talk to someone about the issue

Whenever i get stuck on a bug or design decision I usually talk to people. There is some sort of connection between saying stuff aloud and hearing your own voice or something. Any way walking someone through an issue is usually the quickest way to get unblocked. Brain hears you say it and suddenly everything becomes clear :)

So again if I was to recommend anything is to talk to your team mates whenever you get stuck for a while.

Review code with someone

Over the years i observed a pattern that even in unit tested code you will still find small issues if you do a proper code review with someone on the team. It is usually an edge case or something small but code review in pairs seems a good way to improve quality and share knowledge.

If i was to put numbers on it i would say for ever 5 new classes (each 3-5 screens long) you will find something during a code review :)

Give it time

Finally a very simple note: testing code and finding bugs takes time. The more i rush to develop quickly the more likely i am to commit bugs. If I cant wait to release i try to allocate a fixed amount of time and test the code in this time. Even if i cant find bugs or can't see how to break it i use this time to write unit tests or some other forms of automation.

In general, healthy code is developed when you give enough time to design and test it. It varies very much from project to project but roughly it could be something like: 2/10 design, 5/10 coding, 3/10 testing. So assuming that feature is developed in one day you should probably spend another half day testing it. (i did not mention documentation, info gathering meetings etc. it is just a simple example).

Care to join a discussion on PHP, tackling bugs and ensuring quality? drop a line :)

Comments

Post new comment

Image CAPTCHA

About the author

Artur Ejsmont

Hi, my name is Artur Ejsmont,
welcome to my blog.

I am a passionate software engineer living in Sydney and working for Yahoo! Drop me a line or leave a comment.

Follow my RSS