Tuesday, November 15, 2011

Justifying Unit Testing & TDD

The cost benefits of Unit Testing is a interesting topic to discuss. Anyone who has experience with it or knowledge of it at least, can easily see over time it will reduce costs and increase quality. However selling it to business people always seems difficult. I often liken it to working in a factory, imagine an senior engineer informs management that with a little investment he has an idea that will significantly reduce costs, time-lines and increase quality. He can't exactly quantify, but has seen it work when he worked for a competitor. Most business people I have worked with, always want a dollar figure or a number of hours saved per year. This is almost impossible to estimate with any accuracy, and you get held to it if things look like they are going wrong.

IEEE did a study about the cost of defects at each phase of the software development lifecycle, attaching the graph they published (CostofCorrectingDefects). An online version is available here.

Quote from a study called "Realizing quality improvement through test driven development: results and experiences of four industrial teams". The full article can be found here.

“TDD seems to be applicable in various domains and can significantly reduce the defect density of developed software without significant productivity reduction of the development team. Additionally, since an important aspect of TDD is the creation of test assets—unit, functional, and integration tests. Future releases of these products, as they continue using TDD, will also experience low defect densities due to the use of these test assets”

“Convince the development team to add new tests every time a problem is found, no matter when the problem is found. By doing so, the unit test suites improve during the development and test phases”

“Get the test team involved and knowledgeable about the TDD approach. The test team should not accept new development release if the unit tests are failing”

There's also a good chapter in Roy Osherove's book "The Art of Unit Testing" (the defacto bible on unit testing). He gives an example of a project he worked on that shows two teams developing similar features with the same sized team and similar skill levels:

Stage Team without tests Team with tests
Implementation (coding) 7 days 14 days
Integration7 days2 days
Testing and bug fixingTesting 3 days
Fixing 3 days
Testing 3 days
Fixing 2 days
Testing 1 day
Total: 12 days
Testing 3 days
Fixing 1 day
Testing 1 day
Fixing 1 day
Testing 1 day
Total: 8 days
Overall release time26 days24 days
Bugs found in production7111

Here's a similar story of why unit testing helps teams who use it save time and money versus those that don't: http://pathfindersoftware.com/2009/04/unit-testing-can-you-afford-not-to.

I’d say all software development teams know that unit testing is the most effective way to find bugs. According to IEEE the cost to correct a bug in the coding stage is $977 whereas $7136 in the Testing stage. That is why unit testing which is in the coding stage is so important. For Agile developments this is even more so as the code and design is changing so fast that without unit tests the whole building can collapse and many large scale agile groups see their velocity slow down: Story.