TypeMock Isolator war stories

Like many other developers out there, most of the time we are working with some form of legacy code that when written was not designed to be tested.  Changing existing code to be unit testable can be hard particularly if there really wasn't much care taken with the design originally, or there's been years of hacky maintenance of the code.  I was always curious to give Typemock's Isolator a try to see if it helped making testing of legacy code easier or quicker.

Background

I had two completely different projects in mind.  One a legacy project that is around 7 years old and has been submitted to constant time constrained projects that led to bad coding practices.  This project is essentially a standard 3 tier architecture, although I focus on the WCF / Remoting service layer and below.  I'll refer to this project as the legacy project. 

The other is purely a green fields project.  It is a standalone Windows executable that has been designed and built recently and makes extensive use of interfaces, abstract classes, and virtual methods. I'll refer to this project as the modern project.

I'll try to compare these two distinct scenarios to test if Typemock's Isolator is best suited to one or the other. 

What is Typemock Isolator

By now most modern developers have had some exposure to unit testing, so I won't go into definitions or how-to's here.  There are many unit testing frameworks and utilities that help in unit testing.  Firstly you'll need a unit test runner (nUnit, MsTest, xUnit etc). In all but the most basic of unit tests you will also need a mocking framework (RhinoMock, Moq, NMock, and now with Visual Studio 2012 Microsoft Fakes). Isolator falls into this category and comes packages with other utilities that integrate into Visual Studio to assist in unit testing.  

Immediate Differences on the Surface

Ease of use is gauged simply as: easy, medium, hard

	Isolator Basic Edition	Isolator Essential Edition	RhinoMocks	Moq	Microsoft Fakes
Client License Price	Free	US$799	Free	Free	Requires Visual Studio 2012Ultimate
Server License Price	Free	US$2499	Free	Free	Free
Ease of use (imho)	Medium	Medium	Easy-Medium *1	Easy	Hard *2
Mock Interfaces/Abstracts/Virtuals	Yes	Yes	Yes	Yes	Yes
Mock Concrete classes	No	Yes	No	No	Yes
Mock Static usage	No	Yes	No	No	Yes

*1 - There are alternative usage styles in RhinoMocks. The simple way works most of the time, but sometimes you may need to dig deeper to write more advanced tests. Using the more advanced features does increase the difficulty. (This is based on feedback I get from other devs I work with; I think its a good trade-off for a feature rich framework).
*2 - This may be because my exposure to it is a little limited, however the syntax does feel a little awkward and verbose.

Biggest difference is the price.  Being able to exercise .NET black magic and mock the unmockable doesn't come for free.

Working with the Legacy Project

Legacy code never has just one issue, there are always multiple nasty problems intertwined.  Don't be naive and think that writing Isolator is a magic bullet that will make it easy. It won't. It does make it easier for sure. But it won't help you much with issues like badly designed methods and classes with thousands of lines of code.

Have a look at this for examples of working with Legacy code tests using Isolator.

The temptation is to write tests that ensure every interaction which normally results in brittle tests.  As soon as you refactor to a better design the method will be doing less.  Its normally more effective to focus on use cases for a class rather than focusing on the legacy methods.

So is this an effective strategy on legacy code? Is it good bang for buck - time spent versus quality and robustness of tests going forward?  Can the same be achieved by integration testing with SOAPUI or Selenium for example?

My experience has been you seldom come across legacy code that is well design, but just doesn't have tests. Writing good tests involves focusing on behaviour not tests line by line a method.  This is very difficult with legacy code as the Single Responsibility Principle has most likely been violated as well.  Behaviours will be spread across multiple classes in a not necessarily logical manner. So writing tests straight from the legacy code is a bad idea, it should be refactored as well. The best way to do this is use integration tests (either coded or using SoapUI, Selenium etc) and then refactor the internal code before writing focused unit tests.

Working with the Modern Project

So if you can afford it, does it mean that writing straight-forward code that does not use interfaces, abstracts, or virtuals is no longer necessary? No. No. No. Good design principals still apply.  Just because unit testing is the first consumer of abstractions and extension points in your code does not mean there will be no other future consumers.  Maintaining code that has been SOLID-ly designed is proven to be easier (easier == cheaper) over and over.

So given that you want to write good loosely coupled SOLID code, do you need Isolator?  No, its like say do you need to take vitamins with a balanced diet.  However it still can help.  Isolator really shines when you need to test code that you have little control over, like where your code interfaces into a third party API.  Also there might be times when using its ability to mock Static method calls provides a cleaner way to test something, rather than writing wrappers and interfaces.

Utilities

Do all the fancy shiney utilities and helpers really matter?  Well, it depends.  I personally do not like too many utilities creating noise in my IDE.  But I can see how the Isolator utilities can guide your test design, ensuring your tests are focused by giving a warning if more than one assert is used etc.

Conclusion

If good design principals and cheap maintenance and extension (read as low cost of ownership) are important to you, then abandoning SOLID techniques is a bad idea. That said, you can still use Isolator and write well designed SOLID code.

For modern projects if you are lucky to have Visual Studio Ultimate, no brainer, you don't need Isolator. If you don't have Ultimate its a slightly more difficult decision: It does seem hard to justify spending the money when the other open-source frameworks will definitely serve you well.  The utilities in my humble opinion are nice to haves; these are not compelling enough to drive me to spend the money.  The Isolator utilities do not enforce anyone to follow their advise so there are no guarantees. My opinion is that because Isolator would be difficult to be abstracted away from or switched out once you have a significant number of tests, I don't like the idea of being locked in when I don't feel a compelling need to use it. How about the free edition of Isolator? To be honest, (subjective warning) I prefer a more fluent style like Rhino or Moq. You may also be at the mercy of features being pulled into the paid versions.  Again, this is just my personal preference, not really based on scientific reasons.

Using it to test UI code (ie controllers and code behind) has no benefit in my opinion. This code will change often and you're tests will begin to test ASP.NET and other third parties that are out of scope for your test. UI code always has high churn, so testing individual components starts to offer low bang for buck.  I would prefer integration style testing of UI's (like SpecFlow or Selenium).

For legacy projects, again if you're lucky to have Visual Studio Ultimate, it will be hard to justify this expense.  Otherwise, I strongly believe that if you are refactoring code you should refactor it to a better design including interfaces etc.   However, introducing interfaces and proper design will take time. Legacy code is never simple to cover in tests, Isolator will definitely help you and has some nice features that will help show progress to stakeholders. The counter argument is that integration style testing with tools like Selenium and SoapUI might serve better less brittle tests.  I believe both approaches are important, integration tests ensure behaviour, unit tests ensure good code design and micro-behaviour that is focused.

One final thing:  Using Isolator (or Ms Fakes) to test and validate third party API's is quite effective.  This allows you to keep third party vendors honest that when they make claims about backwards compatibility etc.