WCF Service Contract Code Generator

http://mws.codeplex.com/

MoQ - Running tests in Parallel

It may not be accurate to say that MoQ (Quick start intro) is thread safe when running tests in parallel because it appears it is not in other circumstances.  Check out this blog post where they modified the source to fix a threading issue.

However, it appears that MoQ avoids the issue I described in the previous post about how Rhino falls over intermittently when running tests in parallel.  MoQ avoids this by design.  Here's the modified code from  Test1 converted to MoQ:

    [TestClass]
    public class Test1
    {
        [TestMethod]
        public void Test1a()
        {
            var dependency1 = new Mock<IDependency1>();
            var dependency2 = new Mock<IDependency2>();

            dependency1.Setup(m => m.DependentMethod("System.String")).Returns(false);

            dependency2.Setup(m => m.Kind).Returns(TypeKind.Foo);
            dependency2.Setup(m => m.StringProperty).Returns("System.String");

            var target = new TestSubject(dependency1.Object, dependency2.Object);
            Assert.IsFalse(target.SubjectMethod());
            dependency1.VerifyAll();
            dependency2.VerifyAll();
        }

        [TestMethod]
        public void Test1b()
        {
            var dependency1 = new Mock<IDependency1>();
            var dependency2 = new Mock<IDependency2>();

            dependency1.Setup(m => m.DependentMethod("System.String")).Returns(true);

            dependency2.Setup(m => m.Kind).Returns(TypeKind.Foo);
            dependency2.Setup(m => m.StringProperty).Returns("System.String");

            var target = new TestSubject(dependency1.Object, dependency2.Object);
            Assert.IsTrue(target.SubjectMethod());
            dependency1.VerifyAll();
            dependency2.VerifyAll();
        }

   }

These tests pass every time.

Even the updated brute force parallel tests runs ok.

    [TestClass]
    public class Test2
    {
        [TestMethod]
        public void BruteForceTest()
        {
            var threads = new Thread[30];
            for (int i = 0; i < threads.GetUpperBound(0); i++)
            {
                threads[i] = new Thread(this.ThreadAction);
                threads[i].Start();
            }

            for (int i = 0; i < threads.GetUpperBound(0); i++)
            {
                threads[i].Join();
            }
        }

        private void ThreadAction()
        {
            var dependency1 = new Mock<IDependency1>();
            var dependency2 = new Mock<IDependency2>();

            dependency1.Setup(m => m.DependentMethod("System.String")).Returns(false);

            dependency2.Setup(m => m.Kind).Returns(TypeKind.Foo);
            dependency2.Setup(m => m.StringProperty).Returns("System.String");

            var target = new TestSubject(dependency1.Object, dependency2.Object);
            target.SubjectMethod();
        }
    }

Rhino Mock - Running tests in parallel - not thread safe

I came across this issue while diagnosing why some unit tests intermittently fail.  It appears to be that Rhino Mock (v3.6.1) is not thread safe.

There does appear to be some chatter on this topic when you google it.  Someone has even customised their own version of the Rhino code to be thread safe here.

First off, hats off to Ayende and crew for creating Rhino mocks its a fantastic premium mocking library that I have enjoyed using.  Back when it was written running tests in parallel was fringe at best.

My environment:

• MsTest using a .TestSettings file set to run two tests in parallel.
• Visual Studio 2012 Premium (not that I think this is relevant).
• 8 core processor.
• I'm using the Visual Studio Test Runner to run the tests.

The Code:

Full test code that exposes the issue can be downloaded here.

In Test1 you can see both test methods look fine. 

    [TestClass]
    public class Test1
    {
        [TestMethod]
        public void IsTrivialAssociationShouldReturnFalse()
        {
            var dependency1 = MockRepository.GenerateStub<IDependency1>();
            var dependency2 = MockRepository.GenerateStub<IDependency2>();

            dependency1.Stub(m => m.DependentMethod("System.String")).Return(false);

            dependency2.Stub(m => m.Kind).Return(TypeKind.Foo);
            dependency2.Stub(m => m.StringProperty).Return("System.String");

            var target = new TestSubject(dependency1, dependency2);
            Assert.IsFalse(target.SubjectMethod());
        }

        [TestMethod]
        public void IsTrivialAssociationShouldReturnTrue()
        {
            var dependency1 = MockRepository.GenerateStub<IDependency1>();
            var dependency2 = MockRepository.GenerateStub<IDependency2>();

            dependency1.Stub(m => m.DependentMethod("System.String")).Return(true);

            dependency2.Stub(m => m.Kind).Return(TypeKind.Foo);
            dependency2.Stub(m => m.StringProperty).Return("System.String");

            var target = new TestSubject(dependency1, dependency2);
            Assert.IsTrue(target.SubjectMethod());
        }
    }

The problem only seems to manifest when using the same interface type multiple times in the same test class.
When this runs either one of these tests will intermittently fail given the test settings I am using (running 2 or more in parallel).

You'll either get an Assert failure even though clearly it should not.

A few times I got another weird "System.InvalidOperationException: This action is invalid when the mock object is in replay state."

Its more apparent when you throw many threads at it:

    [TestClass]
    public class Test2
    {
        [TestMethod]
        public void BruteForceTest()
        {
            var threads = new Thread[30];
            for (int i = 0; i < threads.GetUpperBound(0); i++)
            {
                threads[i] = new Thread(this.ThreadAction);
                threads[i].Start();
            }

            for (int i = 0; i < threads.GetUpperBound(0); i++)
            {
                threads[i].Join();
            }
        }

        private void ThreadAction()
        {
            var dependency1 = MockRepository.GenerateStub<IDependency1>();
            var dependency2 = MockRepository.GenerateStub<IDependency2>();

            dependency1.Stub(m => m.DependentMethod("System.String")).Return(false);

            dependency2.Stub(m => m.Kind).Return(TypeKind.Foo);
            dependency2.Stub(m => m.StringProperty).Return("System.String");

            var target = new TestSubject(dependency1, dependency2);
            target.SubjectMethod();
        }
    }

I tried solving this by using the old Rhino syntax of declaring a method local repository with no luck.  I also tried creating dedicated fields for all dependencies for each test and initialising them in the constructor, but this didn't work either.

Solution:

I tested a threadsafe modification to Rhino and it resolved the issues shown in the code example. But I didn't test it extensively.  You can find the blog post here and the DLL download here.  I'm not sure if this is a good idea or not.

Another alternative is to place all your mock expectation calls in the ClassInitialise() of the test class and create separate static fields for each dependency for each test.  This ensures that all tests have had their set up done first.  You should not put this code in a TestInitialise() as this will run prior to every test, and what you're after is running the expectation code exactly once.

    [TestClass]
    public class Test3
    {
        private static IDependency1 dependency1a;

        private static IDependency2 dependency2a;

        private static IDependency1 dependency1b;

        private static IDependency2 dependency2b;

        [ClassInitialize]
        public static void ClassInitialise(TestContext context)
        {
            // For Test a
            dependency1a = MockRepository.GenerateStub<IDependency1>();
            dependency2a = MockRepository.GenerateStub<IDependency2>();

            dependency1a.Stub(m => m.DependentMethod("System.String")).Return(false);

            dependency2a.Stub(m => m.Kind).Return(TypeKind.Foo);
            dependency2a.Stub(m => m.StringProperty).Return("System.String");

            // For Test b
            dependency1b = MockRepository.GenerateStub<IDependency1>();
            dependency2b = MockRepository.GenerateStub<IDependency2>();

            dependency1b.Stub(m => m.DependentMethod("System.String")).Return(false);

            dependency2b.Stub(m => m.Kind).Return(TypeKind.Foo);
            dependency2b.Stub(m => m.StringProperty).Return("System.String");
        }

        [TestMethod]
        public void Test3a()
        {
            var target = new TestSubject(dependency1a, dependency2a);
            Assert.IsFalse(target.SubjectMethod());
        }

        [TestMethod]
        public void Test3b()
        {
            var target = new TestSubject(dependency1b, dependency2b);
            Assert.IsFalse(target.SubjectMethod());
        }
    }

Test 3a and 3b will always pass.

You could also use a lock, although this feels a bit dirty in unit testing, its the least amount of code.

So just a word of warning, you will need to be aware of this only if you take advantage of running tests in parallel.

TypeMock Isolator war stories

Like many other developers out there, most of the time we are working with some form of legacy code that when written was not designed to be tested.  Changing existing code to be unit testable can be hard particularly if there really wasn't much care taken with the design originally, or there's been years of hacky maintenance of the code.  I was always curious to give Typemock's Isolator a try to see if it helped making testing of legacy code easier or quicker.

Background

I had two completely different projects in mind.  One a legacy project that is around 7 years old and has been submitted to constant time constrained projects that led to bad coding practices.  This project is essentially a standard 3 tier architecture, although I focus on the WCF / Remoting service layer and below.  I'll refer to this project as the legacy project. 

The other is purely a green fields project.  It is a standalone Windows executable that has been designed and built recently and makes extensive use of interfaces, abstract classes, and virtual methods. I'll refer to this project as the modern project.

I'll try to compare these two distinct scenarios to test if Typemock's Isolator is best suited to one or the other. 

What is Typemock Isolator

By now most modern developers have had some exposure to unit testing, so I won't go into definitions or how-to's here.  There are many unit testing frameworks and utilities that help in unit testing.  Firstly you'll need a unit test runner (nUnit, MsTest, xUnit etc). In all but the most basic of unit tests you will also need a mocking framework (RhinoMock, Moq, NMock, and now with Visual Studio 2012 Microsoft Fakes). Isolator falls into this category and comes packages with other utilities that integrate into Visual Studio to assist in unit testing.  

Immediate Differences on the Surface

Ease of use is gauged simply as: easy, medium, hard

	Isolator Basic Edition	Isolator Essential Edition	RhinoMocks	Moq	Microsoft Fakes
Client License Price	Free	US$799	Free	Free	Requires Visual Studio 2012Ultimate
Server License Price	Free	US$2499	Free	Free	Free
Ease of use (imho)	Medium	Medium	Easy-Medium *1	Easy	Hard *2
Mock Interfaces/Abstracts/Virtuals	Yes	Yes	Yes	Yes	Yes
Mock Concrete classes	No	Yes	No	No	Yes
Mock Static usage	No	Yes	No	No	Yes

*1 - There are alternative usage styles in RhinoMocks. The simple way works most of the time, but sometimes you may need to dig deeper to write more advanced tests. Using the more advanced features does increase the difficulty. (This is based on feedback I get from other devs I work with; I think its a good trade-off for a feature rich framework).
*2 - This may be because my exposure to it is a little limited, however the syntax does feel a little awkward and verbose.

Biggest difference is the price.  Being able to exercise .NET black magic and mock the unmockable doesn't come for free.

Working with the Legacy Project

Legacy code never has just one issue, there are always multiple nasty problems intertwined.  Don't be naive and think that writing Isolator is a magic bullet that will make it easy. It won't. It does make it easier for sure. But it won't help you much with issues like badly designed methods and classes with thousands of lines of code.

Have a look at this for examples of working with Legacy code tests using Isolator.

The temptation is to write tests that ensure every interaction which normally results in brittle tests.  As soon as you refactor to a better design the method will be doing less.  Its normally more effective to focus on use cases for a class rather than focusing on the legacy methods.

So is this an effective strategy on legacy code? Is it good bang for buck - time spent versus quality and robustness of tests going forward?  Can the same be achieved by integration testing with SOAPUI or Selenium for example?

My experience has been you seldom come across legacy code that is well design, but just doesn't have tests. Writing good tests involves focusing on behaviour not tests line by line a method.  This is very difficult with legacy code as the Single Responsibility Principle has most likely been violated as well.  Behaviours will be spread across multiple classes in a not necessarily logical manner. So writing tests straight from the legacy code is a bad idea, it should be refactored as well. The best way to do this is use integration tests (either coded or using SoapUI, Selenium etc) and then refactor the internal code before writing focused unit tests.

Working with the Modern Project

So if you can afford it, does it mean that writing straight-forward code that does not use interfaces, abstracts, or virtuals is no longer necessary? No. No. No. Good design principals still apply.  Just because unit testing is the first consumer of abstractions and extension points in your code does not mean there will be no other future consumers.  Maintaining code that has been SOLID-ly designed is proven to be easier (easier == cheaper) over and over.

So given that you want to write good loosely coupled SOLID code, do you need Isolator?  No, its like say do you need to take vitamins with a balanced diet.  However it still can help.  Isolator really shines when you need to test code that you have little control over, like where your code interfaces into a third party API.  Also there might be times when using its ability to mock Static method calls provides a cleaner way to test something, rather than writing wrappers and interfaces.

Utilities

Do all the fancy shiney utilities and helpers really matter?  Well, it depends.  I personally do not like too many utilities creating noise in my IDE.  But I can see how the Isolator utilities can guide your test design, ensuring your tests are focused by giving a warning if more than one assert is used etc.

Conclusion

If good design principals and cheap maintenance and extension (read as low cost of ownership) are important to you, then abandoning SOLID techniques is a bad idea. That said, you can still use Isolator and write well designed SOLID code.

For modern projects if you are lucky to have Visual Studio Ultimate, no brainer, you don't need Isolator. If you don't have Ultimate its a slightly more difficult decision: It does seem hard to justify spending the money when the other open-source frameworks will definitely serve you well.  The utilities in my humble opinion are nice to haves; these are not compelling enough to drive me to spend the money.  The Isolator utilities do not enforce anyone to follow their advise so there are no guarantees. My opinion is that because Isolator would be difficult to be abstracted away from or switched out once you have a significant number of tests, I don't like the idea of being locked in when I don't feel a compelling need to use it. How about the free edition of Isolator? To be honest, (subjective warning) I prefer a more fluent style like Rhino or Moq. You may also be at the mercy of features being pulled into the paid versions.  Again, this is just my personal preference, not really based on scientific reasons.

Using it to test UI code (ie controllers and code behind) has no benefit in my opinion. This code will change often and you're tests will begin to test ASP.NET and other third parties that are out of scope for your test. UI code always has high churn, so testing individual components starts to offer low bang for buck.  I would prefer integration style testing of UI's (like SpecFlow or Selenium).

For legacy projects, again if you're lucky to have Visual Studio Ultimate, it will be hard to justify this expense.  Otherwise, I strongly believe that if you are refactoring code you should refactor it to a better design including interfaces etc.   However, introducing interfaces and proper design will take time. Legacy code is never simple to cover in tests, Isolator will definitely help you and has some nice features that will help show progress to stakeholders. The counter argument is that integration style testing with tools like Selenium and SoapUI might serve better less brittle tests.  I believe both approaches are important, integration tests ensure behaviour, unit tests ensure good code design and micro-behaviour that is focused.

One final thing:  Using Isolator (or Ms Fakes) to test and validate third party API's is quite effective.  This allows you to keep third party vendors honest that when they make claims about backwards compatibility etc.