Json Serialization

23-Oct-2009

I've currently been experimenting with performance tuning some REST services.  I was surprised how easy it was to change XML serialisation to Json serialisation.  Just a simply matter of changing System.Xml.Serialization.XmlSerializer to System.Runtime.Serialization.Json.DataContractJsonSerializer.

Seems to be about 30% smaller in size over xml.

Serialisation:

            using (var stream1 = new MemoryStream()) {

                var serializer = new DataContractJsonSerializer(dto.GetType());

                serializer.WriteObject(stream1, dto);

                stream1.Flush();

                stream1.Position = 0;

                var reader = new StreamReader(stream1);

                return reader.ReadToEnd();

            }

Deserialisation:

            using (var stream2 = new MemoryStream(Encoding.Unicode.GetBytes(serialisedInstance)) {

                var ser = new DataContractJsonSerializer(type);

                return ser.ReadObject(stream2);

            }

Size comparison:

<AddressDto>

    <City>Auckland</City>

    <Country>New Zealand</Country>

    <CreatedOn>2008-02-29T00:00:00</CreatedOn>

    <Id>c32288fc-5037-4f22-aba5-fca6d10000b3</Id>

    <Line1>24 Rodney Street</Line1>

    <Line2></Line2>

    <Sid>c32288fc-5037-4f22-aba5-fca6d10000b3@AddressDto</Sid>

    <State>Auckland</State>

    <Suburb>Birkenhead</Suburb>

    <Updated>2007-12-31T07:41:59</Updated>

    <Version>c32288fc-5037-4f22-aba5-fca6d10000b3</Version>

    <Zip>2801</Zip>

</AddressDto>

499 bytes.

{"City":"Auckland","Country":"New Zealand","CreatedOn":"\/Date(951735600000+1300)\/","Id":"36322616-0b72-45b4-aa51-4e7119103f27","Line1":"24 Rodney Street","Line2":"","Sid":"36322616-0b72-45b4-aa51-4e7119103f27@AddressDto","State":"Auckland","Suburb":"Birkenhead","Updated":"\/Date(1199040119000+1300)\/","Version":"36322616-0b72-45b4-aa51-4e7119103f27","Zip":"2801"}

367 bytes.

Performance Testing:

I've done a comparison of this Xml to Json change and now have some hard data on how much smaller and faster Json is.

The test involves calling a REST service to first check for a randomly created string LogOnId availability; this is a HTTP GET call. LogOnId must be unique and a user may choose their preferred handle. After the uniqueness check comes back successful, it calls a create client service call. This is a HTTP POST. Finally it calls GET to return the newly created client.

Each call is time individually. 10 threads will be used to create 1,000 new clients. The REST service has been configured to use an in-memory database to minimuse random Database and network delays. All up 10,000 clients should be created.

The test creating the 10,000 clients for Xml and Json were repeated 3 times each to give an average.

REST Service Mock

8-Oct-2009

Unit testing a class that has a dependency on a external service, can be annoying.  The last thing you want to do is send live calls.  The other options appear to be setting up a test service with a test database, or writing an Echo style service that simply accepts any incoming data and echoes it back.  Neither of these are cool, and require far too much work and plumbing (not to mention slow). The ideal would seem to be the ability to quickly configure and run up a mock http listener to receive requests and give back programmable responses all isolated to the test.

I stumbled across a partial solution here: http://weblogs.asp.net/pglavich/archive/2005/09/04/424392.aspx

I customised, and perhaps simplified for my use case to this: (apologies for the bad copy and paste of my IDE style)

Start at the start, the TestMethod:

        [TestMethod]

        public void SendNotificationTest() {

            var mockService = new MockRestService();

            var subjectUnderTest = new SendNotificationJob() {

                Identity = GetTestData(Guid.NewGuid(), CreateUniqueLogOnId()),

                Url = new Uri(mockService.ListenOnUrl + "TestMethodName")

            };

            mockService.StartAsync();

            subjectUnderTest.SendNotification();

            mockService.Stop();

            Assert.IsTrue(mockService.RequestUrlsReceived.Count == 1);

            Assert.AreEqual(mockService.ListenOnUrl + "TestMethodName", mockService.RequestUrlsReceived[0]);

        }

Secondly the MockRestService class:

    using System;

    using System.Collections.Generic;

    using System.IO;

    using System.Net;

    using System.Threading;

    public class MockRestService {

        private readonly HttpListener testServer;

        private bool keepListening = true;

        private Thread listeningThread;

        public MockRestService() {

            string uniqueServiceName = Guid.NewGuid().ToString().Replace("-", string.Empty);

            string serviceUrl = "http://localhost:60340/" + uniqueServiceName + "/";

            this.testServer = new HttpListener();

            this.testServer.Prefixes.Add(serviceUrl);

            this.ListenOnUrl = new Uri(serviceUrl);

            this.testServer.Start();

            this.ResponsesToReturn = new Queue<string>();

            this.StatusesToReturn = new Queue<HttpStatusCode>();

            this.RequestUrlsReceived = new List<string>();

        }

        public Uri ListenOnUrl { get; private set; }

        public IList<string> RequestUrlsReceived { get; private set; }

        public Queue<string> ResponsesToReturn { get; private set; }

        public Queue<HttpStatusCode> StatusesToReturn { get; private set; }

        public void Stop() {

            this.keepListening = false;

            this.listeningThread.Join();

        }

        public void StartAsync() {

            this.listeningThread = new Thread(() => this.Start());

            this.listeningThread.Start();

        }

        private void Start() {

            do {

                var context = this.testServer.GetContext();

                if (context.Request != null && context.Request.Url != null && !string.IsNullOrEmpty(context.Request.Url.ToString())) {

                    this.RequestUrlsReceived.Add(context.Request.Url.ToString());

                    var writer = new StreamWriter(context.Response.OutputStream);

                    if (this.ResponsesToReturn.Count == 0) {

                        writer.WriteLine("Ok");

                    } else {

                        writer.WriteLine(this.ResponsesToReturn.Dequeue());

                    }

                    writer.Close();

                    if (this.StatusesToReturn.Count == 0) {

                        context.Response.StatusCode = (int)HttpStatusCode.OK;

                    } else {

                        context.Response.StatusCode = (int)this.StatusesToReturn.Dequeue();

                    }

                    context.Response.Close();

                }

            } while (this.keepListening);

        }

    }

Thats basically the crux of it.  If anyone's interested I'll post complete code.

Ben Rees - Oct 25, 2009 12:19 AM

A much tidier revision. Added support for optionally adding headers, and used a event to handle incoming requests.

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading;
using Collections;

/// <summary>
/// A mock to remove dependencies on external REST services. Designed to listen on a Url and Port number using standard Http.
/// </summary>
public sealed class MockRestService : IDisposable {
private readonly HttpListener testServer;
private HttpListenerContext context;

private bool isActive;

private bool keepListening = true;

/// <summary>
/// Initializes a new instance of the <see cref="MockRestService"/> class.
/// </summary>
public MockRestService() {
string uniqueServiceName = Guid.NewGuid().ToString().Replace("-", string.Empty);
string serviceUrl = "http://localhost:60340/" + uniqueServiceName + "/";
this.testServer = new HttpListener();
this.testServer.Prefixes.Add(serviceUrl);
this.ListenOnUrl = new Uri(serviceUrl);
this.testServer.Start();
this.ResponsesToReturn = new EventedQueue<string>();
this.ResponsesToReturn.ItemAdded += (s, e) => {
if (this.isActive) {
throw new InvalidOperationException("You cannot add to the Response Queue after the service has been started.");
}
};
this.StatusesToReturn = new EventedQueue<HttpStatusCode>();
this.StatusesToReturn.ItemAdded += (s, e) => {
if (this.isActive) {
throw new InvalidOperationException("You cannot add to the Status Queue after the service has been started.");
}
};
this.RequestUrlsReceived = new List<string>();
this.ResponseHeaders = new Dictionary<int, KeyValuePair<string, string>>();
}

/// <summary>
/// Gets the URL that has been setup to listen on. The Url is auto-generated to be unique every time this class is instantiated.
/// </summary>
/// <value>The listen on URL.</value>
public Uri ListenOnUrl { get; private set; }

/// <summary>
/// Gets the request urls that have been received. Used to check and assert test case pass.
/// </summary>
/// <value>The request urls received.</value>
public IList<string> RequestUrlsReceived { get; private set; }

/// <summary>
/// Gets the response headers. Keyed by an int zero-based index to indicate which response should be given then header.
/// </summary>
/// <example>
/// To add a header to the second response:
/// using (var serviceMock = new MockRestService) {
/// serviceMock.ResponseHeaders.Add(1, new KeyValuePair("HeaderName", "Some Header content goes in here."));
/// serviceMock.StartAsync();
/// }
/// </example>
/// <value>The response headers.</value>
[System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Design", "CA1006:DoNotNestGenericTypesInMemberSignatures", Justification = "Assessed and is appropriate here")]
public IDictionary<int, KeyValuePair<string, string>> ResponseHeaders { get; private set; }

/// <summary>
/// Gets the responses to return for each request. Responses are given out in queue order.
/// </summary>
/// <value>The responses to return.</value>
public EventedQueue<string> ResponsesToReturn { get; private set; }

/// <summary>
/// Gets the statuses to return. Statuses are given out in queue order.
/// </summary>
/// <value>The statuses to return.</value>
public EventedQueue<HttpStatusCode> StatusesToReturn { get; private set; }

/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
public void Dispose() {
this.Dispose(true);
}

/// <summary>
/// Starts the listener. This will spin up a asynchronous thread to listen for incoming requests.
/// </summary>
public void StartAsync() {
this.isActive = true;
this.testServer.BeginGetContext(this.RequestReceived, null);
}

/// <summary>
/// Releases unmanaged and - optionally - managed resources
/// </summary>
/// <param name="cleanUpManaged"><c>true</c> to release both managed and unmanaged resources; <c>false</c> to release only unmanaged resources.</param>
public void Dispose(bool cleanUpManaged) {
if (cleanUpManaged) {
this.keepListening = false;
this.isActive = false;
if (this.testServer.IsListening) {
this.testServer.Close();
}
}
}

private void RequestReceived(IAsyncResult result) {
if (!this.keepListening) {
return;
}

this.context = this.testServer.EndGetContext(result);
if (this.context.Request != null && this.context.Request.Url != null && !string.IsNullOrEmpty(this.context.Request.Url.ToString())) {
this.RequestUrlsReceived.Add(this.context.Request.Url.ToString());
int requestNumber = this.RequestUrlsReceived.Count - 1;
var writer = new StreamWriter(this.context.Response.OutputStream);
if (this.ResponsesToReturn.Count == 0) {
writer.WriteLine("Ok");
} else {
writer.WriteLine(this.ResponsesToReturn.Dequeue());
}

if (this.ResponseHeaders.ContainsKey(requestNumber)) {
var header = this.ResponseHeaders[requestNumber];
this.context.Response.AddHeader(header.Key, header.Value);
}

if (this.StatusesToReturn.Count == 0) {
this.context.Response.StatusCode = (int)HttpStatusCode.OK;
} else {
this.context.Response.StatusCode = (int)this.StatusesToReturn.Dequeue();
}

writer.Close();
this.context.Response.Close();
}

if (this.keepListening) { // Could be improved to ensure against race-condition (keepListening may be set to false before all calls can be processed).
this.testServer.BeginGetContext(this.RequestReceived, null);
}
}
}