Optimised binary serialisation and serialisation tips

During some research a colleague of mine (thanks Richard)  has done into serialisation lately, two code project articles came to my attention.

Top Ten Caching mistakes, and
Optimizing Serialisation in .NET

The first one talks about some common mistakes and why they are mistakes and also some mitigation techniques.

The second talks about the problems using XML, DataContractSerialisation (or any text based serialisation for that matter).  Basically, they can be much slower than what you require. Also the binary serialisation and deserialisation process can create unnecessary data.  The article shows a custom binary serialiser that looks very useful.

Serialising a Linq Query to Json

IEnumerable<Employee> empJson = from emp in employees
                                where emp.Department == "IT Department"
                                select emp;
var ser = new DataContractJsonSerializer(typeof(IEnumerable<Employee>));
using (stream = new MemoryStream()) {
    ser.WriteObject(stream, empJson);
    string json = Encoding.Default.GetString(stream.ToArray());
    return json;

}

Xaml Serialization and Blend Sample Data

Background

For a project I am currently working on, we are using specialised UI Designers and they are using Blend to import, draw and arrange the graphic assets.  One of the coolest features in Blend 4 is the very nice Sample Data features.  Our process involves delivering partially constructed MVVM controllers so the Designers can use Blend to generate random sample data.  This is all well and good, but it still means the application is totally blank when it runs, hence the generated sample data is called "Design-time Sample Data".  However, the sample data is actually a serialised format of instances of objects:

Blend's Design-time Sample Data

Blend offers 3 ways of creating sample data:

1. From an XML file you have handcrafted yourself. 
2. From a code class.
3. Or manually creating it using a designer within Blend. (Not terribly useful in my opinion as the developers generally set the underlying class structure and names).

I have been using the code class option.

This generates a sample data item in the Data tab.

From this we can now be more specific on what kind of data each field is.  For example string fields can use the following templates to generate sample data:

Blend has created an xaml file to store in serialised form the object instances it has created.

Having sample data in the application makes the designer's job much easier as the see the data in the fields. It also means its easier to create custom templates for collection based items.

Xaml Serialised Data

The Xaml markup is actually just a form of XML serialisation. Here's an example of the file format generated by Blend:

<WpfApplication:MainWindowController 
    xmlns:WpfApplication="clr-namespace:WpfApplication" 
    State="Active">

    <WpfApplication:MainWindowController.CurrentForm>
        <WpfApplication:FormController State="Active">
            <WpfApplication:FormController.CurrentContact>
                <WpfApplication:Contact 
                    Company="A. Datum Corporation" 
                    Email="someone@example.com" 
                    FirstName="Aaberg, Jesper" 
                    MainLine="(111) 555-0100" 
                    Mobile="(111) 555-0100" 
                    State="Active" 
                    Surname="Aaberg, Jesper" />
            </WpfApplication:FormController.CurrentContact>
        </WpfApplication:FormController>
    </WpfApplication:MainWindowController.CurrentForm>

    <WpfApplication:MainWindowController.Dashboard>
        <WpfApplication:DashboardController 
            Capacity="33" 
            Online="True" 
            State="Invalid" 
            Status="Amet dictumst curae donec eleifend" 
            StatusEnum="Online" />
    </WpfApplication:MainWindowController.Dashboard>

</WpfApplication:MainWindowController>

This syntax is significantly better than standard Xml serialisation because the xml type elements are namespaced back to the assembly they are declared in.  Also it follows the same serialisation process as WCF Service contract serialisation in that it serialises all public properties. Standard Xml serialisation only serialises fields and those must be decorated with Xml Attributes! Way too much hassle.  Xaml serialisation is definitely the way of the future.

The Problem

Having design time data is a very tidy solution as it guarantees the design time data doesn't make it into your compiled executable.  The only drawback is that the designers can't run the application to do final visual testing.  One solution to this problem is to use the Xaml Serialisation classes to deserialise the Xaml back into objects and bind to those at runtime.

I feel like I need to come up with a safer more elegant solution so the design time cannot be accidentally left in the executable code, but for now I use a compilation switch #SAMPLEDATA. But hopefully the likelihood of someone releasing the product with the SAMPLEDATA switch intact is low.

public partial class App
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="App"/> class.
        /// </summary>
        public App()
        {
#if (SAMPLEDATA)
            // Sample data used at runtime.  Sometimes useful for visual testing in addition to design time testing in Blend.
            SampleData.SampleDataRepository.MainWindowController =
                SetUpSampleData<MainWindowController>(@".\..\..\SampleData\MainWindowControllerSampleData.xaml");
#endif
        }

#if (SAMPLEDATA)
        private static T SetUpSampleData<T>(string sampleDataFile) where T : class, new()
        {
            // Using Xaml
            if (File.Exists(sampleDataFile))
            {
                var deserialisedObject = XamlServices.Load(sampleDataFile);
                return deserialisedObject as T;
            }
            
            return null;
        }
#endif
    }

You can see I am using a static class with static properties to store the pre-made sample data controllers ready for use.  The sample controllers replace runtime controllers when the SAMPLEDATA switch is present.  .Net 4 includes a new class designed to simplify Xaml serialisation System.Xaml.XamlServices.

public partial class Sample {
        public Sample() {
            InitializeComponent();

#if (SAMPLEDATA)
            this.DataContext = SampleData.SampleDataRepository.MainWindowController;
#endif
        }
    }

Ordinarily when running in "Production" mode the DataContext is set in Xaml (my personal preference), or if it is set in code behind then it would need to be above the #if.

For more complicated scenarios the controllers might need to be interfaced if the are tightly bound to other data sources. This would allow the sample controllers to bypass any data service calls etc.

Serialisation

It doesn't apply to this scenario but it is equally as easy as deserialisation.

string serialised = XamlServices.Save(myObject);

No need to tell it the type or anything else. There are other overloads that take streams and writers and parameters.

For more information:

• Guy Berstein's blog post on the new .Net4 Xaml Services class
• State Street Gang post on Xaml Serialisation in general

Xml Serialization

[Edit: 28-June-2010] Prefer Xaml Serialisation over Xml serialisation.

10-May-2010

Here's a basic example of serialising (Yes that's right in this part of the world that's how its spelt), an object to Xml:

[Edit: BTW you will need to add a reference to System.Runtime.Serialization in addition to the standard C# console app references).

namespace XmlSerialisation {
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO;
    using System.Xml.Serialization;
    using System.Xml;

    public class Program {
        public static void Main(string[] args) {
            var subject = new KeyValuePair<int, string>(21, "Full House Aces High");

            var builder = new StringBuilder();
            using (var writer = new StringWriter(builder)) {
                var serialiser = new XmlSerializer(typeof(KeyValuePair<int, string>));
                builder.AppendLine();
                builder.AppendLine("XmlSerializer:");
                serialiser.Serialize(writer, subject);
                builder.AppendLine();
            }

            using (var writer = new StringWriter(builder)) {
                using (var xmlWriter = new XmlTextWriter(writer)) {
                    var wcfSerialiser = new System.Runtime.Serialization.DataContractSerializer(typeof(KeyValuePair<int, string>));
                    builder.AppendLine();
                    builder.AppendLine("WCF Serialisation:");
                    wcfSerialiser.WriteObject(xmlWriter, subject);
                    builder.AppendLine();
                }
            }

            Console.WriteLine(builder.ToString());
        }
    }
}

Here's the output of the program:

XmlSerializer:

<?xml version="1.0" encoding="utf-16"?>

<KeyValuePairOfInt32String xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

 xmlns:xsd="http://www.w3.org/2001/XMLSchema" />

WCF Serialisation:

<KeyValuePairOfintstring xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xml

ns="http://schemas.datacontract.org/2004/07/System.Collections.Generic"><key>21<

/key><value>Full House Aces High</value></KeyValuePairOfintstring>

One other thing I was trying to prove with this code is the different behaviours of the two different serialisers. The standard XmlSerializer serialises based on public properties (and takes into account Xml attributes you may have decorated your members with, for example [System.Xml.Serialization.XmlElement("ReplacementMemberName")]).

The DataContractSerializer looks at fields for its serialisation process, and will ignore the System.Xml.Serialization attributes.

You also have a Json serialiser at your disposal too, see here for more info.

[Edit 28-June-2010]

Here's an example of Deserialisation:

    private static T SetUpSampleData<T>(string sampleDataFile) where T : class, new()
        {
            if (File.Exists(sampleDataFile))
            {
                // var serialiser = new System.Xml.Serialization.XmlSerializer(typeof(MainWindowController));
                var serialiser = new System.Runtime.Serialization.DataContractSerializer(typeof(MainWindowController));
                using (var stream = new FileStream(sampleDataFile, FileMode.Open, FileAccess.Read))
                {
                    using (var reader = XmlReader.Create(stream))
                    {
                        // var deserialisedObject = serialiser.Deserialize(reader);  // XmlSerialiser
                        var deserialisedObject = serialiser.ReadObject(reader);      // WCF DataContract Serialiser
                        return deserialisedObject as T;
                    }
                }
            }

            return null;
        }

Json Serialization

23-Oct-2009

I've currently been experimenting with performance tuning some REST services.  I was surprised how easy it was to change XML serialisation to Json serialisation.  Just a simply matter of changing System.Xml.Serialization.XmlSerializer to System.Runtime.Serialization.Json.DataContractJsonSerializer.

Seems to be about 30% smaller in size over xml.

Serialisation:

            using (var stream1 = new MemoryStream()) {

                var serializer = new DataContractJsonSerializer(dto.GetType());

                serializer.WriteObject(stream1, dto);

                stream1.Flush();

                stream1.Position = 0;

                var reader = new StreamReader(stream1);

                return reader.ReadToEnd();

            }

Deserialisation:

            using (var stream2 = new MemoryStream(Encoding.Unicode.GetBytes(serialisedInstance)) {

                var ser = new DataContractJsonSerializer(type);

                return ser.ReadObject(stream2);

            }

Size comparison:

<AddressDto>

    <City>Auckland</City>

    <Country>New Zealand</Country>

    <CreatedOn>2008-02-29T00:00:00</CreatedOn>

    <Id>c32288fc-5037-4f22-aba5-fca6d10000b3</Id>

    <Line1>24 Rodney Street</Line1>

    <Line2></Line2>

    <Sid>c32288fc-5037-4f22-aba5-fca6d10000b3@AddressDto</Sid>

    <State>Auckland</State>

    <Suburb>Birkenhead</Suburb>

    <Updated>2007-12-31T07:41:59</Updated>

    <Version>c32288fc-5037-4f22-aba5-fca6d10000b3</Version>

    <Zip>2801</Zip>

</AddressDto>

499 bytes.

{"City":"Auckland","Country":"New Zealand","CreatedOn":"\/Date(951735600000+1300)\/","Id":"36322616-0b72-45b4-aa51-4e7119103f27","Line1":"24 Rodney Street","Line2":"","Sid":"36322616-0b72-45b4-aa51-4e7119103f27@AddressDto","State":"Auckland","Suburb":"Birkenhead","Updated":"\/Date(1199040119000+1300)\/","Version":"36322616-0b72-45b4-aa51-4e7119103f27","Zip":"2801"}

367 bytes.

Performance Testing:

I've done a comparison of this Xml to Json change and now have some hard data on how much smaller and faster Json is.

The test involves calling a REST service to first check for a randomly created string LogOnId availability; this is a HTTP GET call. LogOnId must be unique and a user may choose their preferred handle. After the uniqueness check comes back successful, it calls a create client service call. This is a HTTP POST. Finally it calls GET to return the newly created client.

Each call is time individually. 10 threads will be used to create 1,000 new clients. The REST service has been configured to use an in-memory database to minimuse random Database and network delays. All up 10,000 clients should be created.

The test creating the 10,000 clients for Xml and Json were repeated 3 times each to give an average.