RSS

UniversalSerializer

13 Jul

The source code is available here.

What is it ?

UniversalSerializer is a Serializer/Deserializer for .Net.
In other words, it transcodes an object instance tree into a byte array, and vice versa.

Summary

The objective of UniversalSerializer is to be able to serialize any type with no effort.

  • No need to add attributes nor interfaces to the types (classes & structures).

  • When a class instance is referenced several times, it is serialized only once.

  • Circular references are allowed.

  • Existing serialization or transcoding mechanisms are reused.
    Currently: [Serializable], ISerializable, [ValueSerializer] and [TypeConverter].

  • No-default-constructor types are allowed, if a parametric constructor can be exploited (it is automatic).

  • Not-generic ICollection classes can be serialized if an Add or or an Insert method can be exploited (it is automatic).

Of course, all ordinary constructions as classes, structures, public properties, public fields, enumerations, collections, dictionaries, etc.. are serialized by UniversalSerializer.

When a type is not serializable out-of-the-box, UniversalSerializer offers two mechanisms:

  • ITypeContainers
    We can include the problematic type (or set of types) in a custom class that will manage its serialization and deserialization.

  • A set of filters
    We can block some types, and force the serializer to store select source class’ private fields.

My best wish is that people add more Containers and Filters in the future, and maybe one day all types can be serializable.

Example of usage

var data = new Window();
byte[] DataBytes = UniversalSerializer.Serialize(data);
var data2 = UniversalSerializer.Deserialize<Window>(DataBytes);

That is that simple!

Example for WPF

There is a specialized DLL for WPF, that manages more WPF types:

var data = new Window();
byte[] bytesData = UniversalSerializerWPF.SerializeWPF(data);
var data2 = UniversalSerializerWPF.DeserializeWPF<Window>(bytesData);

Example for WinForms

There is a specialized DLL for WinForms, that manages more WinForms types:

var data = new Form();
byte[] DataBytes = UniversalSerializerWinForm.SerializeWinForm(data);
var data2 = UniversalSerializerWinForm.DeserializeWinForm<Form>(DataBytes);

Why and how is it done ?

I needed a universal serializer, able to serialize anything without modification.

But the existing serializers have drawbacks:

  • Some need special attributes or interfaces.
  • Others do not take fields into account.
  • References are a problem. Usually the instances are duplicated, and the deserialized objects point to different instances.
    Circular references are never managed.
  • None can manage class types with no default (no-parameters) constructor, as far as I know.
  • Particular .Net classes need more attention but are sealed therefore can not be inherited with serialization attributes.

So the solution needed a set of mechanisms and techniques.
We will see these mechanisms in the next chapters.

Original code

UniversalSerializer is built on Mehdi Gholam’s fastBinaryJSON 1.3.7, which serializes to a custom binary format inspired by JSON.
UniversalSerializer adds a certain universality to it by letting us serialize any type.
Please note I had to modify fastBinaryJSON deeply, therefore I can not synchronise any more with the original fastBinaryJSON’s source code. For that reason UniversalSerializer is not an extension of fastBinaryJSON and I will not maintain compatibility in the future.

Serialize no-default-constructor classes

In .Net, default (no-parameters) class constructors are not compulsory. But they are needed by almost all serializers.
And they are frequent in the framework. Example: System.Windows.Controls.UIElementCollection.

The solution in UniversalSerializer is to search for other constructors, and find a correspondence in the class between the constructor parameters and the fields, even if these fields are private.

In UIElementCollection, we have this constructor:

public UIElementCollection( UIElement visualParent, FrameworkElement logicalParent )

And these fields are available in the same class:

  • private readonly UIElement _visualParent;
  • private readonly FrameworkElement _logicalParent;

Types are equal and their name is very close. Enough to let UniversalSerializer try to create an instance with these values.
And it works !

Serialize not-generic ICollection classes

For some obscure reason, the ICollection interface does not provide Add nor Insert methods.
In other words, it defines a read-only collection, contrary to the generic ICollection<T>.
Normally, we could not deserialize a class that implement ICollection and not ICollection<T>.

Fortunately, in the real world nearly all of these classes have a Add or a Insert method, at least for internal use.
UniversalSerializer finds these methods and uses them to deserialize the collection class instance.

Reuse existing mechanisms and manage them in ITypeContainers

In some cases, it is more efficient or only possible to use existing transcoding mechanisms.

When I tried to serialize WPF controls, I discovered that:

  • [TypeConverter] lets transcode the object to some types which can be serialized. (string, byte array, etc.).
    Example: System.Windows.Input.Cursor
  • [ValueSerializer] lets transcode from and to a string.
    Example: System.Windows.Media.FontFamily
  • [Serializable] (and ISerializable) allows the use of BinaryFormatter.
    Example: System.Uri

If you consider FontFamily, you will understand that transcoding it to a string is much easier than trying to save its properties.
And safer, because setting a property can lead to unpredictable consequences on unknown or complex classes.

For these attributes, I created the mechanism of ITypeContainer. A container replaces the source instance by its transcoded value, usually a string or a byte array.
A container can be applied to a set of types, for example all the attribute-compatible types.

Examples and details can be found below.

Filters

Some types need special process, which is done by a set of filters.

Type validator filter

This filter allows you to prevent UniversalSerializer from serializing some problematic types.

For example, I met some classes using System.IntPtr .
Serializing this type only leads to problems since they are only used internally in the classes, even when they are stored in public properties.

Private fields adder filter

This filter tells the serializer to add a particular private field to the serialization data.

For example, System.Windows.Controls.Panel needs _uiElementCollection to fill its Children property, since Children is read-only.
With the filter, the solution is easy. And any type that inherits Panel, such as StackPanel, will benefit this filter.

ForcedParametricConstructorTypes

It is not a filter but a list of types.
When a type is in this list, UniversalSerializer ignores its default (no-parameters) constructor and searches for a parametric constructor.
Example: System.Windows.Forms.PropertyManager . It is much easier to use its parametric constructor than to write a ITypeContainer for this type.

References

Consider this code:

{ // Two references to the same object.
  var data = new TextBox[2];
  data[0] = new TextBox() { Text = "TextBox1" };
  data[1] = data[0]; // Same reference
  byte[] bytesData = UniversalSerializer.Serialize(data);
  var data2 = UniversalSerializer.Deserialize<TextBox[]>(bytesData);
  data2[0].Text = "New text"; // Affects the two references.
  bool sameReference = object.ReferenceEquals(data2[0], data2[1]);
}
  1. UniversalSerializer serializes only one instance of the TextBox.
  2. It deserializes only one instance of TextBox, and two references pointing to it.
    Proofs: sameReference is true, and Text is identical in both references.

Custom ITypeContainer and filters

Given that the main objective of UniversalSerializer is to be able to serialize any type, it is essential we share our experience.

So I tried to make the containers and filters creation as easy as possible, to help you play with them and share your solutions.

Making a ITypeContainer

The objective is to replace the problematic instance by an easy instance. In general, the container will contain very simple types (string, int, byte array, etc..).

Let’s take an example:

/// <summary>
/// . No default (no-param) constructor.
/// . The only constructor has a parameter with no corresponding field.
/// . The field ATextBox has no public 'set' and is different type from constructor's parameter.
/// </summary>
public class MyStrangeClassNeedsACustomerContainer
{
    /// <summary>
    /// It is built from the constructor's parameter.
    /// Since its 'set' method is not public, it will not be serialized directly.
    /// </summary>
    public TextBox ATextBox { get; private set; }
    public MyStrangeClassNeedsACustomerContainer(int NumberAsTitle)
    {
        this.ATextBox = new TextBox() { Text = NumberAsTitle.ToString() };
    }
}

As written in the summary, this class causes some difficulties to the serializer(s).

To overcome the problem, we create a container:

class ContainerForMyStrangeClass : UniversalSerializerLib.ITypeContainer
{
#region Here you add data to be serialized in place of the class instance
public int AnInteger; // We store the smallest, sufficient and necessary data.
#endregion Here you add data to be serialized in place of the class instance

public UniversalSerializerLib.ITypeContainer CreateNewContainer(object ContainedObject)
{
MyStrangeClassNeedsACustomerContainer sourceInstance = ContainedObject as MyStrangeClassNeedsACustomerContainer;
return new ContainerForMyStrangeClass() { AnInteger = int.Parse(sourceInstance.ATextBox.Text) };
}
public object Deserialize()
{
return new MyStrangeClassNeedsACustomerContainer(this.AnInteger);
}
public bool IsValidType(Type type)
{
return Tools.TypeIs(type, typeof(MyStrangeClassNeedsACustomerContainer));
}
public bool ApplyEvenIfThereIsANoParamConstructor
{
get { return false; }
}
public bool ApplyToStructures
{
get { return false; }
}
}

A detail: all methods behave as static methods (but are not), except Deserialize().

Let’s see them more in details:

  • public int AnInteger

    It is not part of the ITypeContainer interface.
    Here we will store the information we will need later at deserialization.

  • ITypeContainer CreateNewContainer(object ContainedObject)

    Used during serialization.
    This is a kind of constructor for this container instance. The parameter will be the source class instance to serialize.

  • object Deserialize()

    Used during deserialization.
    The container instance will produce a new instance, a copy of the source instance, using our field AnInteger.

  • bool IsValidType(Type type)

    Used during serialization.
    Returns true is the type inherits from, or is, the source type.
    This is a filter.
    We can choose to accept inherited types or not, to accept several compatible types, etc..

  • bool ApplyEvenIfThereIsANoParamConstructor

    Used during serialization.
    Returns true if this container applies to class types with a default (no-param) constructor.
    Can be useful to very general containers.

  • bool ApplyToStructures

    Used during serialization.
    Returns true if this container applies to structure types, and not only class types.
    Can be useful to very general containers.

Steps are:

  1. The serializer checks if the source type (MyStrangeClassNeedsACustomerContainer) is managed by a container.
    Our container class (ContainerForMyStrangeClass) answers yes, via IsValidType(), ApplyEvenIfThereIsANoParamConstructor and ApplyToStructures.
  2. The serializer builds an instance of our container, via CreateNewContainer().
    CreateNewContainer builds an instance and sets its field AnInteger.
  3. The serializer stores (serializes) this container instance in place of the source instance.
  4. The deserializer retrieves (deserializes) the container instance.
  5. The deserializer calls Deserialize() and obtains a copy of the source class instance.
    Deserialize() creates this copy using its field AnInteger.

Now we serialize it:

/* This example needs a custom ITypeContainer.
Normally, this class can not be serialized (see details in its source).
But thanks to this container, we can serialize the class as a small data (an integer).
*/
var Params = new UniversalSerializerLib.CustomParameters();
Params.Containers = new UniversalSerializerLib.ITypeContainer[] {
new ContainerForMyStrangeClass()
};
var data = new MyStrangeClassNeedsACustomerContainer(123);
byte[] bytesData = UniversalSerializer.Serialize(data, Params);
var data2 = UniversalSerializer.Deserialize<MyStrangeClassNeedsACustomerContainer>(bytesData, Params);
bool ok = data2.ATextBox.Text == "123";

As you can see, the implementation is very easy.

The Tools static class offers some help:

  • Type Tools.TypeIs(Type ObjectType, Type SearchedType)

    It is equivalent to the C#’s ‘is’, but for Types.
    For example, TypeIs((typeof(List<int>), typeof(List<>)) returns true.

  • Type DerivedType(Type ObjectType, Type SearchedType)

    Returns the type corresponding to SearchedType that is inherited by OjectType.
    For example, DerivedType(typeof(MyList), typeof(List<>)) returns typeof(List<int>) when MyList is

    MyList: List<int> { }.

  • FieldInfo FieldInfoFromName(Type t, string name)

    Returns the FiedInfo of the named field of the type.
    We will use it in the next chapter.

Making a set of filters

Please note the filter mechanism is totally independent from the ITypeContainers.
They can be used together, or separately.

Let’s take an example:

public class ThisClassNeedsFilters
{
public ShouldNotBeSerialized Useless;
private int Integer;
public string Value { get { return this.Integer.ToString(); } }
public ThisClassNeedsFilters()
{
}
public ThisClassNeedsFilters(int a)
{
this.Integer = a;
this.Useless = new ShouldNotBeSerialized();
}
}
public class ShouldNotBeSerialized
{
}

This class (ThisClassNeedsFilters) have some problems:

  • It contains a ShouldNotBeSerialized.
    Let’s imagine the class ShouldNotBeSerialized has to be avoided for some reasons, I don’t know why, maybe it is poisoned !
  • The field Integer is not public and therefore is ignored by the serializer(s).
  • Even the constructor parameter name is different from any field or property.
    Anyway the serializer does not need this constructor, as it already has a default constructor.

To overcome these problems, we write a custom set of filters:

/// <summary>
/// Tells the serializer to add some certain private fields to store the type.
/// </summary>
FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)
{
if (Tools.TypeIs(t, typeof(ThisClassNeedsFilters)))
return new FieldInfo[] { Tools.FieldInfoFromName(t, "Integer") };
return null;
}
/// <summary>
/// Returns 'false' if this type should not be serialized at all.
/// That will let the default value created by the constructor of its container class/structure.
/// </summary>
bool MyTypeSerializationValidator(Type t)
{
return ! Tools.TypeIs(t, typeof(ShouldNotBeSerialized));
}

They are self-explanary:

  • FieldInfo[] MyAdditionalPrivateFieldsAdder(Type t)

    makes the serializer add a private field (Integer) to every source instance of this type (ThisClassNeedsFilters).

  • bool MyTypeSerializationValidator(Type t)

    prevents the serializer from storing any instance of this type (ShouldNotBeSerialized).
    Consequently, any instance of ThisClassNeedsFilters will not set the Useless field.

Now we serialize it:

/* This example needs custom filters.
Normally, this class can be serialized but with wrong fields.
Thanks to these filters, we can serialize the class appropriately.
*/
var Params = new UniversalSerializerLib.CustomParameters();
Params.FilterSets = new FilterSet[] {
new FilterSet() {
AdditionalPrivateFieldsAdder=MyAdditionalPrivateFieldsAdder,
TypeSerializationValidator=MyTypeSerializationValidator } };
var data = new ThisClassNeedsFilters(123);
byte[] bytesData = UniversalSerializer.Serialize(data, Params);
var data2 = UniversalSerializer.Deserialize<ThisClassNeedsFilters>(bytesData, Params);
bool ok = data2.Value == "123" && data2.Useless == null;

The implementation is even easier than with ITypeContainer.

The source code of UniversalSerializer

All sources are C#.

The solution was created with Visual Studio 2012, but should be compatible with VS 2010.
Compilation only needs .Net 4, not any third-party DLLs, the source code is complete.

There are 3 DLLs:

  • UniversalSerializer1.dll
    The main general DLL, with very few dependencies.
  • UniversalSerializerWPF1.dll
    A specialized DLL for WPF, that manages more WPF types.
  • UniversalSerializerWinForm1.dll
    A specialized DLL for WinForms, that manages more WinForms types.

This way, your application will not have useless dependencies.

In the archive, you will find these VS solutions:

  • \UniversalSerializer Tester\UniversalSerializer Tester.sln
    A WPF application with many examples.
  • \UniversalSerializerWinForm\UniversalSerializerWinForm.sln
    A WinForm application with some examples.
  • \UniversalSerializerReleaseLibs\UniversalSerializerReleaseLibs.sln
    This solution generates Release versions of the three DLLs.

Important points

  • The serialization/deserialization process identifies types by their complete name (from type.AssemblyQualifiedName).
    This kind of name depends on the assembly version, therefore please take care of platform and DLL versioning !

  • The current produced format is subject to changes in the future.
    For that reason the DLL name has a version number.
    If you save the produced byte array to a file, I suggest you to add a version information to your object tree.

Future

I would like to improve some aspects:

  • Add more ITypeContainers and filters.
    You can help me. Tell me when types are not serialized correctly. And please share your solution, as containers & filters.
  • Reduce the produced data length.
    I need to create a new binary format.
  • Speed up the process.

Make the serializers’ job easier

This experience taught me why serializers have difficulties with some types and how we can facilitate its job when we author a class.

  1. Write a default (no-param) constructor when possible.
    The serializer will reconstruct the instance from its fields and properties.
  2. If you can not write a default (no-param) constructor, write a public parametric constructor with corresponding private fields.
    The field should have the same type and the same name. In fact the name can be slightly different: Param -> param or _param or _Param.
  3. Implement a ValueSerializerAttribute or a TypeConverterAttribute when an instance can be constructed from something as simple as a string.
    Especially when your class contains many public optimisation fields or properties.
    For example, FontFamily can be constructed from a simple string, no matter if it contains many other informations they all come from this simple string.
    I don’t know if many serializers take these attributes into accounts, but at least UniversalSerializer does.
  4. All optimisation fields or properties should be private, when possible. See above.
  5. When you create an object collection, implement IList.
      Because ICollection does not allow to add items to the collection.
    When you create a generic collection, implement ICollection<>.
      Because IEnumerable<> does not allow to add items to the collection.

Thanks

I thank Mehdi Gholam for his fastBinaryJSON.
I learned interesting things reading his code.
Thank you for sharing it.

Advertisements
 
 

Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: