Hi!

If you develop for .Net (and even if you don't :) the odds are high that you've already heard about Microsoft Silverlight, a platform for developing RIA style applications.

In this post I'd like to share some of our experiences while working on adding support for this platform to db4o, but before we move on, I'd like to stress some points:

  • The goal of this post is to describe the way we plan to add Silverlight support but we cant guarantee that we'll be able to stick to these plans. Also, we assume that readers have a good understanding of the related technologies (db4o, .Net, Silverlight)
  • Even though we're committed to this goal, as of today, Silverlight is not an officially supported platform.
  • All decisions are being taken based on the assumption that db4o most common usage scenario in this platform will be to store objects locally
    (RIA) hence we're focusing on embedded mode. Of course, we are a community driven project so this can change in the future.
  • Last but not least, current code assumes Silverlight Beta 3 (it might work on Silverlight 2 but that hasn't been checked).

That said, let's take a look on how the porting process looks like (strikethrough activities have been completed already):

  1. Feature set / limitations and possible fixes
  2. Understand the required changes / Port the selected projects
  3. Update/extend development tools like Sharpen
  4. Port the tests
  5. Update documentation
  6. Write samples
  7. Integrate into the build
  8. Publish Silverlight artifacts to web site

Feature set / Limitations
 
The first step was relatively simple: given the tight constrains imposed by Silverlight we just picked a doable set of features. For the initial support we decided
to concentrate our efforts in embedded mode, leaving some features out.
 
Also, while working on porting db4o to run on Silverlight we assembled a list of limitations and possible improvements:
  • By default Silverlight apps have a quota of 1.0 Mb of disk space. Using the right APIs it's possible to ask for more (through a user consent dialog). It would be nice if db4o could use this API to increase its quota once it detects that it is running out of space.

  • Only public fields are allowed (AFAIK, Silverlight doesn't support reflection over private fields). That means that fields need to be declared as public to be "storable".

  • db4o requires some additional configuration (this is the most severe limitation IMHO).

  • Client / Server mode: Silverlight doesn't allow direct socket connections so, at least with db4o's current implementation, it's not possible to support CS mode. We could make the communication channel pluggable and, maybe, use "REST" instead of TCP/IP but this would require more investigation.
    [edited on Jun / 23] Some days ago, Ashish Shetty, a kind program manager at Silverlight team just alerted me that this sentence may be misleading and that actually Silverlight support client sockets (with some port restrictions). The main point is that, at least for now, we'll not pursue adding client / server support on Silverlight platform [/edited]

  • Native queries / Linq: Since both technologies rely on Mono.Cecil and friends and these assemblies were not ported to Silverlight it's not possible to support them for now (anyway, we saw some good news regarding Mono.Cecil in this forum thread).
We are planing to investigate whether the Db4oTool can be used to instrument Silverlight assemblies and optimize native queries at compile time. If we manage to instrument Silverlight assemblies Db4oTool could be used to turn storable types into "self reflectors" through interface injection.
 
 
Understanding required changes
 
Understanding the changes required to successfully compile for Silverlight consisted mainly in getting the list of unsupported interfaces/classes/methods and adapting the code accordingly. The biggest changes were due to the lack of some I/O classes as well non generic collections, restrictions on assembly loading and a tighter security model. These constraints were the cause of most of the compilation errors we found.
 
For the I/O classes db4o is using an implementation from Michael Sync (which, by the way, was really helpful in the porting process and is the kind developer willing to spend time and help; hey guy, thank you very much!). Technically speaking, Michael wrote a db4o storage based on .Net IsolatedStorage model.
 
The lack of non generic classes was trickier to solve than we expected; since most of db4o C# code is converted from java (and from some historical reasons we convert from java 1.2 codebase which by the way is converted from java 1.5 code :) most of the collections used were the non-generic ones. This forced us to review the code and, when possible, use the generic version. In some cases the simplest solution was to introduce a non-generic collection (based on its generic counter-part) mimicking the native, non-generic, missing collection.
 
The last restriction in this list was the most frustrating one. db4o relies heavily on being able to load assemblies referenced by persisted objects and Silverlight security model doesn't allow arbitrary assemblies to be loaded explicitly. Actually (AFAIK) it is possible to load assemblies only if you specify a fully qualified assembly name, including its version. The problem with this approach is that db4o doesn't know about the assembly version (to be more precise db4o strips assembly versions away when storing objects). One possible solution would be to try to iterate over the already loaded assemblies and grab the reference from this list instead of trying to reload "already loaded" assemblies, but this brought us another frustration: this same security model disallows iterating over this list also (the decision to always load assemblies instead of applying the process described above was taken based on the fact that the performance impact of calling Assembly.Load() on "already loaded" assemblies is barely perceptible).
 
The solution we found (at least for now) relies on the fact that most applications know, at compile time, the assemblies declaring storable types. On Silverlight, every time db4o needs to get an assembly reference it'll raise a TypeReference.AssemblyResolve (static) event and will allow the application to resolve and return the required reference. The included sample shows a possible implementation for this. We are thinking on having a new class encapsulate the details of this pattern but we decided to postpone the introduction of this class until we better understand the restrictions and get some feedback on the design.
 
[edited on Jun / 23] Ashish Shetty, just pointed me to this page and explained that using this API we could workaround the assembly loading issue. We are gonne to investigate it soon.[/edited]
 
Current Status
 
As of today the following projects compile successfully under the Silverlight platform (see Db4o-Silverlight-2008.sln):
  • Core (Db4objects.Db4o)
  • Tests (Db4objects.Db4o.Tests)
    • No CS tests
    • No Migration tests
    • No Interoperability tests
    • No Linq / NQ tests
  • Optional (Db4objects.Db4o.Optional)
  • Db4oUnit
  • Db4oUnit.Extensions
Unfortunately successful compilation has very little to do with a software being usable / stable. Right now we have a subset of a full db4o test suite running against this platform. As we manage to run more of these tests we are sure that new issues will be found (for instance this one was caught when we started to run the first tests). 
 
 
Required changes to Applications
 
In order to make the discussion more practical I've written a very simple CRUD Silverlight application. Keep in mind that this is my first Silverlight application :), so please give me some feedback if you see that something smells bad :)
 
Before you continue reading I suggest you download the application source code and take a look at it (bellow you can see a screenshot)
 
 
Through this really innovative user interface (seriously, it's a break through :) it's possible to add a new person by entering its first name/last name and then pressing the "add" button. To query, fill in the fields you want to query on (first name and/or last name) and press "query" button. To delete an object, select it in the listbox and press "delete". Thats it :)
 
The most important aspect of this sample lies in the following lines:
 
        #region Assembly.Load() issue workaround

        private static readonly IDictionary _assemblyCache = new Dictionary();
        static App()
        {
            //_assemblyCache[AssemblyNameFor(typeof(Queue<>))] = typeof(Queue<>).Assembly;
            //_assemblyCache[AssemblyNameFor(typeof(List<>))] = typeof(List<>).Assembly;
            _assemblyCache[AssemblyNameFor(typeof(Person))] = typeof(Person).Assembly;
            _assemblyCache[AssemblyNameFor(typeof(Db4oFactory))] = typeof(Db4oFactory).Assembly;
           
            TypeReference.AssemblyResolve += (sender, args) =>  args.Assembly = _assemblyCache[args.Name];
        }

        #endregion
 
and
 
        private static IEmbeddedConfiguration Config()
        {
            IEmbeddedConfiguration config = Db4oEmbedded.NewConfiguration();
            config.File.Storage = new IsolatedStorageStorage();
            return config;
        }
 
in app.xaml.cs file.
 
Basically it just populates a dictionary mapping from an assembly name to the respective assembly reference and registers for TypeReference.AssemblyResolve events. Once this event gets raised we just look for the assembly name in our map. This is required as a workaround on assembly loading constraints.
 
Note that any type referenced by your model (stored objects) should have its assembly listed in this map otherwise bad things will happen to you :) Jokes apart, if you fail to list an assembly in this map db4o will handle types defined in this assembly as generic classes, i.e, classes that don't have the original class definition, and this will not work as expected in most cases. Since this is hardly the desired behavior, a possible improvement would be to throw an exception (inside AssemblyResolve event if the requested assembly could not be found in the map) instead of returning null.
 
Also note that we declared the Person's fields as public otherwise we'd get exceptions at runtime. We discussed these limitations on the "limitations" topic above.
 
 
Wrapping it up
 
In order to support Silverlight we're required to introduce changes into db4o (including some deprecated classes/methods that were wiped out). For developers, the most important requirements are to register for TypeReference.AssemblyResolve event in order to return assembly references and to use IsolatedStorageStorage, as shown in the sample.
Silverlight support is on the way, but even though we had a steady progress (it can be tracked through COR-1611 issue) we do have some homework to do before we can declare it officially supported. 
 
Based on our initial list of tasks we can see that currently we are working on porting / adapting the tests. As the next steps we're considering the following:
  1. Integrate Silverlight into the build (mainly run the tests in our continuous build)
  2. Finish porting tests
  3. Investigate performance issues
  4. Update documentation
  5. Write samples
  6. Publish Silverlight artifacts to the website
Of course, community involvement is a strong force that'll help us decide how much effort we shall put into this. So, if you are learning or already develop for the Silverlight platform we invite you to grab the db4o sources from our svn repository, try it with your projects and provide as much feedback as possible (reporting any issues, questions, suggestions, ideas, etc).
 
Finally, fell free to drop me a line at adriano at db4o dot com if you need any help to get started.
 
 [edited on Jun / 23] As I said, some days after publishing this post, Ashish Shetty got in touch to alert me about some issues; I'd like to thank you for taking the time :)[/edited]
 
Best.
 
Adriano