MetaSharp – A CodeDom based Template Engine using MGrammar

I’ve been working on a tangential project related to NBusiness for a couple of weeks now and I just wanted to take a moment to get a few of my thoughts out. The project I have been working on I am tentatively calling “MetaSharp” for now. It’s been fun and educational but hopefully it will have real usefullness when it is done. I wanted to have a fully working example before I publicly posted the code (since it’s basically prototype quality right now) but if anyone is interested in seeing what I have so far feel free to ask and I’ll hook you up somehow.

I’ll try to start at the beginning to justify my rationale for creating this strange project. I’ve been working on NBusiness for quite a while now and while I’ve really had NBusiness “working” almost all along I have never quite been able to get it where I want it to be (complete). If I had to sum up the entire process of working on NBusiness into one sentence it would be “creating a DSL is hard”. That’s an understatement frankly. Let me see if I can lay out the various layers required for DSL creation.
·         Domain Objects
·         Parser
·         Compiler
·         Template Engine
·         Build Integration
·         Tooling Support
The first three items are actually relatively easy and pretty fun. This is what we all know how to do, write code to parse strings and stick values into objects. No problem. It turns out the next three layers which really provide the fit, finish and ultimate usability of your DSL are not easy at all. Build integration isn’t really that bad actually but tooling integration can be a real bear. In the case of a DSL you really want syntax hilighting and intellisense and nice IDE integration for file templates and things like that. Maybe a few additional context menus in your IDE and such. For me I have been trying to integrate into Visual Studio and I can officially say that I have sunk well over half my time into that aspect alone and it has been one of the hardest things I have ever tried to do. Visual Studio is also architected such that I had to completely redo my parser and compiler to be compatible with the needs of Visual Studio. Very painful.
But what is really hanging me up now is what I consider to be a large gap in the .NET  DSL world and that is a suitable templating engine. By templating engine I mean something that can take metadata and translate it into code.
I mean we have a bunch out there but they’re all (as far as I know) effectively giant string builders. They suffer from Tag Soup and and are bound strongly to a specific language implementation. For NBusiness I want to support side by side integration with any .NET language, C# or VB or Python or whatever. And re-creating all of these templates for every language is not an option. It’s too much upfront work and it’s too much long term maintenance. I absolutely need templates that are based on the CodeDom so I can be language agnostic… But if you’ve ever tried to use the CodeDom you know how hard it is to work with. Because of this users are very unlikely to actually make their own templates (which is almost always necessary) and when they do it is a very painful process. So I’ve been stuck in this cunundrum for quite a while, how can you make a template engine that is both based on the CodeDom but has the ease of use of a string builder?
Enter MGrammar. Using MGrammar I have found a way to define a DSL for generating code. This DSL turns out to be a full fledged programming language in and of itself with the caveat of being restricted only to that which is CLS compliant. I have combined this DSL with the capability to create templates (to extend the language, similar to macros in Boo) and databinding similar to what you have in XAML. The end result allows you to do something similar to this:
namespace Example:
    import System;
 
    template One:
        public class {Binding Name}:
            {SequenceBinding Items, Template=Two}
        end
    end
 
    template Two:
        private field {Binding Type} _{Binding Name};
        public property {Binding Type} {Binding Name}:
            get:
                return this._{Binding Name};
            end
            set:
                this._{Binding Name} = value;
            end
    end
end
(This is just an example, the end result might not actually be exactly this syntax)
Which when compiled will generate a class called OneTemplate that inherits from Template and returns a CodeTypeDeclaration object from it’s Generate method. Extensions such as the BindingExtension show here can be custom objects to extend behaviors but in this case it binds the name of the class to the Name property (or Name sequence node of an MGraph tree) of the provided metadata.
Technically you could write your entire project in pure MetaSharp code but more likely you will write all of your static classes in your rich language of choice and simply use MetaSharp to define templates. Since this is all compiling down to CodeDom objects I have cooked up some MSBuild tasks that simply translate those objects into the code of the project the files exist in. You could share this same file in a VB or C# project and it would compile to the same thing in both assemblies.
Currently I am working on a prototype using the Song example from the MGrammar sample code that will allow you to write songs that generate song classes using templates like these. It’s almost working… the CSharpCodeProvider is throwing a random NullReferenceException with no useful error messages. Which is one reason why a DSL like this is helpful, it should be able to abstract away the pain of working directly with the CodeDom.

LinqBinding Markup Extension

Earlier today on Twitter Aaron Erickson was berating the inability to write linq statements directly in Xaml. While I’m not totally convinced that is a good idea to begin with I decided to come up with a little proof of concept to see if it’s even feasible. I was able to get about half way to what I would like to see by creating a custom markup extension called LinqBinding. The result allows you to create bindings such as this:

<sys:String x:Key=”query”>id &gt; 1 &amp;&amp; id &lt; 4 || name==”six”</sys:String>
<ListBox ItemsSource=”{e:LinqBinding {StaticResource query}, Binding={Binding}}” />
The query in the string is the linq statement you would put into the where clause normally. I was unable to figure out how to use a Binding statement to a TextBox instead of having the hard-coded string but I’m pretty confident it is possible somehow (I’ll leave that for another blog post for now).
In order to accomplish this I used the code found in the Dynamic Linq examples talked about by Scott Guthrie in his blog.
Here is the sample: LinqExtensionSample.zip

LambdaExpression.CompileToMethod … not nearly as cool as I had hoped.

I was messing around with the new dynamic expressions in C# 4 (found in System.Linq.Expressions). One new thing I noticed was the method CompileToMethod on LambdaExpressoin. The first parameter is a MethodBuilder so I got really excited, finally an easy way to create dynamic methods on Types!

Wrong, turns out the method has to be Static, also it cannot accept, as a parameter, the Type the method is being built on. So this basically completely negates the entire purpose. At this point you might as well simply compile it to a delegate and use that instead. Bummer.

Here is an explanation of why:

After much investigation, it turns out this is by won’t fix for CLR 4.0

You can’t use expression trees to model instance methods or constructors. First problem: you can’t create the “this” parameter, because all you have in hand is a TypeBuilder, but that is not the final type. We can create a delegate with the TypeBuilder, but CLR won’t let us create a LambdaExpression with that delegate (because the type is not finished). You can workaround that by making the “this” parameter be of type object, then you end up with meaningless casts. Worse, calls to your own methods don’t work, because MethodBuilder doesn’t implement enough of reflection for expression trees to do their normal sanity checks.

DynamicMethods run into their own problems. When emitting into a DynamicMethod, we can’t add our Closure argument to the signature of the DynamicMethod. Without a closure,

  • DynamicMethods run into some serious limiations:
    They can’t have nested lambdas at all (due to a CLR limitation: can’t ldftn to a DynamicMethod).
  • Some things that are O(1) or amortized O(1) become O(N), such as RuntimeVariables and Switch on strings. This is really sneaky for the user who won’t expect things to suddenly be slower.

This needs work we plan for ETs v3, and the design around the support will likely change.

One potential work around I have contemplated but haven’t gotten up the energy to try would be to try to generate a matching interface first, then pass that around as a parameter to the static method. You’d have to give access to all fields through explicitly implemented properties though, destroying encapsulation in the process. So if suppose you wanted to generate something like this:

public class Test
{
    private int foo;

    public int Bar()
    {
        return this.foo;
    }
}

Instead you would generate something like this:

public interface ITest
{
    int foo { get; set; }

    int Bar();
}

public class Test : ITest
{
    private int foo;

    int ITest.foo
    {
        get { return this.foo; }
        set { this.foo = value; }
    }

    public int Bar()
    {
        return Test.Bar(this);
    }

    public static int Bar(ITest test)
    {
        return test.foo;
    }
}

The benefits of using the Linq expressions to build your method bodies quickly diminish with this sort of workaround however. You might as well just go back to the ILGenerator (*shudders*). This would probably work because the static Bar method can accept a parameter of a Type already dynamically created.

Asynchronous Programming With Iterators

I just saw this on the visual Studio Start page blogs.msdn.com/charlie/archive/2008/12/03/jeff-richter-video-on-asynchronous-programming-and-his-power-threading-library.aspx

It’s a pretty interesting video worth checking out. It seems like one of those good ideas that translate into a very small library that can be used to do powerful things.

To sum it up briefly (in case you can’t afford to watch the 20 minute long video) Jeff has realized that the state machine-esque code generated for you when you create an iterator in C# (i.e. “yield return”) is exactly the same type of code you’d need to write to comply with the standard .NET asynchronous programming model. Therefore writing your asynchronous code inside of an iterator (with help from his AsyncEnumerator class) allows you to write code in a synchronous format and rely on the C# iterator expansion capabilities to translate that into asynchronously compatible code.

As clever and interesting as this is I’m not happy about the (mis)use of the yield syntax to work around a compiler limitation. I have submitted this idea to the Boo Lang user group because this seems like the perfect example of the potential power of meta programming. In the case of C# you have some macro expansion for certain baked in keywords (using, lock, yield, etc.), and it just so happens that the expansion of the yield keywords is exactly what you need to match some other pattern but I believe you could do this in Boo in a much cleaner way.

Imagine having an actual “async” keyword and an “wait” keyword? It would do nearly the same thing but would make it all feel so much cleaner. Anyway, food for thought.

Consider