Dynamic Pipelines and Debugging DSL Transformations

I’ve been working on a dynamic transformation library for MetaSharp the last couple of weeks. I think I’m finally past the prototype phase and am almost ready to make it real. There is still lots of work for it to be actually usable but I think that the infrastructure is in place to enable me to solve all of the rest of the problems. Here are some samples of how I have it working so far:

A Simple Eval

int z = pipeline.Eval<int>("x * y", new { x = 3, y = 7 });

Delegate Compilation With a More Complex Context

public class MathContext
{
    public double x;
    public double y;

    public double sin(double d)
    {
        return Math.Sin(d);
    }

    public double cos(double d)
    {
        return Math.Cos(d);
    }
}

Func<MathContext, double> equation = pipeline.Compile<MathContext, double>("sin(x) * cos(y)");
double result = equation(new MathContext { x = 3, y = 7 });

I want to allow you to add white-listed namespaces to the pipeline also to allow “new” objects and static methods to be called using those namespaces. And of course the actual code can be more complex then one line expressions. The new expressions in .net 4 include all statements as well. Hopefully we’ll get better support for dynamic types in the future, but I may have some tricks up my sleeve for doing that as is, we’ll see.

Anyway, one of the things I needed to do to get it to work was to create a more robust object as a result of a transformation. Previously you’d get in some object then simply yield whatever you wanted to transform it into. This was ok for the CodeDom where you could stuff the original AST nodes into a metadata dictionary on any CodeObject but it doesn’t work for the DLR objects. At some point you need to know what a node came from in order to understand the context of how to apply it in a following transformation. For example, a method invocation node, needs to know if what it is invoking is a property, field or method reference and do something different for each case.

So, now when you yield transformed objects it stuffs them into a Transform<T>. Which contains the original node, the yielded transformation nodes and all child Transforms. The end result is 3 trees, the original AST tree, the new AST tree and a tree showing the relationship between the two trees.

This is necessary to actually do transformations but one of the cool side effects I want to play around with is that the relationship tree (the tree of Transform<T> objects) could be really cool to visualize. I really want to make a Debugger Visualizer where you can pop open a dialog and see a visual representation of the transformation. I’m envisioning something like this:

Where as your mouse moves over a node in the original tree you show the nodes it transformed into. In this image I have a AST representing a Song DSL. When run through the pipeline it transforms into CodeDom objects (or whatever but CodeDom in this case). Here you can see the Note is being transformed into objects that would generate code that looks like this “this.Play(key, octave, duration)”. As you move up to the bar you’d see a more complete representation of the generated tree and as you move up to Song you’d see the entire tree.

That is the ultimate goal, in reality, to do this transformation takes several steps and it would be more complicated than this. But now that we have all of the Transform trees you could track these sorts of things and display it visually like this. A tool such as this could be invaluable for debugging complex transformations!