Language Workbench in Visual Studio

image

This is far from polished and complete but here you can see the grammar defined in foo.g being dynamically compiled and used to parse the text in the grammar console. You can see how the [Keyword] attribute is used to generate syntax highlighting in that grammar window also, as well as automatic error generation for the errant ‘!’.

Next up is visualization of the node graph produced by the parse as well as error information. Very promising progress though!

List<T> is dead, long live LINQ

Well, ok, maybe not completely dead. It has it’s uses but for the most part I find myself using an actual list less and less. I do end up using ObservableCollection<T> for ViewModels still, of course, but that can’t be avoided.

I do find myself slowly banishing it from my daily coding habits however, thanks to LINQ. I am finding that the covariant interface IEnumerable<T> is almost always a better choice as a return value for methods and properties and with the help of the higher order set functions provided along with LINQ there is nothing you can’t do without List<T> that you could have done with it.

Also, I would like to mention that when I am talking about LINQ I am talking exclusively about the extension methods found in System.Linq.Enumerable not the actual internal DSL syntax in C# / VB. I almost never use that syntax since it can’t express all of the functions I need and is sometimes hard to debug.

I did have to add one extension method to help ease general use of LINQ however.

public static class EnumerableExtensions
{
    public static IEnumerable<T> ToEnumerable<T>(this T @object)
    {
        if (@object == null)
            return System.Linq.Enumerable.Empty<T>();

        var enumerable = @object as IEnumerable;
        if (enumerable == null)
            return new[] { (T)@object };

        return enumerable.Cast<T>();
    }
}

This allows you to ensure than any object can be used as an IEnumerable<T>. With this in hand you can replace any method on List<T>. Here is an example of a bunch of methods on List<T> and their equivalent LINQ translations.

var a = 1;
var b = 2;
var c = 3;

var list = new List<int>();
var set = Enumerable.Empty<int>();

list.Add(a);
set = set.Concat(a.ToEnumerable());

list.Remove(a);
set = set.Where(i => i != a);

list.AddRange(new[] { a, b });
set = set.Concat(a.ToEnumerable()).Concat(b.ToEnumerable());

list.Sort();
set = set.OrderBy(i => i);

list.Contains(a);
set.Any(i => i == a);

var count1 = list.Count;
var count2 = set.Count();

list.FindAll(i => i == a);
set.Where(i => i == a);

list.IndexOf(a);
set.TakeWhile(i => i == a).Count();

list.FindLast(i => i == a);
set.Last(i => i == a);

list.GetRange(0, 2);
set.Skip(0).Take(2);

list.Insert(1, c);
set = set.Take(1).Concat(c.ToEnumerable()).Concat(set.Skip(1));

list.Reverse();
set = set.Reverse();

list.TrueForAll(i => i == 0);
set.All(i => i == 0);

Note that in order to get some of the features of List<T> you have to compose various LINQ functions together, such as Insert. At first this may seem overly verbose but the benefit is an incredible flexibility. There is nearly no limit on what you can express with LINQ because of this composability (also you could create extensions that simply wrap up the complex composition with a neater signature). List<T> has a few more methods that I will leave up to you to translate but LINQ has many more functions and function composition possibilities that I would highly recommend discovering.

Executing the above code and printing list and set results in this output.

list: { 2 3 1 }
set: { 2 3 1 }

When you use LINQ your set is immutable at every point in time, which is very helpful in case you would like to try to use the Parallel Extensions (aka PLINQ). One other interesting aspect of the LINQ extensions is deferred execution. This means that none of my set operations above are even executed until I iterate over the set in my print function. This allows me to change the value of ‘a’, reiterate and receive a different result. This can be confusing if you’re unaware of it but it can be a huge win if you understand it well. This is all possible because of the functional nature of LINQ and (as best as I can understand) is an example of a Monad in .NET. And, of course, coroutines in C# compliment LINQ very nicely as well (aka yield return). I think it would be very interesting to experiment with a language where every variable is essentially treated as a set, NULL is not possible but empty sets are. An object is simply a set of 1. It would be interesting.

Another very cool aspect of LINQ is Lambda Expressions and how they are related to Expression Trees and the DLR. It’s extremely powerful and they are intimately tied together.

Also, do check out IEnumerable<T>’s mirrored twin: IObservable<T>.

Building Dynamic Types in C# using IronPython

I have created a small routine that allows you to use the IronPython runtime to create dynamic Types without requiring you to write any Python code.

The normal use case for IronPython in C# requires you to provide the IronPython runtime with some source code files, which will essentially eval and return to you a context which you can interact with dynamically. With some help on the forums I was able to find a way to provide an IronPython AST to the runtime instead of source code. This will be very useful for MetaSharp, since I already have a lot of mechanisms for creating an AST I will only need to simply create a transformer that will change an existing AST into IronPython AST and you can suddenly compile any DSL into arbitrarily complex object hierarchies at runtime.

One of the reasons for this is that I want to create a language workbench tool, where you can experiment with your transformations dynamically without having to recompile in the traditional sense. Also, it just gives you more options at runtime. This should be very handy.

Here is an example of what I’m talking about. My example code is very simple, create a class that implements a simple .net interface then create an instance of that class and call it’s method. Here is the actual python version:

#sample.py
from ConsoleApplication4 import IExample

class Example(IExample):
    def Do(self):
        return "hello python"

Executing and calling this at runtime in C# is quite easy to do out of the box actually.

var runtime = Python.CreateRuntime();
runtime.LoadAssembly(typeof(IExample).Assembly);
dynamic python = runtime.UseFile("sample.py");
            
var example = (IExample)python.Example();
Console.WriteLine(example.Do());

The rub here though is that you need that python source file laying around somewhere. Or you could generate code, put it in a temp file and use that instead but nonetheless, the creation of the dynamic object is done by creating python code only. My problem was to figure out how to bypass most of the parsing semantics and to provide an AST directly to the runtime instead of code files. The AST could be created by a MetaSharp transform step.

So instead you can now do this:

var runtime = Python.CreateRuntime();
runtime.LoadAssembly(typeof(IExample).Assembly);

var iexampleImport = typeof(IExample).Import();

var classDefinition = Dlr.Class(
    "Example",
    iexampleImport.ToEnumerable(),
    Dlr.Function(
        "Do",
        Dlr.Self().ToEnumerable(),
        "hello python!".AsConstant().Return()));

dynamic python = runtime.Compile(
    iexampleImport,
    classDefinition);

var example = (IExample)python.Example();
Console.WriteLine(example.Do());

I won’t go into all of the extension methods I created (see link to source at the end) but suffice it to say that the Dlr.Class method is creating a ClassDefinition object. Passing these into the runtime.Compile method is where all of the magic happens.

Figuring out what to do in the Compile extension method was the real hard part. It’s not well documented and some of the side effects are downright bizarre, clearly the API was not intended to be (mis)used in this way.

Fortunately for you, all you have to do is download the file below and you’re off creating dynamic Types using the DLR!

http://cid-dfcd2d88d3fe101c.skydrive.live.com/embedicon.aspx/blog/justnbusiness/DlrCompile.zip

Coercing Types and Unloading Assemblies

I was doing a little experimentation recently related to building dynamic assemblies in .NET. One thing I found interesting was that if you create an instance of a dynamic Type then use the “dynamic” keyword to consume it, your dynamic assembly will never be unloaded.

That sort of defeats the purpose if you ask me. I was back to using reflection to call my dynamic objects. Not fun.

So I whipped up a quick dynamic coercion routine. Basically what it does is takes any instance and any interface and builds a wrapper object that implements the interface and calls a matching method on the instance. The instance does not need to implement the interface or even know about it. The coercion method will give the appearance of casting the instance into a interface it doesn’t really implement.

var instance = CreateDynamicAdder();
var adder = instance.Dynamic().Coerce<IAdder>();

Dynamic() is an extension point for type System.Object. Coerce is a extension method:

public static T Coerce<T>(this IDynamicExtensionPoint extensionPoint)
{
    if (!typeof(T).IsInterface)
        throw new ArgumentException("Type T must be an interface.");

    if (extensionPoint == null)
        throw new ArgumentNullException("extensionPoint");

    var assemblyBuilder = Dynamic.CreateDynamicAssembly();
    var wrapperType = BuildWrapperType(
        assemblyBuilder, 
        typeof(T), 
        extensionPoint.Instance.GetType());
    return (T)Activator.CreateInstance(wrapperType, extensionPoint.Instance);
}

So in the first code snippet I get an object that has a method I want to call that I know looks like this:

int Add(int x, int y)

I don’t know the exact Type and I don’t have any interfaces I can cast it into, but I know that this method is there, possibly through documentation, convention or debugging. The Coercion routine will allow me to call dynamic objects with an illusion of strong typing.

So I create an interface that matches the signature I am expecting.

public interface IAdder
{
    int Add(int x, int y);
}

And coerce the instance into this interface. What’s interesting is that when I do it this way, both the dynamic assembly creating the adder as well as the dynamic assembly creating the wrapper are unloaded when I drop references to them.

static void Main(string[] args)
{
    var numberOfAssemblies = AppDomain.CurrentDomain.GetAssemblies().Count();
    Console.WriteLine("Starting assembly count: " + numberOfAssemblies);

    var instance = CreateDynamicAdder();
    var adder = instance.Dynamic().Coerce<IAdder>();
    int z = adder.Add(7, 11);
    Console.WriteLine("Add result: " + z);
    instance = null;
    adder = null;

    numberOfAssemblies = AppDomain.CurrentDomain.GetAssemblies().Count();
    Console.WriteLine("Assembly count before collect: " + numberOfAssemblies);

    GC.Collect();
    GC.WaitForPendingFinalizers();

    numberOfAssemblies = AppDomain.CurrentDomain.GetAssemblies().Count();
    Console.WriteLine("Assembly count after collect: " + numberOfAssemblies);

    Console.ReadKey(true);
}

Here is the resulting screen shot, proving the dynamic assemblies are unloaded.

image

Download the full sample here:

http://cid-dfcd2d88d3fe101c.skydrive.live.com/embedrowdetail.aspx/blog/justnbusiness/DynamicCoerce.zip

Dynamically Compiled Lambdas vs. Pure Reflection

I’ve spent a little time messing around with expression trees in .net 4 over the last couple of weeks and so I decided to put together a test comparing the performance of compiling a lambda expression on the fly vs. simply using basic reflection to invoke a method. You will probably see some of this in a patch to CSLA for Silverlight in the future, since it is currently calling pure reflection. Regular CSLA uses DynamicMethod and uses the ILGenerator to build a similar call to a method. And actually, looking in reflector, that is exactly what the LambdaExpression.Compile() method is doing also, its is simply easier to build up the code you are trying to compile using the Expression tree objects.

http://cid-dfcd2d88d3fe101c.skydrive.live.com/embedrowdetail.aspx/blog/justnbusiness/Reflection%20vs.%20Lambda.cs

Setup

MethodInfo barMethod = typeof(Foo).GetMethod("Bar");
MethodInfo bazMethod = typeof(Foo).GetMethod("Baz");

int iterations = 100000;

Foo foo = new Foo();
object[] p1 = new object[] { 100, "blah" };
object[] p2 = new object[] { 100, "blah", Guid.Empty };
public class Foo
{
    public static int staticCount = 0;
    public int count = 0;

    public void Bar(int x, string y)
    {
        count++;
    }

    public static void Baz(int x, string y, Guid z)
    {
        Foo.staticCount++;
    }
}

Rather than doing all of this in the body of each test I wanted them both to have everything setup and even, in both cases we are getting the method to call with basic reflection. The Foo class is the class we will be running the test on and the fields are there simply to give the method body something to do so the compiler doesn’t optimize it out.

Invoke via Reflection

DateTime start = DateTime.Now;
for (int x = 0; x < iterations; x++)
{
    barMethod.Invoke(foo, p1);
    bazMethod.Invoke(null, p2);
}
DateTime end = DateTime.Now;
Console.WriteLine("Time with reflection: {0}.", (end - start).TotalMilliseconds);

Here we’re very simply calling the Invoke member of MethodInfo. The first method is a member method the second is static.

Invoke via Delegate

start = DateTime.Now;
Action<object, object[]> call = null;
Action<object, object[]> staticcall = null;
for (int x = 0; x < iterations; x++)
{
    if (call == null)
    {
        call = CreateCaller(barMethod);
        staticcall = CreateCaller(bazMethod);
    }

    call(foo, p1);
    staticcall(null, p2);
}
end = DateTime.Now;
Console.WriteLine("Time with dynamic lambdas: {0}.", (end - start).TotalMilliseconds);

Here we are building two delegates on the first iteration and calling it directly for subsequent iterations. I’m using Action<object, object[]> to allow me to very generically call any method. The more you know about the method you are trying to call the nicer the delegate signature could be (and probably a little faster) but this is also designed to be extremely general.

Also, note that I am using a null check for caching but in reality you would probably have something more complex like a dictionary. This would add a little extra overhead onto the execution time.

Create Caller via Expression Builder

Here is where the real interesting part takes place, here we are using the System.Linq.Expressions namespace to build a lambda expression dynamically based on the MethodInfo object passed in. The build lambda is compiled into a Delegate designed specifically to call that method. There are two types of methods built, one for member methods and one for static methods.

using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
private static Action<object, object[]> CreateCaller(MethodInfo method)
{
    int index = 0;
    var p1 = Expression.Parameter(typeof(object), "instance");
    var p2 = Expression.Parameter(typeof(object[]), "parameters");
    var parameters = from p in method.GetParameters()
                     select Expression.Convert(
                            Expression.ArrayAccess(
                            p2,
                            Expression.Constant(index++)),
                            p.ParameterType);

    Expression instanceCheck = null;
    Expression call = null;
    if (method.IsStatic)
    {
        instanceCheck = Expression.IfThen(
            Expression.NotEqual(
                p1,
                Expression.Constant(null)),
            Expression.Throw(
                Expression.New(
                    typeof(ArgumentException).GetConstructor(
                        new Type[] { 
                                typeof(string),
                                typeof(string) }),
                    Expression.Constant(
                        "Argument must be null for a static method call."),
                    Expression.Constant("instance"))));

        call = Expression.Call(
            method,
            parameters);
    }
    else
    {
        instanceCheck = Expression.IfThen(
            Expression.Equal(
                p1,
                Expression.Constant(null)),
            Expression.Throw(
                Expression.New(
                    typeof(ArgumentNullException).GetConstructor(
                        new Type[] { typeof(string) }),
                    Expression.Constant("instance"))));
        call = Expression.Call(
                    Expression.Convert(p1, method.DeclaringType),
                    method,
                    parameters);
    }

    var lambda = Expression.Lambda<Action<object, object[]>>(
            Expression.Block(
                instanceCheck,
                call),
            p1,
            p2);

    return lambda.Compile();
}

Results

When running the test application here are the results:

Time with reflection: 391.0391.
Time with dynamic lambdas: 15.0015.

Not surprisingly the dynamic lambdas vastly outperform standard reflection. But please note that this is in Milliseconds and is after 100,000 iterations. This sort of optimization is really only needed on areas where reflection is used heavily and frequently.

Action vs. Delegate

One of the interesting things I learned from this test was the difference between compiling a strongly typed delegate (such as System.Action) and a System.Delegate. Simply changing my lambda test code to this:

for (int x = 0; x < iterations; x++)
{
    if (call == null)
    {
        call = CreateCaller(barMethod);
        staticcall = CreateCaller(bazMethod);
    }

    call.DynamicInvoke(foo, p1);
    staticcall.DynamicInvoke(null, p2);
}

(notice the .DynamicInvoke() instead of direct invocation)

Changes the tests to yield the results:

Time with reflection: 393.0393.
Time with dynamic lambdas: 925.0925.

Which is dramatically worse than standard reflection! So the trick is that you really need to get a strongly typed delegate for it to be worthwhile at all to call. My above experiment attempts to solve the problem by using a general purpose Action<object, object[]> (which should probably be a Func<object, object, object[]> now that I think about it) and uses casting to resolve all of the problems with calling the method incorrectly but the fact of the matter is that the more you know about the method signature the better you will be able at optimizing it. For example, if you are able to know that there will never be any return values, that is a big optimization, or if you know a common base class for the instance method or if there will never be a static method then you can dramatically tweak this even further. The general solution is much more complex however.

Dynamic Pipeline in MetaSharp

I’ve been working on a prototype of a Dynamic Pipeline for MetaSharp using the new Linq Expressions in .NET 4. I’m pretty darn excited about it, here is what I have working so far:

[Test]
public void BinaryVariablesTest()
{
    var result = this.pipeline.Eval<int>("x * y", new { x = 3, y = 7 });
    Assert.AreEqual(21, result);
}

What I’m doing here is compiling the string as MetaSharp code then using a node visitor to generate the appropriate linq expressions. The trick here is the context parameter being passed in. This object contains all of the properties and methods allowed to be called from the expression, essentially it’s the “this” parameter. To get this to work you can use the handy dandy ConstantExpression. Here is how I’m doing it:

public class ReferenceExpressionVisitor 
    : TransformVisitor<linq.Expression, ReferenceExpression>
{
    protected override IEnumerable<linq.Expression> Visit(
        IEnumerable<linq.Expression> items, 
        ReferenceExpression visitedObject)
    {
        IDynamicContextService parameterService = 
            this.ServiceProvider.Locate<IDynamicContextService>();
    
        yield return linq.Expression.PropertyOrField(
            linq.Expression.Constant(parameterService.Context),
            visitedObject.Name);
    }
}

The constant allows us to call properties or fields on the context object directly. You should even be able to attach lambda expressions to fields so you can call those as well. I have to do some restructuring in order to enable everything like Method calls and whatnot but it should all be doable. Very sweet. Also the dynamic pipeline will have to have some type of caching provider, clearly you wouldn’t want to rebuild these expressions every single time you try to execute it. Optionally I could just give back a delegate instead of executing it directly, then let the caller do all the caching. That sounds easier and more flexible actually now that I think about it.

What’s cool about this though isn’t the fact that you can compile expressions dynamically (which is pretty cool but there are others out there already) but the fact that you will be able to dynamically compile DSLs… which reduce to dynamic expressions. Imagine this:

when truck is late:
  apply discount(.1);

when truck is early:
  apply bonus(.1);

when truck cost > $10,000:
  notify

You compile your DSL into MetaSharp nodes, transform it into Common language nodes then transform that into dynamic linq expressions. Very sweet! You might transform this into something like:

if(truck.late)
{
    context.discount(.1);
}
if(truck.early)
{
    context.bonus(.1);
}
if(truck.cost > 10000)
{
    context.notify();
}

Where you’d have a custom ShippingContext object with all of the methods and properties you wanted to expose in your DSL. This would be really handy for creating all sorts of systems that have the potential to have rapidly changing rules and also, potentially, enable business analysts to author their own rules.

The Illusion of Strongly Typed XML with C#

This is essentially a redo of an earlier blog post of mine on how to create the illusion of a strongly typed xml in Boo using Duck Typing, except this time we’re using the dynamic keyword in C#.

http://cid-dfcd2d88d3fe101c.skydrive.live.com/embedrowdetail.aspx/blog/justnbusiness/XmlBuilder.cs

Here is an example of how it might look to use such a thing.

dynamic builder = new XmlBuilder();
builder.Main.One = "Hello";
builder.Main.Two = "World";

Console.WriteLine(builder.ToString());

Console.WriteLine(
    "{0} {1}!",
    builder.Main.One,
    builder.Main.Two);

builder = new XmlBuilder();
builder.Main["One"] = "Hello";
builder.Main["Two"] = "World";

Console.WriteLine(builder.ToString());

Console.WriteLine(
    "{0} {1}!",
    builder.Main["One"],
    builder.Main["Two"]);

XmlBuilder is a class I created myself, it inherits from DynamicObject and it intercepts TryGetMember, TrySetMember, TryGetIndex and TrySetIndex in order to allow you to navigate an internal XmlDocument object. Here is the output when the above is run.

<Main><One>Hello</One><Two>World</Two></Main>
Hello World!
<Main One=”Hello” Two=”World” />
Hello World!

Notice the use of properties corresponds to drilling into elements and the use of indexers corresponds to drilling into Attributes. This is just a proof of concept but you could see how this sort of thing would be useful.

Here is the complete XmlBuilder class that makes it all happen.

public class XmlBuilder : DynamicObject
{
    private Func<string, XmlElement> createElement;
    private Func<string, XmlAttribute> createAttribute;
    private XmlNode node;

    public XmlBuilder()
    {
        this.node = new XmlDocument();
        this.createElement = s => ((XmlDocument)this.node).CreateElement(s);
        this.createAttribute = s => ((XmlDocument)this.node).CreateAttribute(s);
    }

    private XmlBuilder(
        XmlNode node, 
        Func<string, XmlElement> createElement, 
        Func<string, XmlAttribute> createAttribute)
    {
        this.node = node;
        this.createElement = createElement;
        this.createAttribute = createAttribute;
    }

    public override bool  TryGetMember(GetMemberBinder binder, out object result)
    {
        XmlNode found = this.node[binder.Name];
        if (found == null)
        {
            found = this.createElement(binder.Name);
            this.node.AppendChild(found);
        }

        result = new XmlBuilder(found, this.createElement, this.createAttribute);
        return true;
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        XmlNode found = this.node[binder.Name];
        if (found == null)
        {
            found = this.createElement(binder.Name);
            this.node.AppendChild(found);
        }

        found.InnerText = value.ToString();
        return true;
    }

    public override bool TryGetIndex(GetIndexBinder binder, object[] indexes, out object result)
    {
        if (indexes.Length != 1)
        {
            throw new InvalidOperationException("You may only specify a single index.");
        }

        XmlNode found = null;
        if (indexes[0] is string)
        {
            string name = (string)indexes[0];
            found = this.node.Attributes[name];
            if (found == null)
            {
                found = this.createElement(name);
                this.node.AppendChild(found);
            }
        }
        else if (indexes[0] is int)
        {
            int index = (int)indexes[0];
            found = this.node.Attributes[index];
        }

        if (found != null)
        {
            result = new XmlBuilder(found, this.createElement, this.createAttribute);
            return true;
        }
        else
        {
            result = null;
            return false;
        }
    }

    public override bool TrySetIndex(SetIndexBinder binder, object[] indexes, object value)
    {
        if (indexes.Length != 1)
        {
            throw new InvalidOperationException("You may only specify a single index.");
        }

        XmlAttribute found = null;
        if (indexes[0] is string)
        {
            string name = (string)indexes[0];
            found = this.node.Attributes[name];
            if (found == null)
            {
                found = this.createAttribute(name);
                this.node.Attributes.Append(found);
            }
        }
        else if (indexes[0] is int)
        {
            int index = (int)indexes[0];
            found = this.node.Attributes[index];
        }

        if (found != null)
        {
            found.InnerText = value.ToString();
            return true;
        }
        else
        {
            return false;
        }
    }

    public override string ToString()
    {
        return this.node.InnerXml;
    }
}