Meta Syntax Ramblings

I’ve been hung up for a while here on simplifying my meta syntax for Meta# since I am not real happy with the way my current prototype works. What I should say is that I have, so far, gone with a purely declarative syntax that ends up very verbose, much like XAML. This isn’t very fun for code that is supposed to be user readable.

I’m thinking of changing the {…} syntax to always mean binding to the model and [|…|] to be meta code. I am also thinking of creating a special expand keyword that allows you to switch into a code generation mode. For example, something like this:
template Blah:
                int count = 0;
                expand f in {Foo} where {Bar as int} < 100 && count % 2 == 0:
                                private field {Type} _{Name};
                                [| count++; |]
                end
end
Which is an ugly example but you get the point. So inside of a template you are writing literally generated code unless you have an expand statement. The expand statement would allow you to specify some things you want to expand upon and an optional where clause to turn sections of the template into meta code. If you want to inject logic in the middle of an expansion block you would use the [|…|] operators to switch back into generating literal code. This is essentially the opposite of Boo, where you write everything in literal code then use the [|…|] to indicate meta blocks rather than literal logic. The sample above would generate to code such as this:
public class BlahTemplate : Template
{
protected override IEnumerable<CodeObject> Generate(object Blah)
{
    int count = 0;
    foreach(object f in Binder.Bind(“Foo”, Blah))
    {
      if(Binder.Bind<int>(“Bar”, f) < 100 && count % 2 == 0)
      {
         string code = “private field {0} _{1};”;
         yield return Compile(string.Format(code, Binder.Bind(“Type”, f), Binder.Bind(“Name”, f)));
         code++;
      }
    }  
}
}
(Except it would be a for block not foreach due to limitations in the CodeDom)
I think I like this a little better (still deciding) because most of the time you won’t need anything other than binding statements inside of your expansion block and then you end up with very clean looking template code. The other thing I like is that using the {…} purely for binding makes it a lot simpler. I was encountering an issue where the property you are trying to bind to was actually a string due the way MGrammar works but you want it to be treated as an int. Doing it purely as a Binding syntax makes this a lot simpler I think {Property(.SubProperty)* (as Type)?}. The only question that concerns me is if you’ll ever actually need anything other than these keywords to solve any problem. I suppose I can always add more…
So if you were doing this for the song sample it might look like this:
template Song:
expand s in {Song}:
    public sealed class {Name} implements Song:
      public constructor:
        expand b in {Bars}:
          this.bars.Add(new Bar([|{Note1}|], [|{Note2}|], [|{Note3}|], [|{Note4}|]);
        end
      end
    end
end
end
The meta statement surrounding the bound notes would be because the Note is actually a string but what you really want is a PrimitiveExpression(string). Using the meta statement syntax would force the string to be interpreted as a code block instead of string. I might be able to figure out a way to not need that but as of now I think it’s necessary.
The other weird thing is how to handle mixed binding expressions. For example this is easy to figure out:
public field {f.Type} {f.Name};
But how do you handle this:
Public field {f.Type} _{f.Name};
Here, for the fields name we have a mix of a string and a binding expression where as in the first sample it was just a binding expression. What I’m thinking is to expand the syntax for a Binding expression to be any token that contains {…} rather than starts and ends with it. It will basically translate everything inside of the brackets into a string format to a Binder call such as:
_{f.Name} -> string.Format(“_{0}”, Binder.Bind(“Name”, f))
Here the bind method knows how to get values out of basic objects with properties using reflection or MGrammar sequence elements using the MGrammar API.
Also, if anyone has something better than “: end” for blocks other than “{ }” please let me know.

MetaSharp – A CodeDom based Template Engine using MGrammar

I’ve been working on a tangential project related to NBusiness for a couple of weeks now and I just wanted to take a moment to get a few of my thoughts out. The project I have been working on I am tentatively calling “MetaSharp” for now. It’s been fun and educational but hopefully it will have real usefullness when it is done. I wanted to have a fully working example before I publicly posted the code (since it’s basically prototype quality right now) but if anyone is interested in seeing what I have so far feel free to ask and I’ll hook you up somehow.

I’ll try to start at the beginning to justify my rationale for creating this strange project. I’ve been working on NBusiness for quite a while now and while I’ve really had NBusiness “working” almost all along I have never quite been able to get it where I want it to be (complete). If I had to sum up the entire process of working on NBusiness into one sentence it would be “creating a DSL is hard”. That’s an understatement frankly. Let me see if I can lay out the various layers required for DSL creation.
·         Domain Objects
·         Parser
·         Compiler
·         Template Engine
·         Build Integration
·         Tooling Support
The first three items are actually relatively easy and pretty fun. This is what we all know how to do, write code to parse strings and stick values into objects. No problem. It turns out the next three layers which really provide the fit, finish and ultimate usability of your DSL are not easy at all. Build integration isn’t really that bad actually but tooling integration can be a real bear. In the case of a DSL you really want syntax hilighting and intellisense and nice IDE integration for file templates and things like that. Maybe a few additional context menus in your IDE and such. For me I have been trying to integrate into Visual Studio and I can officially say that I have sunk well over half my time into that aspect alone and it has been one of the hardest things I have ever tried to do. Visual Studio is also architected such that I had to completely redo my parser and compiler to be compatible with the needs of Visual Studio. Very painful.
But what is really hanging me up now is what I consider to be a large gap in the .NET  DSL world and that is a suitable templating engine. By templating engine I mean something that can take metadata and translate it into code.
I mean we have a bunch out there but they’re all (as far as I know) effectively giant string builders. They suffer from Tag Soup and and are bound strongly to a specific language implementation. For NBusiness I want to support side by side integration with any .NET language, C# or VB or Python or whatever. And re-creating all of these templates for every language is not an option. It’s too much upfront work and it’s too much long term maintenance. I absolutely need templates that are based on the CodeDom so I can be language agnostic… But if you’ve ever tried to use the CodeDom you know how hard it is to work with. Because of this users are very unlikely to actually make their own templates (which is almost always necessary) and when they do it is a very painful process. So I’ve been stuck in this cunundrum for quite a while, how can you make a template engine that is both based on the CodeDom but has the ease of use of a string builder?
Enter MGrammar. Using MGrammar I have found a way to define a DSL for generating code. This DSL turns out to be a full fledged programming language in and of itself with the caveat of being restricted only to that which is CLS compliant. I have combined this DSL with the capability to create templates (to extend the language, similar to macros in Boo) and databinding similar to what you have in XAML. The end result allows you to do something similar to this:
namespace Example:
    import System;
 
    template One:
        public class {Binding Name}:
            {SequenceBinding Items, Template=Two}
        end
    end
 
    template Two:
        private field {Binding Type} _{Binding Name};
        public property {Binding Type} {Binding Name}:
            get:
                return this._{Binding Name};
            end
            set:
                this._{Binding Name} = value;
            end
    end
end
(This is just an example, the end result might not actually be exactly this syntax)
Which when compiled will generate a class called OneTemplate that inherits from Template and returns a CodeTypeDeclaration object from it’s Generate method. Extensions such as the BindingExtension show here can be custom objects to extend behaviors but in this case it binds the name of the class to the Name property (or Name sequence node of an MGraph tree) of the provided metadata.
Technically you could write your entire project in pure MetaSharp code but more likely you will write all of your static classes in your rich language of choice and simply use MetaSharp to define templates. Since this is all compiling down to CodeDom objects I have cooked up some MSBuild tasks that simply translate those objects into the code of the project the files exist in. You could share this same file in a VB or C# project and it would compile to the same thing in both assemblies.
Currently I am working on a prototype using the Song example from the MGrammar sample code that will allow you to write songs that generate song classes using templates like these. It’s almost working… the CSharpCodeProvider is throwing a random NullReferenceException with no useful error messages. Which is one reason why a DSL like this is helpful, it should be able to abstract away the pain of working directly with the CodeDom.

Getting a CodeDomProvider in an MSBuild Task

Trying to get the correct CodeDomProvider inside of an MSBuild task wasn’t as easy as I would have liked. Well actually the code itself is pretty simple but finding out how to actually do it was difficult. There doesn’t appear to be a profusion of people doing such a thing so the blogosphere and forums are fairly sparse with related information. After messing around with it for a couple of hours I think I finally found a pretty reliable (aka it doesn’t feel like a dirty hack) way to do it.

The two key bits of information is the <Language /> PropertyItem that each language defines in its own common targets file (by convention) and the CodeDomProvider.GetAllCompilerInfo() method. Here is my targets file.
 
<!–Reference the assembly where our tasks are defined–>
<UsingTask TaskName=“MetaSharp.MSBuild.TemplateTask” AssemblyFile=“$(MSBuildExtensionsPath)\MetaSharp\MetaSharp.MSBuild.dll” />
 
<!–Compile target (this is the target that calls the compiler task)–>
<Target Name=“BeforeBuild”>
    <Message Text=“Building: @(MetaSharpTemplate)” />
    <TemplateTask Templates=“@(MetaSharpTemplate)” Language=“$(Language)”>
      <Output TaskParameter=“Generated” ItemName=“Compile” />
    </TemplateTask>
</Target>
</Project>
The key here is the Language=”$(Language)” property on the task. And here is my task with a Linq statement to find the correct CodeDomProvider.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Build.Utilities;
using Microsoft.Build.Framework;
using System.Collections;
using System.CodeDom;
using System.CodeDom.Compiler;
 
namespace MetaSharp.MSBuild
{
      publicclassTemplateTask : Task
      {
            // Properties
            [Required]
            publicITaskItem[] Templates { get; set; }
 
            [Required]
            publicITaskItem Language { get; set; }
 
            [Output]
            publicITaskItem[] Generated { get; set; }
 
            publicoverridebool Execute()
            {
                  base.Log.LogMessage(0, “Building MetaSharp templates.”);
 
                  if (this.Language == null || string.IsNullOrEmpty(this.Language.ItemSpec))
                  {
                        base.Log.LogError(“You must have a Language PropertyItem defined somewhere in your project files to specify which CodeDomProvider to use (i.e <Language>C#</Language>)”);
                        returnfalse;
                  }
 
                  CodeDomProvider provider = FindProvider(this.Language.ItemSpec);
 
                  returntrue;
            }
 
            privateCodeDomProvider FindProvider(string language)
            {
                  CodeDomProvider[] providers = (from info inCodeDomProvider.GetAllCompilerInfo()
                                                               from l in info.GetLanguages()
                                                               where l.ToUpperInvariant() == language.ToUpperInvariant()
                                                               selectCodeDomProvider.CreateProvider(l))
                                                               .ToArray();
                 
                  CodeDomProvider provider = null;
                  if (providers.Length == 0)
                  {
                        Log.LogError(“Unable to find a valid CodeDomProvider for this project type. Try adding a valid Language property item to your msbuild project file”);
                  }
                  elseif (providers.Length > 1)
                  {
                        // It would be surprising if this ever happened…
                        Log.LogError(“Found multiple valid CodeDomProviders for this Language type. Try adding a less ambiguous Language property item to your msbuild project file”);
                  }
                  else provider = providers[0];
 
                  return provider;
            }
      }
}
For those of you who are curious MetaSharp is the tentative name of the CodeDom DSL I have been working on using MGrammar. It’s coming along pretty well, this task will build MetaSharp code files to be compiled along with the project they are contained in. There are lots of details to be shored up but the basic use cases are working right now. When I have things a little more polished I will probably create another post with some samples.