Tag: NBusiness

The 5 Laws of Code Generation

This is a refresh of an old blog post. The more I look at Oslo and contrast it with what I have been working on I’m trying to verify that NBusiness isn’t actually redundant or made obsolete by Oslo. Obviously the two will be competitors, but what I’m trying to figure out is what actually is the difference between the two and more importantly why is NBusiness the more correct solution to the problem?

I would like to start out by saying that Oslo and NBusiness are both DSLs and core to any DSL is the transformation of the metadata into more concrete code, be that a less abstract DSL, actual executable code or something else entirely. So what I’m calling Code Generation is essentially that transformation process. Additionally what used to be called an “intermediate format” I’m now simply calling Metadata. To me, in this context, metadata is simply the name for the actual DSL declaration for a given application.

So here are those rules again, to be reframed into the context of Oslo and how I feel it may violate those rules.

1. Code generation is controlled through modifiable templates.

Translation of your metadata into another form should never be done through a black box. I’m not entirely sure how customizable the SQL generation of Oslo is but from what I understand it’s pretty opaque. In fact the entire system of translating MGraph into another form should be completely transparent and built such that it is very easy to be shaped into any form. If anyone can explain to me how Oslo translates M into sql and show me how I can do it myself, and alter the SQL that is generated then I’ll be happy to change my mind on this one but it feels rather opaque to me at this moment.

2. Code generation is done during the build process.

I would like to add an amendment to this one and specify that code generation can also legitimately be done dynamically as well. So the new rule could be more accurately changed to “Intermediate forms of metadata should never be persisted”. Not that you couldn’t write it out to temp folders but the point is that the integration of a DSL into an application should be seamless, you shouldn’t have to have multiple manual steps to get it all integrated. Whether this is done at runtime or at build time is irrelevant. Presumably the many command line apps that come with Oslo in order to transform your DSL into something that goes into the repository will be streamlined but this is the sort of thing that should be avoided.

3. Code generation is done from an intermediate format.

I would like to amend this rule to instead be “Metadata must have a single source of truth.” I think that Oslo has a pretty good system for the intermediate format (M) but it doesn’t follow the single source of truth rule. M is simply translated into the Repository, which is the real source of truth but is not necessarily synchronized in the reverse with the original M code (as far as I know at least).

To me Oslo violates this rule simply by the existence of the Repository. The repository is essentially a “second source of truth” and can be edited from multiple sources. To me the DSLs should be the single source of truth and the repository should be essentially a temp file or part of the output of the build. Editing the repository should simply be a matter of editing your entities.

4. The intermediate format is under source control with versioning.

I would like to amend this rule also to specify that only deltas must be part of each revision. So maybe this rule could be simply changed to “The metadata declaration must be versionable and mergable”. Which, usually means that your DSL needs to be textual. I would be willing to buy into a binary format for metadata but only if it had the ability to be versioned with my preferred source control system as deltas not just a giant binary blob and as long as it didn’t break any of the above rules in the process.

5. The code generation templates are also under source control.

This is to place the same constraints on the transformation system as the metadata itself. This will usually translate into the idea that your templates should also be textual. Again, if you want it to be some binary format then there needs to be ways to allow source control tools to persist only deltas. So this rule could be changed to be “The transformation templates must be versionable and mergable”.

Summary

Here is a summary of these revised rules:

  1. Metadata transformation is controlled through modifiable templates.

  2. Intermediate forms of metadata should never be persisted.

  3. Metadata must have a single source of truth.

  4. The metadata must be versionable and mergable.

  5. The transformation templates must be versionable and mergable.

MetaSharp Vision for the Future

I was just having some ideas and wanted to put it down somewhere partly for myself and partly to get some feedback.

One of the next things I want to do is to convert the compile-to-CodeDom parts of MetaSharp into a Vistor Pattern so that I can use the same system to compile to CodeDom or Generate MetaSharp or to transform the AST or whatever I want. This will bring a lot of flexibility and power to the whole system.

I was thinking about a post by Ayende Rahien the other day called M is to DSL as Drag and Drop is to Programming and specifically I was thinking about the quote “If you want to show me a DSL, show me one that has logic, not one that is a glorified serialization format.“ And what I took this to mean is that there is no logic in this DSL. Which can still be declarative but will often time have concepts like less-than or greater-than or equal-to. Certainly not limited by this but these are fairly common. To me his complaint (which is valid) is that with an external DSL, no matter how easy it is to write a grammar, it is still hard to expression logic with a grammar, and furthermore it is just as hard to translate that logic into something executable.

With an internal DSL, such as you get with Boo, you can easily just author keywords for your DSL but you get all of the logical operators for free, which is very nice of Boo. But unfortunately with an internal DSL you not only get the logical operators for free you are forced to get them. With an internal DSL you can do less work to get it working but you are not operating in a constrained universe. This has trades offs but lets certainly not dismiss it. There plenty of use cases where this is the preferred way of doing it.

However there are some distinct benefits of an external DSL, the major tradeoff being the effort required to implement it. The main benefit is that you can constrain your universe such that only allowable logic can happen in the correct spots. It’s like a sandboxed language, which I like to call a constrained universe. And believe it or not constraint can actually be freeing.

So my sudden flash of insight this morning was when I realized that actually, with MGrammar, you can choose to import grammars defined in other assemblies and use the syntax and tokens defined there. So when you choose to use MetaSharp by adding a reference to the assembly you can actually also import the MetaSharp.Lang grammar and easily make use of the BinaryExpression syntax in your own DSL (or anything else). Then I was also thinking that you could probably make use of the same AST serialization tools and (soon to be) AST transformation Visitors to build your own DSLs without a lot of the extra work. Using that type of system you could probably transform directly into executable code completely without using the templating at all, haha! Simply transform your custom AST nodes into standard supported Nodes, or write your own visitor that can handle your custom nodes. Your custom visitor could probably also tap into the templating system so you could write the AST transformation as a MetaSharp template if you desired as well.

This would put MetaSharp into the role of being an extensible compiler system where custom external DSLs can opt-in to standard language grammar where appropriate rather than not even being able to opt-out as in current internal DSLs. This is powerful idea and I think it is well within my grasp.

MetaSharp – A CodeDom based Template Engine using MGrammar

I’ve been working on a tangential project related to NBusiness for a couple of weeks now and I just wanted to take a moment to get a few of my thoughts out. The project I have been working on I am tentatively calling “MetaSharp” for now. It’s been fun and educational but hopefully it will have real usefullness when it is done. I wanted to have a fully working example before I publicly posted the code (since it’s basically prototype quality right now) but if anyone is interested in seeing what I have so far feel free to ask and I’ll hook you up somehow.

I’ll try to start at the beginning to justify my rationale for creating this strange project. I’ve been working on NBusiness for quite a while now and while I’ve really had NBusiness “working” almost all along I have never quite been able to get it where I want it to be (complete). If I had to sum up the entire process of working on NBusiness into one sentence it would be “creating a DSL is hard”. That’s an understatement frankly. Let me see if I can lay out the various layers required for DSL creation.
·         Domain Objects
·         Parser
·         Compiler
·         Template Engine
·         Build Integration
·         Tooling Support
The first three items are actually relatively easy and pretty fun. This is what we all know how to do, write code to parse strings and stick values into objects. No problem. It turns out the next three layers which really provide the fit, finish and ultimate usability of your DSL are not easy at all. Build integration isn’t really that bad actually but tooling integration can be a real bear. In the case of a DSL you really want syntax hilighting and intellisense and nice IDE integration for file templates and things like that. Maybe a few additional context menus in your IDE and such. For me I have been trying to integrate into Visual Studio and I can officially say that I have sunk well over half my time into that aspect alone and it has been one of the hardest things I have ever tried to do. Visual Studio is also architected such that I had to completely redo my parser and compiler to be compatible with the needs of Visual Studio. Very painful.
But what is really hanging me up now is what I consider to be a large gap in the .NET  DSL world and that is a suitable templating engine. By templating engine I mean something that can take metadata and translate it into code.
I mean we have a bunch out there but they’re all (as far as I know) effectively giant string builders. They suffer from Tag Soup and and are bound strongly to a specific language implementation. For NBusiness I want to support side by side integration with any .NET language, C# or VB or Python or whatever. And re-creating all of these templates for every language is not an option. It’s too much upfront work and it’s too much long term maintenance. I absolutely need templates that are based on the CodeDom so I can be language agnostic… But if you’ve ever tried to use the CodeDom you know how hard it is to work with. Because of this users are very unlikely to actually make their own templates (which is almost always necessary) and when they do it is a very painful process. So I’ve been stuck in this cunundrum for quite a while, how can you make a template engine that is both based on the CodeDom but has the ease of use of a string builder?
Enter MGrammar. Using MGrammar I have found a way to define a DSL for generating code. This DSL turns out to be a full fledged programming language in and of itself with the caveat of being restricted only to that which is CLS compliant. I have combined this DSL with the capability to create templates (to extend the language, similar to macros in Boo) and databinding similar to what you have in XAML. The end result allows you to do something similar to this:
namespace Example:
    import System;
 
    template One:
        public class {Binding Name}:
            {SequenceBinding Items, Template=Two}
        end
    end
 
    template Two:
        private field {Binding Type} _{Binding Name};
        public property {Binding Type} {Binding Name}:
            get:
                return this._{Binding Name};
            end
            set:
                this._{Binding Name} = value;
            end
    end
end
(This is just an example, the end result might not actually be exactly this syntax)
Which when compiled will generate a class called OneTemplate that inherits from Template and returns a CodeTypeDeclaration object from it’s Generate method. Extensions such as the BindingExtension show here can be custom objects to extend behaviors but in this case it binds the name of the class to the Name property (or Name sequence node of an MGraph tree) of the provided metadata.
Technically you could write your entire project in pure MetaSharp code but more likely you will write all of your static classes in your rich language of choice and simply use MetaSharp to define templates. Since this is all compiling down to CodeDom objects I have cooked up some MSBuild tasks that simply translate those objects into the code of the project the files exist in. You could share this same file in a VB or C# project and it would compile to the same thing in both assemblies.
Currently I am working on a prototype using the Song example from the MGrammar sample code that will allow you to write songs that generate song classes using templates like these. It’s almost working… the CSharpCodeProvider is throwing a random NullReferenceException with no useful error messages. Which is one reason why a DSL like this is helpful, it should be able to abstract away the pain of working directly with the CodeDom.

justnbusiness entities

In case you hadn’t noticed I used NBusiness as the backend for this very site. Well due to popular demand I am including the entities I used to create this very site as well as the sql scripts for the custom factory methods. Feel free to build your own blog with this code! I’ll probably make another post related to how to roll out changes to your goddaddy (or other web host) relatively easily using an “install” system.
 
Download the Entity Project here.
 
All you need to do is install the latest version of NBusiness. Add a reference to this entity project and build your code. You can use the sql scripts generated by NBusiness (or run the schema updater by right clicking the project file in the solution explorer) then run the custom sql script included with this project.

XAML View Engine

I just had an interesting idea and wanted to blog it down before forgetting since I can’t really spend time investigating it now.

Well I’ve been working with NVelocity and now Spark to use as my view engine for template generation. I’m very happy with Spark right now (since it actually works) and I have been generating templates like a fool the last couple days. I think I might have a preview release of NBusiness 3 out for testing sometime soon.

That being said there is definitely still something missing. It works… it’s a heck of a lot better than NVelocity but it’s not quite perfect. I was just mentioning this to a friend and suddenly had a jolt of inspiration. I had the idea that you could, quite possibly, declare your entire template in XAML . Your XAML syntax could support binding and you could have looping and conditional controls right in it. This would allow you to declaratively define code in arbitrary languages. Pretty interesting… here is a little snippet of what I was envisioning:

<Class Attributes=”Public, Abstract” Base=”BusinessBase” Name=”{Binding Name}”>

   <Repeat Collection=”{Binding Fields}”>

      <Field Name=”field” Attributes=”Private” Type=”{Binding Path=Type}” Name=”{Binding Path=Name, Converter=FIeldNameConverter}” />

      <Property Attributes=”Public” Type={Binding Path=Type}” Name={Binding Path=Name}”>

         <Property.Getter><Return Expression=”{Binding RelativeSource=field}” /></Property.Getter>

      </Property>

   </Repeat>

   …

</Class>

Honestly, I’m thinking that all I have to do is create these objects which would inherit from DependencyObject and the XAML serializer will do the rest… I doubt it would be that simple but I may have to try it out later tonight.

The thing is, once you have this template into a DOM you could then run it through some sort of Conversion process to create CodeDom objects which would produce code for any arbitrary language! The trick would be to make it flexible enough to support conversion to things like HTML or whatever. Wow, this could be interesting.