Farewell Microsoft

For the past 6 years, almost to the week, I have been an employee of Microsoft. I was part of a small development team based out of an office in St. Paul, Minnesota. About 3 weeks ago our office was shutdown due to a wave of layoffs taking place in Microsoft and as a result I was also laid off. I have been given the option to relocate to Redmond and apply for a new job if I wish, but I have decided that I do not want to relocate right now after all. So it is at this point that I am saying farewell to Microsoft.

I worked on Expression Blend, Visual Studio and finally Internet Explorer. I have mostly enjoyed my time there, especially working with some great people over the years. It was not without its technical and social challenges to overcome but for the most part I come away from the experience better and stronger I believe. I believe the senior management and the changes they are making are actually good for Microsoft and look forward to seeing them ascend and take their rightful place among the tech giants once again.

I harbor no ill will towards Microsoft and I find myself feeling more relieved and excited to be onto a new chapter than I am worried or stressed, though this kind of uncertainty is never fun. So it is with mixed emotions I say farewell Microsoft, and thanks for all the fish.

WP_20141008_008 ??????????? WP_000011 WP_000043 (1) WP_20140918_001

paktc, an npm package

Just to learn about the process more I created a small and semi-useful (ok not-that-useful :) npm package called paktc. Which stands for “Press any key to continue…”.

Which does exactly what the name implies; it prints out that message then waits for you to press something before continuing. This is a standard thing to be added to default console applications for C# and its useful for cases where you’re using an IDE (e.g. Visual Studio with nodejs tools extension) that pops up a command prompt and then quickly closes it again before you can see the results.

paktc (Press any key to continue…)

Prints ‘Press any key to continue..’ when in debug mode and waits for input before closing.

example

// your console application code here...

require('paktc') // Press any key to continue...

install

With [npm] (http://npmjs.org) do:

npm install paktc

 

It was a very interesting experience to work with npm, it’s amazingly easy and simple to use. I can’t imagine any significant platform in the future not having a comparable package manager and still being successful. You should check it out if you haven’t already.

 

New meta# nuget package published

I uploaded a new meta# NuGet package:

PS> Install-Package metasharp

It only has pattern matching api’s in this package now. To make patterns, just start with MetaSharp.Transformation.Pattern and all of the pattern factories are static methods from there. A simple example:

var p = Pattern.OneOrMore(Pattern.Range(‘a’, ‘z’));
var m = Pattern.Match(p, “metasharp”);
if (m.Matched) { … }

 

This package is .net 2.0 now too, so you can use it in more places (such as Unity3d). I’m planning on putting the actual grammars and build tasks etc into other packages so that the main package remains very portable. Also I changed it so that ‘p’ in the above scenario is a Pattern ast object instead of a Delegate. Which means that you could theoretically visit the pattern and transform it, itself. Also tracing produced objects will give you the pattern that produced it as well as the metadata.

The most major improvement of the API from before is the fact that you don’t have to implement INode on the objects you produce. Before everything had to be an INode and so you would have to do some awkward wrapping / unwrapping if your tree didn’t implement INode but now it just works with any plain old CRL object. Also it uses DynamicInvoke on productions so you can just bind them directly to any method with parameters of any type:

var p = Pattern.Production(
    Pattern.Variable(“x”, Pattern.Any()),
    (int x) => x + 1); // throws if the value consumed isn’t an int

 

I hope that makes sense. My first attempt made use of generics to attempt to flow static typing through the productions (which is definitely possible) but the insufficient type inference and support for monads in C# caused the complexity of the patterns to be unbearable. I found that if you use object and DynamicInvoke instead, you get similar results with reasonable complexity. The end result is that this api isn’t strictly monadic, since there is no Bind or Select/SelectMany. But it is monadic in spirit, all of the objects involved in the pattern matching operation are immutable and are safely parallelizable, except possibly your custom productions.

I’m pleased with the performance so far, the Or pattern executes each branch in parallel. I haven’t really stressed it that far yet but my 210 unit tests take about .45 seconds to run total so that seems pretty good. I want to create a Reflection.Emit visitor so I can get even better performance for rules but, unfortunately, that API isn’t available on all platforms (e.g. WinRT) so I’m not sure how best to do that.

The next big step is to create a new package with a C# grammar + roslyn transformer. Before it was a 1-off language. This time I plan to redo that work except as C# so that it is more appealing to users. I also want to change the grammar language to not be inheritance based, but compositional. This will require a little experimentation but I think it will make it a lot more appealing. The lack of extensibility in Roslyn is a real bummer because as far as I can tell there is no way for me to easily extend Roslyn to add pattern syntax to C# that way. The result is that I can leverage roslyn to generate assemblies from my AST but I still have to implement my own C# grammar if I want to extend it. A bummer but not harder than what I did before.

If you get a chance to try it out, I would appreciate feedback on the Pattern api though :)

Enjoy!

Ludum Dare April 2014

Here are some screen shots from my first attempt at a Ludum Dare challenge :) It’s woefully incomplete and I don’t have any modeling skills so I did it all with primitive shapes in Unity3D. But its fun!

ants-ludumdare2014 ants-ludumdare2014_2

In case you’re wondering what Ludum Dare is, the short version is that it is a 48 hour game programming contest that you have to do completely alone just for fun. There is a theme, this time the theme is “Beneath the Surface”.

Download | Source

Embedding Lua into a Windows Store app

I recently decided to do something a little fun and new. I wanted to make a DirextX based Windows Store app that was primarily driven by Lua. Unsurprisingly, I couldn’t find a really solid demo of how to do this so there were a number of hurdles to overcome.

DirectX

The first hurdle, of course, was to figure out how to actually render something in DirextX using the new winrt api’s! I didn’t need anything too complicated, so I decided to go with the simple and easy to follow tutorial over at www.DirectXTutorial.com. The code I am using in this sample is essentially the first part of that tutorial with a Lua engine sitting on top of it. I highly recommend going through that tutorial if you’re not already an expert and maybe pay for the premium content if you like it enough :)

Script API

After following through the tutorial I had a working starting point. My next step was to write some code in Lua, that would represent the API I wish I had. After several changes this is what I ended up with:

luadx:background({ 0.0, 0.2, 0.4, 1.0 })
luadx:scene("main", function (scene)
    scene:object("triangle", function (triangle)
        triangle:add("geometry", Geometry.Vertex({
            { 0.0, 0.5, 0.0 },
            { 0.45, -0.5, 0.0 },
            { -0.45, -0.5, 0.0 }
        }))
    end)
end)

The idea was that I wanted to create an API based on Scenes, GameObjects and Components rather than dealing with DirectX functions directly. In this example, the luadx object is actually a module imported into the global scope. It exposes a couple members: background and scene. The background function allows us to set the background color we want to use when redrawing the scene on each step of the render loop. The scene function creates a Scene object and passes it to the included function.

Once we have a Scene we can create GameObjects by calling the object function on the scene. This creates an GameObject and adds it to the Scene and passes it into the provided initialization function, where we can add components to it. In this small sample I only have one component type which is a Vertex component. The Vertex component actually creates a vertex buffer and renders it during the render loop.

This is similar to the structure of how games are laid out in tools like Unity, though this sample is extremely simplistic. Unity is very interesting and worth looking at if you haven’t already. One thing to note is how this pattern relies on Composition rather than inheritance. A scene is composed of game objects, a game objects is composed of scenes and other game objects… In a real game engine you need a few more features to really facilitate composition such as the ability to send messages between components. This sample isn’t that robust but you could use your imagination.

Lua 5.2

The second big hurdle is how to actually get Lua, and add it to your project. I ran into several problems actually, the worst of which was that the compiled versions of Lua available for download are not compatible with winrt which led to me having to download the source, modify it slightly and figure out how to compile it again in Visual Studio 2013. I will include the full source for my working sample with this post but I will also try to explain what I had to do to get it to work.

The first thing I did was slightly modify luaconf.h.

#if WINAPI_FAMILY_PARTITION(WINAPI_FAMILY_DESKTOP_APP)
#define LUA_WINRT
#undef LUA_DL_DLL
#endif

We can use the family partition macro to determine if we are in winrt, and then create a new LUA_WINRT define and also undefine LUA_DL_DLL. Next I had to go into loadlib.c and modify the setpath function. The goal of this modification is to remove calls to getenv, which are not available in winrt apps. Once we have this the only trick is figuring out how to build lua in Visual Studio 2013. For this you need to make a C, lib based project and add the following preprocessor definitions: LUA_OPNAMES;_CRT_SECURE_NO_WARNINGS. After that link your store app to the lua52.lib file you built and you should be good to go!

Lua interop

Finally the fun stuff! This is actually the bulk of the work because I needed to learn quite a few new things. I needed to learn both the Lua language and runtime itself but also the C api for interoping with it. Fortunately, while the Lua API is quite complex, it is also extremely clean and elegant. In my app the Lua interop starts in the Host class.

In most Lua samples you will see a helper macro called lua_open() being used to create the new lua_State but this appears to have been deprecated and so now you use luaL_newstate(). Now that you have your state you would typically call lua_loadlibs(L), but sadly not all of the libraries can be loaded into winrt without further modifications. So instead you include them one by one:

luaopen_base(_state);
luaopen_table(_state);
luaopen_string(_state);
luaopen_math(_state);
luaopen_bit32(_state);
luaopen_coroutine(_state);
luaopen_debug(_state);
luaopen_package(_state);

These are all of the default packages I was able to load without trouble. Next I push the Host object into the lua registry:

lua_pushlightuserdata(_state, this);
lua_setfield(_state, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);

This allows me to access the Host object from any native function called by Lua. This is handy because as you’ll soon find out Lua is really only compatible with static functions in C++, so you need to add references to your application into the state of Lua itself. The Lua registry is interesting because it’s essentially a global object that isn’t accessible from script.

Next we import the custom luadx and Geometry modules using the luaL_requiref function. This will put the objects we create in the Require functions into the global scope with the given variable names but it will also make it available for use via the require keyword. If this sample was any more complex I would probably not put these into the global scope but would instead have the user require all needed modules.

Lua has a very interesting API. It’s centered around the concept of a stack and most of the functions use the stack to work. You will push values on and off of the stack and many of the functions make assumptions about the objects that are expected to be at the top of the stack. You have to get quite comfortable with these ideas to work with Lua effectively, and I have lots of examples in this project but rather than go through them all I want to show a single example of a function that I consider to be representative of how you work with Lua in general.

int Scene::New(lua_State *L)
{
    static const int Expected_Args[] = {
        LUA_TFUNCTION,
        LUA_TSTRING,
        LUA_TTABLE,
        LUA_TNONE
    };

    if (!Util::ValidateArgs(L, Expected_Args))
        return 0;

    auto name = lua_tostring(L, -2);
    lua_remove(L, -2);

    lua_getfield(L, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);
    auto host = (Host*) lua_touserdata(L, -1);
    lua_pop(L, 1);

    auto scene = (Scene*) lua_newuserdata(L, sizeof(Scene));
    auto scene_ref = luaL_ref(L, LUA_REGISTRYINDEX);
    scene->Initialize(name, scene_ref);
    host->Add(scene);

    // Set scene metatable...
    lua_rawgeti(L, LUA_REGISTRYINDEX, scene_ref);
    if (luaL_newmetatable(L, LUA_SCENE_TYPENAME) == 1)
    {
        static const luaL_Reg Scene_Members[] = {
            { "object", &GameObject::New },
            { NULL, NULL }
        };

        // Initialize metatable...
        lua_pushstring(L, "__index");
        luaL_newlib(L, Scene_Members);
        lua_settable(L, -3);

        lua_pushstring(L, "__gc");
        lua_pushcfunction(L, &Scene::Gc);
        lua_settable(L, -3);
    }

    lua_setmetatable(L, -2);
    lua_call(L, 1, 0);
    return 0;
}

This is the function that is called when luadx:scene is called from script. The first thing we do in this function is attempt to validate the provided arguments. I created a little helper function called Util::ValidateArgs, which looks at the top items in the stack and verifies that they match the types I am expecting. If they do not match then lau_error is called with a string describing what went wrong.

static const int Expected_Args[] = {
    LUA_TFUNCTION,
    LUA_TSTRING,
    LUA_TTABLE,
    LUA_TNONE
};

if (!Util::ValidateArgs(L, Expected_Args))
    return 0;

Next we retrieve the name from the first argument:

auto name = lua_tostring(L, -2);
lua_remove(L, -2);

Note that we are using the index value -2, which means that we are accessing and removing the second element from the top of the stack. If you were to use +2 then you would be accessing the second element from the bottom of the stack. And since this is our first argument, it is the second from the top, the argument at -1 is the function, or the second argument. It can be a little confusing at first to deal with negative indices but you get used to it pretty fast. I created a very useful Util::Dump(L) function that prints out the types of objects in the stack so I can do a sanity check whenever I want.

Next we need to get a reference to the Host object, which we can do by pulling it out of the registry. The lua_getfield function can be used to point at the lua registry and pull the host out of a uniquely named slot. It is then put onto the stack which we can convert to a pointer by calling lua_touserdata and then popping it off the stack.

lua_getfield(L, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);
auto host = (Host*) lua_touserdata(L, -1);
lua_pop(L, 1);

Next we want to create a new Scene and add it to the host.

auto scene = (Scene*) lua_newuserdata(L, sizeof(Scene));
auto scene_ref = luaL_ref(L, LUA_REGISTRYINDEX);
scene->Initialize(name, scene_ref);
host->Add(scene);

Here I am also getting a reference to the Lua object that points to the Scene. I can use this later to push the object onto the stack and call members on it or perform other operations on it. This is how you would call back into a Lua script function from native code for example.

Next we attempt to set a metatable onto the object.

lua_rawgeti(L, LUA_REGISTRYINDEX, scene_ref);
if (luaL_newmetatable(L, LUA_SCENE_TYPENAME) == 1)
{

When you call luaL_newmetatable it will return 1 if it actually creates the metatable and 0 if the table already exists. In both cases it will push it onto the stack. We really want a metatable here because it allows us to tap into some special funtions for the object, namely the __index and the __gc functions. The __gc function will be called if the object is collected and we can then perform any custom operations for releasing memory we may have allocated. The __index member allows us to append callbacks to native functions onto our script object. In the case of the scene we want to add an “object” function. When you attempt to access a field in Lua if that field doesn’t exist it will then call into the __index function on the objects metatable and allow it to handle the call instead. By adding a table to the __index property the Lua runtime will automatically create a function that will just redirect calls to that tables members instead. It’s basically a slightly more robust version of javascripts prototypal inheritance.

static const luaL_Reg Scene_Members[] = {
    { "object", &GameObject::New },
    { NULL, NULL }
};

// Initialize metatable...
lua_pushstring(L, "__index");
luaL_newlib(L, Scene_Members);
lua_settable(L, -3);

lua_pushstring(L, "__gc");
lua_pushcfunction(L, &Scene::Gc);
lua_settable(L, -3);

Finally we call the function passed in as the second argument, leaving the table containing the Scene on the top of the stack and return 0, which indicates that the top 0 values should be popped off the stack and popped back on the stack of the calling function.

lua_setmetatable(L, -2);
lua_call(L, 1, 0);
return 0;

And this is about it, the rest is just variations on this theme. You can pop objects containing pointers into the stack and store them for reference later. You can pull pointers out and call functions on them. You can push values and throw errors and debug into the runtime. Its really quite fun and surprisingly easy to do!

Conclusion

Lua is pretty fun and not that hard to embed into your application. It’s not totally cookie cutter for windows but its solvable. As far as C code goes Lua is incredibly clean and really a quintessential example of how to do native code right.

Download

Full Code: luadx.zip