Embedding Lua into a Windows Store app

I recently decided to do something a little fun and new. I wanted to make a DirextX based Windows Store app that was primarily driven by Lua. Unsurprisingly, I couldn’t find a really solid demo of how to do this so there were a number of hurdles to overcome.

DirectX

The first hurdle, of course, was to figure out how to actually render something in DirextX using the new winrt api’s! I didn’t need anything too complicated, so I decided to go with the simple and easy to follow tutorial over at www.DirectXTutorial.com. The code I am using in this sample is essentially the first part of that tutorial with a Lua engine sitting on top of it. I highly recommend going through that tutorial if you’re not already an expert and maybe pay for the premium content if you like it enough :)

Script API

After following through the tutorial I had a working starting point. My next step was to write some code in Lua, that would represent the API I wish I had. After several changes this is what I ended up with:

luadx:background({ 0.0, 0.2, 0.4, 1.0 })
luadx:scene("main", function (scene)
    scene:object("triangle", function (triangle)
        triangle:add("geometry", Geometry.Vertex({
            { 0.0, 0.5, 0.0 },
            { 0.45, -0.5, 0.0 },
            { -0.45, -0.5, 0.0 }
        }))
    end)
end)

The idea was that I wanted to create an API based on Scenes, GameObjects and Components rather than dealing with DirectX functions directly. In this example, the luadx object is actually a module imported into the global scope. It exposes a couple members: background and scene. The background function allows us to set the background color we want to use when redrawing the scene on each step of the render loop. The scene function creates a Scene object and passes it to the included function.

Once we have a Scene we can create GameObjects by calling the object function on the scene. This creates an GameObject and adds it to the Scene and passes it into the provided initialization function, where we can add components to it. In this small sample I only have one component type which is a Vertex component. The Vertex component actually creates a vertex buffer and renders it during the render loop.

This is similar to the structure of how games are laid out in tools like Unity, though this sample is extremely simplistic. Unity is very interesting and worth looking at if you haven’t already. One thing to note is how this pattern relies on Composition rather than inheritance. A scene is composed of game objects, a game objects is composed of scenes and other game objects… In a real game engine you need a few more features to really facilitate composition such as the ability to send messages between components. This sample isn’t that robust but you could use your imagination.

Lua 5.2

The second big hurdle is how to actually get Lua, and add it to your project. I ran into several problems actually, the worst of which was that the compiled versions of Lua available for download are not compatible with winrt which led to me having to download the source, modify it slightly and figure out how to compile it again in Visual Studio 2013. I will include the full source for my working sample with this post but I will also try to explain what I had to do to get it to work.

The first thing I did was slightly modify luaconf.h.

#if WINAPI_FAMILY_PARTITION(WINAPI_FAMILY_DESKTOP_APP)
#define LUA_WINRT
#undef LUA_DL_DLL
#endif

We can use the family partition macro to determine if we are in winrt, and then create a new LUA_WINRT define and also undefine LUA_DL_DLL. Next I had to go into loadlib.c and modify the setpath function. The goal of this modification is to remove calls to getenv, which are not available in winrt apps. Once we have this the only trick is figuring out how to build lua in Visual Studio 2013. For this you need to make a C, lib based project and add the following preprocessor definitions: LUA_OPNAMES;_CRT_SECURE_NO_WARNINGS. After that link your store app to the lua52.lib file you built and you should be good to go!

Lua interop

Finally the fun stuff! This is actually the bulk of the work because I needed to learn quite a few new things. I needed to learn both the Lua language and runtime itself but also the C api for interoping with it. Fortunately, while the Lua API is quite complex, it is also extremely clean and elegant. In my app the Lua interop starts in the Host class.

In most Lua samples you will see a helper macro called lua_open() being used to create the new lua_State but this appears to have been deprecated and so now you use luaL_newstate(). Now that you have your state you would typically call lua_loadlibs(L), but sadly not all of the libraries can be loaded into winrt without further modifications. So instead you include them one by one:

luaopen_base(_state);
luaopen_table(_state);
luaopen_string(_state);
luaopen_math(_state);
luaopen_bit32(_state);
luaopen_coroutine(_state);
luaopen_debug(_state);
luaopen_package(_state);

These are all of the default packages I was able to load without trouble. Next I push the Host object into the lua registry:

lua_pushlightuserdata(_state, this);
lua_setfield(_state, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);

This allows me to access the Host object from any native function called by Lua. This is handy because as you’ll soon find out Lua is really only compatible with static functions in C++, so you need to add references to your application into the state of Lua itself. The Lua registry is interesting because it’s essentially a global object that isn’t accessible from script.

Next we import the customĀ luadx and Geometry modules using the luaL_requiref function. This will put the objects we create in the Require functions into the global scope with the given variable names but it will also make it available for use via the require keyword. If this sample was any more complex I would probably not put these into the global scope but would instead have the user require all needed modules.

Lua has a very interesting API. It’s centered around the concept of a stack and most of the functions use the stack to work. You will push values on and off of the stack and many of the functions make assumptions about the objects that are expected to be at the top of the stack. You have to get quite comfortable with these ideas to work with Lua effectively, and I have lots of examples in this project but rather than go through them all I want to show a single example of a function that I consider to be representative of how you work with Lua in general.

int Scene::New(lua_State *L)
{
    static const int Expected_Args[] = {
        LUA_TFUNCTION,
        LUA_TSTRING,
        LUA_TTABLE,
        LUA_TNONE
    };

    if (!Util::ValidateArgs(L, Expected_Args))
        return 0;

    auto name = lua_tostring(L, -2);
    lua_remove(L, -2);

    lua_getfield(L, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);
    auto host = (Host*) lua_touserdata(L, -1);
    lua_pop(L, 1);

    auto scene = (Scene*) lua_newuserdata(L, sizeof(Scene));
    auto scene_ref = luaL_ref(L, LUA_REGISTRYINDEX);
    scene->Initialize(name, scene_ref);
    host->Add(scene);

    // Set scene metatable...
    lua_rawgeti(L, LUA_REGISTRYINDEX, scene_ref);
    if (luaL_newmetatable(L, LUA_SCENE_TYPENAME) == 1)
    {
        static const luaL_Reg Scene_Members[] = {
            { "object", &GameObject::New },
            { NULL, NULL }
        };

        // Initialize metatable...
        lua_pushstring(L, "__index");
        luaL_newlib(L, Scene_Members);
        lua_settable(L, -3);

        lua_pushstring(L, "__gc");
        lua_pushcfunction(L, &Scene::Gc);
        lua_settable(L, -3);
    }

    lua_setmetatable(L, -2);
    lua_call(L, 1, 0);
    return 0;
}

This is the function that is called when luadx:scene is called from script. The first thing we do in this function is attempt to validate the provided arguments. I created a little helper function called Util::ValidateArgs, which looks at the top items in the stack and verifies that they match the types I am expecting. If they do not match then lau_error is called with a string describing what went wrong.

static const int Expected_Args[] = {
    LUA_TFUNCTION,
    LUA_TSTRING,
    LUA_TTABLE,
    LUA_TNONE
};

if (!Util::ValidateArgs(L, Expected_Args))
    return 0;

Next we retrieve the name from the first argument:

auto name = lua_tostring(L, -2);
lua_remove(L, -2);

Note that we are using the index value -2, which means that we are accessing and removing the second element from the top of the stack. If you were to use +2 then you would be accessing the second element from the bottom of the stack. And since this is our first argument, it is the second from the top, the argument at -1 is the function, or the second argument. It can be a little confusing at first to deal with negative indices but you get used to it pretty fast. I created a very useful Util::Dump(L) function that prints out the types of objects in the stack so I can do a sanity check whenever I want.

Next we need to get a reference to the Host object, which we can do by pulling it out of the registry. The lua_getfield function can be used to point at the lua registry and pull the host out of a uniquely named slot. It is then put onto the stack which we can convert to a pointer by calling lua_touserdata and then popping it off the stack.

lua_getfield(L, LUA_REGISTRYINDEX, LUA_HOST_REGISTRY_KEY);
auto host = (Host*) lua_touserdata(L, -1);
lua_pop(L, 1);

Next we want to create a new Scene and add it to the host.

auto scene = (Scene*) lua_newuserdata(L, sizeof(Scene));
auto scene_ref = luaL_ref(L, LUA_REGISTRYINDEX);
scene->Initialize(name, scene_ref);
host->Add(scene);

Here I am also getting a reference to the Lua object that points to the Scene. I can use this later to push the object onto the stack and call members on it or perform other operations on it. This is how you would call back into a Lua script function from native code for example.

Next we attempt to set a metatable onto the object.

lua_rawgeti(L, LUA_REGISTRYINDEX, scene_ref);
if (luaL_newmetatable(L, LUA_SCENE_TYPENAME) == 1)
{

When you call luaL_newmetatable it will return 1 if it actually creates the metatable and 0 if the table already exists. In both cases it will push it onto the stack. We really want a metatable here because it allows us to tap into some special funtions for the object, namely the __index and the __gc functions. The __gc function will be called if the object is collected and we can then perform any custom operations for releasing memory we may have allocated. The __index member allows us to append callbacks to native functions onto our script object. In the case of the scene we want to add an “object” function. When you attempt to access a field in Lua if that field doesn’t exist it will then call into the __index function on the objects metatable and allow it to handle the call instead. By adding a table to the __index property the Lua runtime will automatically create a function that will just redirect calls to that tables members instead. It’s basically a slightly more robust version of javascripts prototypal inheritance.

static const luaL_Reg Scene_Members[] = {
    { "object", &GameObject::New },
    { NULL, NULL }
};

// Initialize metatable...
lua_pushstring(L, "__index");
luaL_newlib(L, Scene_Members);
lua_settable(L, -3);

lua_pushstring(L, "__gc");
lua_pushcfunction(L, &Scene::Gc);
lua_settable(L, -3);

Finally we call the function passed in as the second argument, leaving the table containing the Scene on the top of the stack and return 0, which indicates that the top 0 values should be popped off the stack and popped back on the stack of the calling function.

lua_setmetatable(L, -2);
lua_call(L, 1, 0);
return 0;

And this is about it, the rest is just variations on this theme. You can pop objects containing pointers into the stack and store them for reference later. You can pull pointers out and call functions on them. You can push values and throw errors and debug into the runtime. Its really quite fun and surprisingly easy to do!

Conclusion

Lua is pretty fun and not that hard to embed into your application. It’s not totally cookie cutter for windows but its solvable. As far as C code goes Lua is incredibly clean and really a quintessential example of how to do native code right.

Download

Full Code: luadx.zip

Terraria – Lava Dome

I’ve been having some fun with the new Terraria update. Leo and I built a Lava Dome. It’s a wall filled with lava surrounding our entire base. The entrances are filled with active blocks so we can flip a switch during an invasion and the doors will fill with lava. It’s admittedly completely impractical… but pretty awesome anyway :)

You can also see our art gallery.

lavadome v1.0

The automatic lava generator is pretty easy to make too. Just make a large chamber and put a inlet pump at the bottom and an outlet pump at the top. Wire them up with a 1 second timer and let it go. It won’t generate lava automatically but what will work is to put blocks separated by a single space all next to each other then take a bucket and dump a single load on top of each divider. Those will continue to split and if you leave and come back later your chamber will be full. You can then wire up a different outlet pump to pump all the lava wherever you want.

This game is ridiculously fun and engrossing. The new update is worth checking out even if you have played Terraria before.

Virtual Reality Language Workbench

Here’s a little sketch I made of a concept for a language workbench designed for use with an Oculus Rift and Razor Hydra. The idea is that you would have a “cockpit” view of your workbench. In the center of the view you would have a graphical representation of a pattern, which you could manipulate with the hydra. On the far left is a 2D toolbar for actions on the far right is an assortment of available patterns. To the left of the pattern is a view of some test input to the right is the production of the data through the pattern.

You would need a way to model complex data visually though, and the output view is dynamic based on the data. You would be able to zoom into the output display to see it as a running app. This is what’s rumbling around in my head these days…

vr_ide