Luabind: Transparent Lua FFI and Embedding

Introduction

luabind is a header-only library that facilitates bi-directional bindings between C++ and Lua. I began this project as a learning exercise, to practice more advanced modern C++ features like template metaprogramming, constexpr, SFINAE, C++20 “concepts”/“requires”, CTAD, and more.

Though you can find more examples of the usage in the tests for luabind, here is one of the unittests which demonstrates some of its capabilities:

TEST(LuaBind, Expose) {
  luabind::Lua lua;
  lua["timesTwo"] = [](int x) -> int {
    return x*2;
  };
  lua << R"(
            callIntoCFunc = function(a)
                return timesTwo(a)
            end
        )";
  int x = lua["callIntoCFunc"](4);
  ASSERT_EQ(x, 8);
}

Comparison to Raw Lua API

Many programming languages expose APIs to embed and extend the language. For example, Python has a C API. Ruby has a C API. Lua’s own C API has excellent documentation.

Many language C APIs revolve around a value type represented by some sort of opaque pointer. For example, Python has the PyObject. Ruby has VALUE. Lua, on the other hand, uses a stack to store objects in the Lua interpreters state. The consumer of the C API does not have direct access to the object, but can push and pop items from the stack, and query information about the stack. The doc describes how to call Lua from C, like this:

/* call a function `f' defined in Lua */
double f (double x, double y) {
  double z;

  /* push functions and arguments */
  lua_getglobal(L, "f");  /* function to be called */
  lua_pushnumber(L, x);   /* push 1st argument */
  lua_pushnumber(L, y);   /* push 2nd argument */

  /* do the call (2 arguments, 1 result) */
  if (lua_pcall(L, 2, 1, 0) != 0)
    error(L, "error running function `f': %s",
              lua_tostring(L, -1));

  /* retrieve result */
  if (!lua_isnumber(L, -1))
    error(L, "function `f' must return a number");
  z = lua_tonumber(L, -1);
  lua_pop(L, 1);  /* pop returned value */
  return z;
}

This rough edge of the Lua API is what I aim to experiment with using luabind. Can we improve the developer experience of calling Lua from C++, and of calling C++ from Lua? Luabind lets us rewrite the body of the function above to the following, while maintaining error handling, runtime performance, and type safety:

int z = lua["f"](x, y);

This will have zero overhead over a handwritten implementation. Likewise, can we improve calling C++ from Lua? A simple example from the doc shows the implementation of a “sin” function:

static int l_sin (lua_State *L) {
  double d = lua_tonumber(L, 1);  /* get argument */
  lua_pushnumber(L, sin(d));  /* push result */
  return 1;  /* number of results */
}

Instead, whenever we need a Lua function pointer (one that takes in a lua_State and returns an int), we can automatically generate one from the native C++ function using: luabind::adapt(&sin). As a C++ author, your exposure to the lua_State struct and the Lua stack is greatly diminished.

Comparison to Existing Solutions

There do exist wrappers around Lua (and other interpreter C APIs) which provide similar functionality. Though luabind is more of a practice/academic exercise than an attempt to make a serious library, it is interesting to compare luabind to other options.

For example, swig is a popular program which lets you create wrappers around native C code for many different languages. It supports lots of features luabind does not, such as “userdata” conversions, classes with operator overloads becoming custom metatable entries, etc. On the other hand, swig requires an extra build step and “*.i” files to generate C source that can be compiled after. Additionally, while swig provides nice ergonomic language extension functionality to create new native modules, it does not appear to provide much assistance for embedding, or calling into Lua from C or C++. Where swig is a dependency that must be installed to run the build step, luabind can be dropped into an existing project as a single header file, and integrates with the host project’s build system.

There is also LuaBridge, which offers bidirectional binding between Lua and C++. However, the code style is much more imperative. Some of the example code is actually very reminiscent of a “builder”-style pattern. LuaBridge also does not do anything to hide the existence of the lua_State from the developer–this could be a good or bad thing. LuaBridge does provide transparent conversions to and from native C++ types, like luabind.

Tech Topic 1: Ergonomic global retrieval and assignment

To make setting and getting globals in the Lua state simple, the top-level luabind::Lua class has an operator[] that takes in a string and returns a GetGlobalHelper. Because GetGlobalHelper exposes templated assignment, cast, and call operators, you can write code like:

lua["someGlobal"] = 1;
std::string otherGlobal = lua["otherGlobal"];
int res = lua["someFunction"](1, "stringInput");

Tech Topic 2: Ergonomic calls into Lua from C++

Mentioned above is the call operator of GetGlobalHelper. The developer experience it exposes is extremely simple–just retrieve a global function and call it as if you were invoking a regular function in C++. However, this is hiding a great deal of complexity the developer would otherwise have to deal with directly. For example, let’s look at the code above. On the third line, we get the “someFunction” global, which is simple enough. However, upon using the call operator of the result of lua["someFunction"] (a GetGlobalHelper), we need to take care of a few things.

1. Pushing the Lua function that is the global onto the Lua stack
2. Pushing correctly typed arguments onto the stack
3. Invoking the Lua function
4. Pop off the correctly typed output of the result of the function call, and return it from the call operator

In order, this is how we do those things:

Push the Lua function onto the Lua stack

GetGlobalHelper‘s call operator returns a CallHelper, whose cast operator or destructor will either call callWithReturnValue or callWithoutReturnValue, depending on if the CallHelper was ever casted (as it was implicitly in the code example above). These eventually call pushFunctionAndArgs. This pushes the value of the global by the requested function’s name, if it exists.

Push the correctly typed arguments onto the Lua stack

If the CallHelper is casted (as it is to “int” on the third line of the code above), CallHelper will call callWithReturnValue with the function arguments, and use the inferred type of the cast operator to pop off the right output type and return that. callWithReturnValue uses template argument deduction to push the right arg types onto the stack. We can only definitively know if CallHelper is never casted in its destructor. So a never-casted CallHelper will call callWithReturnValue in its destructor. Like callWithoutReturnValue, callWithReturnValue uses template argument deduction to push the right argument types onto the stack. This is done in pushFunctionAndArgs, which uses a fold expression to push each arg in order:

(detail::toLua(fState, aArgs), ...);

detail::toLua has a constexpr if-yard that dispatches to the right Lua C API function to push the right type onto the stack. The result is compile-time expansion to precisely the same code a human would write to imperatively push each argument in order.

Invoking the function

callWithReturnValue and callWithoutReturnValue, invoke the function using lua_pcall after pushing the function and arguments onto the stack:

auto errCode = lua_pcall(fState, sizeof...(aArgs), 0, 0);

Because we statically know the number of arguments and the number of returns, there is some good safety here against programmer error. We use lua_pcall as this will trap any errors and prevent a crash.

Pop off the return value from the Lua stack

callWithReturnValue does the same as callWithoutReturnValue to invoke the function, but also returns a RetHelper whose templated cast operator will pop off the requested type from the stack.

Tech Topic 3: Ergonomic calls into C++ from Lua

The Lua API allows you to compile a module that you can import from a Lua interpreter. This lets you call native code from the scripting language, which is nice for extending the languages capabilities. However, the API is hard to use. You must construct a special registry table of C-style function pointers to functions that take in a state pointer and return a status int. Those functions traditionally must use the raw Lua C API to pop the function’s argument and push its return value. We can simplify this with the function luabind::adapt. adapt takes in any C++ callable, and returns a C-style function pointer to an inferred function that does all the right marshalling and unmarshalling. To do this, we have a couple steps:

1. Store a long-lived reference to the callable thing
2. Get a pointer to a function that uses template argument deduction to push and pop the right types and invoke the stored C++ callable

First, luabind::adapt has a defaulted non-type-template-parameter whose default value is the type of an empty lambda. Because lambdas have unique types by definition, this means every unique call to luabind::adapt stores a unique copy of the callable, and returns a distinct C-style pointer. The callable copies are stored in a global. This is safe, because every instance owns a unique specialization of the global which is not addressable outside of luabind::adapt.

Second, adapt returns a pointer to adapted given the callable and unique types. adapted itself takes in a lua_State and returns an int. Thus, it is the the responsibility of the body of adapted to do the right push and pop operations. Again, because the unique lambda type is unique, every call to adapt returns the address of a different specialization of adapted. adapted uses getArgsAsTuple which in turn uses a fold expression to pop the right typed arguments off the stack and return them as a C++ tuple:

return {detail::fromLua<Args>(aState) ...};

adapted then uses std::apply to call the stored unique copy of the callable with the argument tuple “splatted” out. Based on the callable’s return type (void or some other type), adapted may or may not push a return value onto the stack.

Tech Topic 4: How do I detect if a type T is callable?

This question is harder than one would think to answer in C++. There are several entities that can be “called” in C++:

• Pointer to function
• Pointer to member function
• std::function
• As a generalization of std::function, any struct or class with an operator(), which may or may not be templated or overloaded

The C++17 and later offers std::is_invocable. However, this requires knowledge at code-authoring time of the argument and return types you are testing on the callable thing. C++20 added std::invocable, which is implemented as a “concept”. However, this also requires knowledge of the argument types. In luabind, there is a need to differentiate any arbitrary “callable thing” from other types, regardless of argument or return types. How do we get around the limitations of the standard functions? We can do this with clever application of concepts. We can start with a concept-free implementation that does not detect callable structs or classes:

template <typename T>
constexpr bool is_callable_v = std::is_function_v<std::remove_pointer_t<T>> ||
    std::is_member_function_pointer_v<T>;

To complete is_callable_v, we can give ourselves a blank to fill in:

template <typename T>
constexpr bool is_callable_v = std::is_function_v<std::remove_pointer_t<T>> ||
    std::is_member_function_pointer_v<T> ||
    has_call_operator_v<T>;

has_call_operator_v<T> is where the standard falls short. There’s a property of the call operator which makes it a compile-time error to attempt to get the address of a call operator where the identity of the operator is ambiguous. For example, this can occur if a class inherits from two classes, both of which provide a call operator. Imagine “testing” if class A is callable by subclassing it in class B that also inherits from a known-callable class C. If taking the address of the call operator on class B results in a compiler error, then class A must be callable. However, this test is not useful before C++20 as a compiler error cannot be turned into a compile-time value. In C++20 with concepts however, we can test if a block of code compiles by putting it in a constraint! If the constraint in a requires expression would compile, it is a compile-time value of true. Now we can write:

struct MightCollide {
  void operator()() {}
};

struct WontCollide {};

template <typename T>
struct DetectCollision : public std::conditional_t<std::is_class_v<T>, T, WontCollide>, public MightCollide {
};

template <typename T>
constexpr bool has_call_operator_v = !requires() { &DetectCollision<T>::operator(); };

Why can’t we just make DetectCollision directly inherit from MightCollide and T? Why do we need to conditionally inherit from T or WontCollide? In C++, there is a compile-time error for inheriting from something that is not a class. However, this error occurs at an earlier stage than the concepts are checked. So if has_call_operator_v is ever provided a non class (even after a short-circuiting compile-time condition like is_callable_v!), then the following won’t compile:

struct MightCollide {
  void operator()() {}
};

template <typename T>
struct DetectCollision : public T, public MightCollide {
};

template <typename T>
constexpr bool has_call_operator_v = !requires() { &DetectCollision<T>::operator(); };

Tech Topic 5: Error Handling

Error handling is an interesting issue with the Lua C API. Calling arbitrary Lua code from C++ can result in arbitrary errors. Likewise, Lua calling arbitrary C++ code through a module can result in exceptions reaching the Lua interpreter. It would be good if Lua can deal with Lua errors, and C++ can deal with C++ exceptions. To accomplish this, we really only need to focus on two areas:

1. Lua calls into C++ where there is a function wrapped in luabind::adapted
2. C++ calls into Lua using lua_pcall

Everything else can be left as-is. lua_pcall traps errors and returns a return code. In callWithReturnValue, callWithoutReturnValue, and loadScript, handleLuaErrCode checks the return error code, and if there is an error, we throw an exception based on the code’s value and the error string pushed onto the stack by lua_pcall. This ensures any C++ code calling into Lua can simply use try/catch to catch exceptions as a result of errors in Lua, and action can be taken depending on the type of the exception. If the exception escapes and the program stops, a useful error can be shown.

On the other side, luabind::adapted must be responsible for making sure no C++ errors make their way into Lua from native code, as this may cause the Lua interpreter to crash. To do this, we wrap the invocation of the callable with try/catch, and turn the error into a Lua error using:

try {//...
} catch (std::exception &e) {
  luaL_error(aState, e.what());
  return 0;
}

The result of the composition of these two error handling ideas is that authors in each language only need to think about idiomatic error handling in that language. They don’t need to know anything special about this library.

Tech Topic 6: Tables containing arbitrary typed elements

Luabind trivially supports vectors of different types being converted to tables in Lua. However, a vector in C++ can only contain uniformly typed elements, and the “key” is always a numeric index. In Lua, tables can contain non-numeric keys, and the values of the fields can be different types. Three standard candidates for the data structure are not suitable for different reasons:

• A struct in C++ would be a tempting way to store information that could be translated to a table–however, C++20 coes not standardize enough reflection capability to acquire the names of struct fields at compile-time.
• A map might be another tempting data structure in C++, but while maps contain pairs with keys you can easily read, the values have to all be the same type, just like vectors.
• Tuples might be nice since they can store fields with different types. However, the “key” is always numeric.

Instead, luabind introduces a new data-type called “table”, which allows tuple-like storage of multiple fields with arbitrary types. It improves on tuple by allowing compile-time operations on field names. Fields can be retrieved or modified with their correct types by addressing them by name.

The table class contains a tuple of fields. The field class stores both a value, and a “discriminator”. The discriminator’s only requirement is it must store the name of a field in a constexpr-friendly way. std::string would have been nice, but it has incomplete support for constexpr in MSVC. Instead, we just make an array of chars of sufficient size.

A convenience character literal operator, operator ""_f, converts a literal expression like “foo”_f to a discriminator containing characters ‘f’, ‘o’, and ‘o’. To provide some symmetry, table has a method “f” which is templated on the descriminator to retrieve the correct field.

The end result is a very interesting table experience in C++ with luabind, with no additional runtime overhead over something like a tuple:

TEST(LuaBind, Table) {
  using namespace luabind::meta::literals;
  using namespace luabind::meta;

  table<
      field<"biz"_f, int>,
      field<"buz"_f, bool>> bar{{1}, {false}};

  auto x = bar.f<"biz"_f>();
  auto y = bar.f<"buz"_f>();

  ASSERT_EQ(x, 1);
  ASSERT_EQ(y, false);

  bar.f<"buz"_f>() = true;
  ASSERT_EQ(bar.f<"buz"_f>(), true);

  bar.f<"biz"_f>() = 10;
  ASSERT_EQ(bar.f<"biz"_f>(), 10);
}