Meta-programming in Nim - FOSDEM talk companion post

5th February 2019 - Presentations , Nim

This article is intended as a companion to the lightning talk I held at FOSDEM 2019. If you haven't seen the talk yet the official recording from the FOSDEM site is embeded below (if you want you can also grab the slides for this presentation on that site).

After all the fun we had at FOSDEM last year I decided to go again this year, but this time I wanted to contribute something. So I decided to submit for a lightning talk, a short distilled presentation that might get more people interested in trying out Nim. But instead of doing yet another introduction to Nim talk that would only scratch the surface and not get into anything that was particularly thrilling I decided to do a more specific presentation on a feature of Nim. As you probably now know the feature I landed on was meta-programming. It was certainly one of the aspects of Nim that caught my attention when I first got into using the language, and a feature that can be used across all kinds of different projects for all targets.

My talk was mostly based off of this very good article on metaprogramming in Nim. But I wanted to fill in some of the gaps that I didn't have time to go over in my talk with this post. Since the talk is intended to mostly be about the meta-programming aspects I didn't really have all that much time to go through the actual introduction to what Nim is in any more detail than a couple of bullet points. And this list of bullet points tend to raise some questions, I certainly was sceptical of what Nim promised the first time I read about it.

Deeper look at Nim

First off let's have a look at the targets, it compiles to C/C++/Objective C along with JavaScript. Now this might look like a weird combination, and surely not everything that works in C can work in JS. In fact these targets are quite different, but they all share many of the great language features like the strict type system, and of course meta-programming. This means that you can develop full stack applications in a single language, without writing a single line of JavaScript. In fact projects like the Nim forums have a server part that compiles to C and a front-end part that compiles to JS. And Reel Valley, which is a game written entirely in Nim, has a server in Nim compiled to C, and a client in Nim that compiles to C for it's Windows, Linux, iOS, and Android targets, and through Emscripten to WASM for it's Facebook target. This means that the same code-base could run as desktop applications, phone applications, and in your browser (it is techincally possible to use the JS target for this as well, but the speed of JS is often not sufficient). Another thing that usually comes up when mentioning that Nim targets C is what kind of speed you can expect. For surely when compiling to C you can't be faster than hand-written C code right? In much the same way that the C compilers are able to spit out more optimised assembly code than most people can write, Nim will output highly optimised C code that will actually outperform hand-written C code in many cases. Of course the possibility to write better C code is always present, but it would be horrible to read, something which Nim doesn't need to think about when generating it's output.

The next thing that might have raised some eyebrows is that Nim uses a garbage collector. But unlike the implementations in other languages like Java and JavaScript, which have given garbage collectors a bad reputation, Nim's garbage collector is a lot more controllable. In fact you are able to completely turn it off and manage memory manually, or you can control when and for how long it is allowed to run, or just let it be and do its thing. This means that Nim can fit for every task from micro-controller programming to games and servers that don't want GC lag spikes, or just your everyday script or application.

More about meta-programming

With that out of the way it's time to get to the topic of meta-programming. Now meta-programming is not unique to Nim, but it's fairly uncommon for compiled, imperative languages. This is partially caused by the fact that in order to do it Nim has itself as a target in a sense. Since macros are written in Nim the compiler needs to be able to execute arbitrary Nim code. This means that the compiler is actually running a Nim VM that runs your macro code. This does have some limitations, for example it's not possible to call C libraries like you normally can in Nim from inside a macro. It is however possible to do things like read and write files, run external scripts and programs, etc. Alongside the AST-based templates and macros Nim is also able to use this VM for regular computation, passing around complex compile-time object structures and everything else your normal program can do. This means that you are able to write simple pre-compute macros to e.g. embed a sine values table in a micro-controller program, or parse the interesting parts out of a third party format, all without having to copy-paste things or make complex custom pipelines. I've even implemented a Protobuf parser directly as a macro so you always know that your compiled Nim program is using the latest version of your Protobuf specification.

Now for a word of caution. I mention in my talk that meta-programming can be used to make things safer, faster, and more read- and maintainable. But as with any abstraction it also has its pitfalls. If you do something wrong in you macro it might be very hard for the person using it to understand why it doesn't do what it's supposed to. Take this simple example:

template checkIt(it: int) =
  if it == 3:
    echo "It is 3: ", it

var it = 0
proc addToIt(): int =
  result = it
  it += 1

for _ in 0..5:
  checkIt(addToIt())
  echo "it: ", it

You might expect this to output something like this:

it: 1
it: 2
it: 3
It is 3: 3
it: 4
it: 5
it: 6

But what we would actually see is that when we use it to output the value after the check, the addToIt procedure gets called again and the output becomes:

it: 1
it: 2
it: 3
It is 3: 4
it: 5
it: 6
it: 7

If we wrap line 11 in a expandMacros debug call we can see what our template actually does:

if addToIt() == 3: echo ["It is 3: ", addToIt()]

This is because it's the AST that we pass in that it expands to, and not the value. The same reason is why our optimised logger worked, so it's a practical feature once one is aware of it. But it's important to make sure that your macros and templates have a very well-defined behaviour so that the user doesn't get confused. The wxWidgets macro I demo'ed at the end of the presentation tries to solve this by being a very simple mapping from DSL to wxWidgets code. If you know how to write wxWidgets code and you have read the simple list of substitutions it should be rather obvious what the output of the macro is going to be. This also means that it is easy to read the official wxWidgets C++ documentation and adapt that into the macro. Generally speaking you want to do as little magic as possible with your macros.

Final remarks

If you are intrigued by what you've seen so far feel free to read up on the official Nim site, peruse this web-site for some more in-depth posts, or head over to the community page to find your favourite way to ask us questions.