Hide
Scraps
RSS

Dynamic libraries in Nim

20th February 2023 - Guide , Nim , Programming

This is a question which has been asked in various places over the years, and recently on the forum in multiple separate threads. Seeing how I’ve built commercial software with Nim dynamic libraries, I thought I’d chime in with my knowledge. But since there are multiple threads out there and this is a large-ish topic I figured it would be better to do it as a post on here.

The problem

As you’re probably aware Nim compiles via C, and this means that it should be possible to write dynamic libraries in it. Whether it’s for loading into an existing program or you want to write a Nim program which loads dynamic libraries the question is much the same. How can we achieve this? It might be tempting to simply throw on the --app:lib compilation switch and think that we’re done, but it’s not quite that easy. We essentially have three problems:

  • Nim and C differs quite a lot, we have to make them able to talk to each other
  • Nim has a GC, so we need to figure out how to control that from a dynamic library
  • And last but not least we need to actually compile a dynamic library and possibly load it

Problem 1: Nim/C interface

As I said Nim and C are quite different, and if we want to create a dynamic library which is loaded by a third party we need to make sure that external application can actually understand what our Nim dynamic library is doing. This is actually quite a large topic, and one I’m currently in the process of writing another article about, but there are some things we particularly need to keep in mind. First of if we want to use procedures from the program that loads us or let the loading program call any of our procedures the signatures must match exactly. This means that the arguments, the return type, and what’s called the “calling convention” all have to match. For arguments and return types it’s fairly simple, integers in C turns into cint in Nim, strings in C (char *) turns into cstring, floats in C turns into cfloat etc. You might get away with saying an integer in C is an int in Nim, but that’s not technically correct and might break on some platforms, so better safe than sorry and use the correct types. If you get passed a pointer to a struct you need to recreate that object precisely, and then use ptr SomeObject, if you get a pointer to an object and a length with the idea being to iterate over it as an array you should use ptr UncheckedArray[SomeObject] as this will tell Nim that it can treat the pointer as an array, but without any bounds checks. This means that you need to manually check the length or risk dropping of the end of your array and going into potentially unallocated memory and crash. Remember, these signatures need to match exactly!

Now for the “calling conventions”, essentially just a fancy term for how the underlying machine code is generated. Different compilers have come up with different nifty ways of doing this, and it again has to match exactly. If the C sources doesn’t specify a calling convention (they probably don’t) you should use {.cdecl.} as that essentially means “do what C would do”. This is again a case where you can get it technically wrong, but it could still work on your machine because your compiler happen to use the calling convention you chose. But you should use the correct one none the less.

Because of C’s fairly weak type system, and the fact that Nim trusts you completely with the signatures you say come from C, writing a proper wrapper can be painstaking work. If you’re wrapping any amount of C code past trivialities I’d really recommend using Futhark as it makes this work incredibly much easier. But that’s the topic of my next article, so let’s not get into it here.

Problem 2: Nim has a GC

Nim is a garbage collected language by default. This means that unlike C you don’t have to worry about calling malloc and free to get and release memory. Instead Nim will at carefully selected points during execution free up any resources which aren’t being used, and maybe even stop for a very short while to scan around for cycles. Nim also has global variables which can require more complex initialisation than what a typical C global variable needs. Normally you don’t have to think about this, Nim will generate a main function that is where normal programs start execution which deals with setting all this up for us. But when creating dynamic libraries things are a little bit different. By default Nim will hook into either DllMain on Windows or __attribute((constructor))__ on POSIX to create hooks which by convention will be called whenever the dynamic library is loaded. This should work, but many dynamic library systems will include some library_init and library_deinit procedues that will be called by the loader of the dynamic library so resources can be allocated and properly freed (and if you’re creating a similar system you will very likely want to add these to your system as well). The recommended way to ensure that the initialisation is done is to call it from this procedure. To achieve this compile with --noMain which disables the default main initialisation hook, and then call a procedure called NimMain ourselves. This will set up the garbage collector and initialise any global memory. Be sure to call this as soon as possible in your code, if you accidentally use garbage collected memory before you call this then that will cause issues. To access this procedure in your Nim code you need to add this definition:

proc NimMain() {.cdecl, importc.}

If you load more than one dynamic library written in Nim into the same program, or if you load one dynamic library written in Nim into a program written in Nim and you want to pass Nim garbage collected types (including raising exceptions) between them they must agree on which memory management system. This is done through an extra dynamic library named nimrtl.(so|dll|dynlib) (short for Nim RunTime Library). To build this you need a copy of the Nim library which is often installed along with Nim itself. To build this library see this section of the Nim manual and then use -d:useNimRtl when building your program or library to link against this.

If you only communicate between the Nim dynamic libraries/program with manually managed types this step is not required. In this case they aren’t really aware of the fact that the pieces aren’t in fact just normal C code.

I would also reccomend to compile with what is going to become the new default in Nim 2.0, namely --gc:orc. Simply add that to your compilation command or nim.cfg file and you’re good to go. If you compile with --gc:arc there is technically nothing to set up for the GC in NimMain, but to initialise global variables (including from imported libraries) you should still make sure to call it.

On the topic of global variables, if your dynamic library is unloaded the loader should call library_deinit or similar. Nim doesn’t currently support a way of freeing all its allocated memory (it is simply assumed that when the process exits it is done automatically). So nulling out all global memory and calling GC_FullCollect would be a good idea to do it this case. This oversight is documented and work on fixing it is tracking in this GitHub issue

Problem 3: Compiling a dynamic library

During the last two sections you’ve seen a couple of switches that helps with this problem. And you’ve seen some pragmas to generate proper Nim code for export. However our job is not quite done yet. Every procedure you want to call and which will be made available through the dynamic library has to be labeled with {.importc, dynlib.} and everything you want the library loader to see needs to be labeled with {.exportc, dynlib.}.

To compile you will need to use --app:lib, --noMain if you intend to call NimMain yourself (recommended), and potentially -d:useNimRtl. You can also throw on any other flags like normal. And with that we should have a fully functioning dynamic library, written entirely in Nim! The final result would look a little something like this:

proc NimMain() {.cdecl, importc.}

proc library_init() {.exportc, dynlib, cdecl.} =
  NimMain()
  echo "Hello from our dynamic library!"

proc library_do_something(arg: cint): cint {.exportc, dynlib, cdecl.} =
  echo "We got the argument ", arg
  echo "Returning 0 to indicate that everything went fine!"
  return 0 # This will be automatically converted to a cint

proc library_deinit() {.exportc, dynlib, cdecl.} =
  echo "Nothing to do here since we don't have any global memory"
  GC_FullCollect()

And it can be compiled with something a bit like this:

nim c -d:release --app:lib --noMain --gc:orc ourlibrary.nim

View all entries