Creating condensed shared libraries (Embedding NimScript pt. 3)

23rd June 2020 - Guide , Nim

In the first part of this article series we looked at why we would potentially want to be able to run NimScript embedded into our Nim programs. In part two we dove a bit deeper and looked at how we could do this embedding, and how we could pass data from the scripts to our native code. In this final article (for now) we’ll look at how we can distribute a program that embeds NimScript, and how we can create our own standard library distributions depending on what we need.

Distributing NimScript

When embedding NimScript into our application we are essentially bundling the compiler into our code. Or at least the parts that are required for NimScript. And since Nim creates nice standalone binaries it might be easy to think that after this embedding has taken place we have everything we need to ship our program. But that is not quite the case. When NimScript is evaluated it still requires certain parts from the standard library of Nim. If the user has Nim installed already, which is the case for Nimble and other Nim ecosystem tools that uses NimScript already, this isn’t an issue. But if we want to use NimScript as a configuration language in a standalone project we need a way to ship the standard library features that we need.

What we need

When compiling NimScript support into our program it adds about 3Mb to our binary size. But the standard library adds another 5Mb (only 1.2Mb zipped though), which is comparatively pretty big. This will of course go up if we have any external dependencies that we would want to add, should their licenses permit it. In addition to this, some parts of the standard library doesn’t work in NimScript for various reasons. Depending on what you intend to do with your NimScript, or allow your users to do with it, you might also want to remove certain modules that don’t really make any sense to include simply because no-one is likely to use them. You could of course easily support using a local library instead of the one you ship. This way people who have Nim install could use the entire standard library, and it could also allow for Nimble packages to be used.

Getting the used libraries

Importing a module in Nim obviously requires the file that this module is implemented in, but it also relies on the files of the modules it imports, and so on. In addition Nim always requires the system.nim file and its dependencies. Figuring out exactly which files you have to ship manually can be a bit tedious, so I set out to create a way to do this automatically. My first approach was simply to run nim e to execute a NimScript snippet multiple times while deleting files from the target library folder until it couldn’t delete more files without failing. This worked, but it isn’t exactly the cleanest solution..

Nim compiler to the rescue

As with my initial experiments with running NimScript it turns out the solution was under my nose the entire time. The Nim compiler package is really flexible and can be used to implement all sorts of tools that need to think like the Nim compiler. So it wasn’t too hard to start with nimsuggest (which uses the compiler sources, and which I was familiar with from my work on nimlsp) and the code path for --genDeps:on in the compiler and modify it to load modules from the standard library and register the used dependencies in a table. This allowed me to write a simple tool that can be used to extract only the modules required and their dependencies.

Conclusion

In case you want to build NimScript into your application and want to shave off a couple megabytes this is certainly a possibility. There is room for improvement, for example by removing comments and things behind when not defined(nimsuggest), but at the end of the day there really isn’t too much to save.