How to embed NimScript into a Nim program (Embedding NimScript pt. 2)
31st May 2020 - Guide , Nim , Programming
In the first part of this three part article series we looked at why we would want to embed NimScript into our Nim applications. Particularly why we would want to use it for configuration. Whether or not that is your goal, this article will explain how this can be achieved. When I started looking into this I started in the obvious place, Nimble. For those of you not very well versed in the Nim ecosystem, Nimble is the package manager that can be used both to install libraries, but also programs written in Nim. It can also be used as the build tool for your Nim applications as it will ensure you’re running the correct version of Nim and that you have all the defined dependencies (and that your dependencies are declared in your Nimble project file). But since its running NimScript it also supports build tasks, so you can define your own tasks and have Nimble run them. Much to my dismay however Nimble doesn’t actually embed NimScript. What it does instead is to take the .nimble file and prepend a series of imports, one of which is the Nimble API, and then simply runs the file with nim e which evaluates the file as a NimScript file. While this is rather clever, it wouldn’t work for using NimScript as a settings file. It could work to create a settings file however, just run the script and have it spit out some easily machine-readable configuration, but we wouldn’t be able to do things like register callbacks or other more complicated tasks like that. After abandoning Nimble as a viable candidate on how to do this I found two GitHub repositories, komerdoor/nim-embedded-nimscript and Serenitor/embeddedNimScript the former which haden’t been updated in four years and pointed be to use the latter instead. And the latter which had at least updated to the 0.20.0 version of Nim (Note: as of writing the current version is 1.2.0) but which I found quite opinionated (although it does have some nice NimScript type conversion which we’ll come back to later). Now after asking on IRC about this I finally got pointed to a test in Nim that shows how to use the “new” NimScript API from 2018. Throughout all this we’ll be looking at some code snippets, these are available as runnable examples on GitHub
Running NimScript
NOTE: This uses fairly new features of the compiler API, in order to install the required version of the compiler package simply run nimble install compiler@#head. You also need to be on the devel version of Nim. It is possible to achieve the same things as this without the devel versions, but for ease this article is limited to these versions.
The moment you’ve all been waiting for, running some NimScript from within our program. Let’s start of with the very basic, simply running a NimScript file:
import compiler/nimeval, os
let
stdlib = "/home/peter/Projects/Nim/lib"
intr = createInterpreter("script.nims",
[stdlib,
stdlib / "pure",
stdlib / "core"])
intr.evalScript()
intr.destroyInterpreter()
This is pretty simple, create an interpreter, evaluate the script, and destroy the interpreter again. You will see however that I’m passing my local clone of the Nim standard library (and two sub-parts of the standard library) to the interpreter to create it. But if we’re going to ship our application without Nim we will need to ship these with our program, more on this later.
By creating a simple script.nims file next to our binary and adding a simple echo "Hello world" in it we can see that running the compiled program will output “Hello world”. Change the string in the script and the output changes without having to re-compile!
Now this reads the script from the file script.nims, but we could also pass in our script as a string instead by opening it in a llstream and passing that to evalScript. This would then simply use the filename passed in for error messages and such, convenient if we want to prepend an import or similar to the script. Now this evalScript will run everything top-level in our program, which might be all that we need to e.g. run a build process. But we want to use this for configuration, so we need to be able to get information out from the script. We can use exportedSymbols to get all the exported symbols from our script (this would be everything marked with the Nim export marker *). And we can use getGlobalValue in order to get the value of a symbol.
import compiler/renderer # To get the correct `$` procedure
for sym in intr.exportedSymbols:
echo sym.name.s, " = ", intr.getGlobalValue(sym)
The above snippet will work for simple stuff like variables set to strings or integers. But exportedSymbols lists all symbols, so if you throw an exported procedure in there this will crash (we’ll look at calling these later). What we actually get back from getGlobalValue is a PNode which is the VMs internal representation of a node. We can check a symbol with sym.typ.kind against TTypeKind in compiler/ast to figure out what a symbol actually is, and from there parse the value into a Nim type. This is however still a manual process, and more complex types can be a bit challenging.
If we don’t want to iterate over all the exported symbols we can also get a symbol for any given string by using selectUniqueSymbol. So for example we can unwrap a sequence of integers by doing something like this:
import compiler/ast
let sym = intr.selectUniqueSymbol("testSeq")
var output: seq[string]
let val = intr.getGlobalValue(sym)
for data in val.sons:
output.add data.strVal
echo output
Of course if testSeq didn’t exist, or if it wasn’t a container of strings this would fail. In the accompanying examples I’ve added some extra checks to verify the type. But to avoid having to do all this checking, and write proper error messages for arbitrary types it is perhaps more useful to put the types you need in a string or file that you either append or import into the script. This means that not only is the script guaranteed to contain the variable, but since Nim is statically typed it means that it can’t change from the type you specify.
This isn’t too bad for sequences, but if we look at something like a table for example it’s easy to see why this is a bit less than ideal.
import compiler/llstream
intr.evalScript(llStreamOpen("""
import tables
let testTable* = {"hello": 42, "world": 100}.toTable
"""))
echo intr.getGlobalValue(intr.selectUniqueSymbol("testTable"))
Which will output:
(data: [(0, "", 0), (0, "", 0), (0, "", 0), (4220927227, "world", 100),
(0, "", 0), (0, "", 0), (0, "", 0), (613153351, "hello", 42)],
counter: 2)
Looking at the type of the value through the typ field shows us that it is a tyGenericInst and looking at the first child of the value tells us it’s a table (through sym.typ[0].sym.name.s). We can also use typeToString from compiler/types to give us a string representation, but this shouldn’t be used for anything else than showing it to the user. So while it’s definitely possible to look through data this way it’s not the most ergonomic thing.
Marshalling types for transport
Another option for passing complex data structures is of course to marshal our data-types into some transport format, and then unmarshal them on the other end. For example Nim has a pretty solid JSON module that can be used both on runtime and on compile-time. The above table for example when converted to JSON by the JSONs module convert to JSON operator % and then to a string with $ will look exactly like you would expect:
{"hello": 42, "world": 100}
And by using the JSON module to unmarshal the data on the compiled code side we can easily get the same complex type as we had in NimScript:
let
jsonstr = intr.getGlobalValue(intr.selectUniqueSymbol("jsonTable")).getStr()
parsed = parseJson(jsonstr)
table = parsed.to(Table[string, int])
echo table["hello"]
echo table["world"]
Of course this adds a couple of extra steps, but as a quick and easy way to get data to and from scripts it is definitely a good option. And you can easily inject some code that would do this conversion with the user of your embedded NimScript being none the wiser.
There are of course also other ways to marshal this data, I simply chose JSON here as it’s easy to work with.
Calling between script and compiled code
Up until now we’ve only looked at scripts that will execute everything in their main body as soon as we call them. But what about dynamically interacting with a script?
So what if we wanted to allow the script to react to certain things we do in our program? For example if we’re using it for configuration we might want to configure keyboard shortcuts, or possibly allow the script to trigger actions on certain events. For this there is the selectRoutine procedure, it is a wrapper around selectUniqueSymbol that overrides the symbol kinds that can be returned to only accept templates, macros, functions, methods, procedures, and converters. We can of course do this overriding ourselves, for example if we specifically only want to accept functions (to ensure that the callback can’t have side effects) we can use intr.selectUniqueSymbol("<symbol name>", {skFunc}). With the PSym this returns we can then use callRoutine to actually call the procedure. But again we run into the same issues with PNodes not being easily marshallable to Nim types:
let
foreignProc = intr.selectRoutine("scriptProc")
ret = intr.callRoutine(foreignProc, [newIntNode(nkIntLit, 10), newIntNode(nkIntLit, 32)])
echo ret.intVal
This scriptProc that we select here obviously needs to exist in the script, and have the signature we expect. This can be ensured by forward declaring the procedure in an implicit import or addition to the script (this is done in the example code).
But what if we want to expose certain things in our code to the script, ie. to let the script call procedures we define? This can be useful to allow it to do IO or other things NimScript isn’t allowed to do by default, or just as a way to have it affect our state in a deeper way than simply returning values. In my WM for example it would be natural to expose things like “fullscreen current window” or “move window to workspace” and allow the script to call these things from a keyboard shortcut callback. For this we need to use implementRoutine. Note that this can also be used to override an implementation of a procedure, so a library could contain dummy procedures that just throw an error when called, and then implement these in the compiled program. This would allow tools to see that the procedure exists and what signature it has, this is used in the stdlib for a variety of procedures, and createInterpreter will register these routines for you. Note that when implementing routines this way they still have to be declared in code somehow for the script to work:
import compiler/[vmdef, vm]
intr.implementRoutine("*", "script", "compilerProc", proc (a: VmArgs) =
echo "This is happening in compile-time code"
a.setResult(a.getInt(0) + a.getInt(1))
)
intr.evalScript(llStreamOpen("""
proc scriptProc*(one, two: int): int
proc compilerProc*(one, two: int): int = discard
""" & script))
let
# The scriptProc found in the runnable example calls compilerProc, so this calls scriptProc which
# then runs the above implementation of compilerProc
foreignProc = intr.selectRoutine("scriptProc")
ret = intr.callRoutine(foreignProc, [newIntNode(nkIntLit, 10), newIntNode(nkIntLit, 32)])
echo ret.intVal
The arguments to implementRoutine are package, module, and finally the name of our routine and the procedure itself. Package is stdlib for things in the standard library, otherwise you can use * or the name of an external dependency if you have any. Module must be the name of the module where this is found, if you use the same name as the file it will be seen as local, otherwise it will only be available if the script imports that module. The arguments are again a bit tricky to use as they come in as VmArgs. There are however some convenience procedures in compiler/vmhooks.nim to set the result to different types and to get arguments of different basic kinds like integers, booleans, and strings, but also to get PNodes for the more complex types.
Conclusion
With the new compiler API interfacing with NimScript is fairly simple in Nim, however the marshalling of types can be a bit tricky. In the next article we’ll look at how we can deploy NimScript without having to ship the full Nim stdlib so it can be shipped as a more standalone application.