Fixing a broken Firefox session
27th April 2021 - Guide
In order to stifle my tab-hoarding habits I have recently set my Firefox install to not recover sessions when I restart the browser. This means that every time I turn my computer off for the day it will delete all the random documenation pages and weird links people have sent me over the course of the day so they don't slowly add up. My habit was so bad that I could easily amass some 100-200 odd tabs before I cleared them out. Part of the problem here is that I've just never been much of a bookmarks person, so I've just kept all the stuff I might want to read one day open as a tab. My new system of starting with a fresh session every time would however be a bit tedious if it closed absolutely all my tabs. As you're probably aware there exists a whole slew of web-apps nowadays, some of which I'd like to have open with my browser. For example my webmails and various chat applications (both for work and for leisure), my HomeAssistant dashboard to be able to easily control various things around the house, and a couple of those aforementioned articles I just wanted to save for later. Firefox luckily has a pinned tab feature for this, where you can pin a tab in a window and it will load all the pinned tabs when you open the browser. These tabs survives a browser restart, even when you don't have session recovery turned on, and unlike just leftover tabs they don't require you to click on them when you first open your browser to load them. Exactly what I need!
Where things went wrong
So you might imagine my dismay when I opened my browser this morning, just to discover that all my pinned tabs where gone! Of course since I only have about 6-8 of them it wouldn't be much work restoring them by hand, but where's the fun in that! And if this was a repeat issue I certainly wouldn't like to have to do it every time. Asking around on the #firefox IRC channel I was quickly pointed to ~/.mozilla/firefox/xxxxxxxxx.default/sessionstore-backups which contains the previous.jsonlz4 file that holds the previous browser state. Quickly copying this to a safe location in order to not have Firefox rewrite it I was told that simply replacing ~/.mozilla/firefox/xxxxxxxxx.default/sessionstore.jsonlz4 with this file would reset my session. Unfortunately after shutting down the browser, replacing the file, and restarting it yielded exactly the same results. So it was time to dig deeper!
The mozlz4 file format
Having a look at the extension of this file quickly made me realise that this was some kind of compressed JSON file. So I opened it up in Xarchiver which was happy to extract a nice JSON payload for me. Looking into the JSON file it was easy to see I'd come to the right place, there where all my pinned tabs, even with history! But the problem seemed to be that I had closed the window which shows the pinned tabs and later closed the browser itself. As a side note the great jq utility that can, amongst other things, pretty print JSON, came in handy here. In this file there is a windows array along with an array named _closedWindows. And all my nice pinned tabs where in the latter. Easy enough right? Just move my window from the _closedWindows over to the windows array, right? Well, that would fix the JSON, but we still need the original compressed file format. Looking at the extension it looks like it would be a normal LZ4 compressed file. But looking at the output of file show that this is "Mozilla lz4 compressed data". Yay.. A proprietary format, great. Looking for ways to create these files it seems like the format was actually created before LZ4 had a standard on-disk format, so Mozilla created their own. This fortunately means that it is fairly trivial to convert between the two once you know how. And even looking at what Xarchiver was running to do the decompression was simply a re-structured version with the standard LZ4 decompression tool.
Creating the header
Looking around the web the mozlz4 format is said to have an 8 byte magic sequence at the start, this is a common thing in file formats that allows utilities to recognise the file even if the file extension is unknown or missing. In this case that 8 byte magic is simply the NULL terminated string "mozLz40". This is followed by a 4 byte 32-bit integer (probably unsigned) recording the original file size in little-endian. Generating this header is easy enough, a simple cat fixedsession.json | wc -c gives us the amount of bytes in the actual file, and a simple echo -n along with some hex escape codes can be used to output text to a file without a newline. The normal LZ4 header is only 11 bytes, so we need to strip that of first, this can either be done with tail -c+12 fixedsession.json.lz4 > fixedsession.json.lz4.noheader or with dd bs=11 skip=11 if=fixedsession.json.lz4 of=fixedsession.json.lz4.noheader. Now we can just concatenate those two files together with the result of the previous echo and the new headerless LZ4 file and we should be good.
Not so fast there! After all this work, much to my dismay, I got a decryption error from Xarchiver when I tried to open the file. So something was obviously still not right. And it turns out it was just a matter of how each format detects the end of the stream. In the standard LZ4 format the end is defined as the last block having a size of 0. This essentially means that there is a 32-bit 0 at the end of the file, possibly followed by a 32-bit CRC checksum. The Mozilla proprietary format on the other hand seems to just decrypt until it has the amount of data defined in the size field of the header. So when Xarchiver added this 0 size block to re-headered version it now had extra data at the end of the file, which apparently is treated as an error. For some formats this isn't an issue, and this fact can be used to hide two files (such as a JPG and a ZIP file) in one, making it appear as either when opened with a standards compliant program. My guess is that Firefox would probably have read the file just fine with the extra bytes on the end, but I decided to strip them off with a simple head command just to be sure. And lo and behold, it worked!
But why?!
Most people when they encounter issues like this will just curse their computer, the software, or the developers then manually restore their stuff and go about their day. And I probably didn't save any time doing it this way (well, as long as this happen only once, now I know what to do it would probably be faster to do it this way). But every once in a while it's nice to take a little deep-dive into how your programs actually work. You'll probably learn a little bit along the way, and you'll most certainly get better and solving similar problems for the one time you might actually really need to. And I've written it down here, not only to potentially help a lost soul trying to find a way to restore thousands of tabs, but also to remind myself of how to do it in the future, or just as a pointer to the kind of things to look for when doing stuff like this.