Server migration and downtime

21st October 2017 - Web , DevLog

This site has been down for a couple of days now. In fact the entire server has been down, which probably pained me more than it did you. Reason for this is that I have various services running on this server, most importantly perhaps my private git server. I've written before how I use git to maintain this site but that is only one of the many things I do with it. Typically I will start working on a project on my own server, then when it's more mature I will upload it to GitHub for sharing. The server also serves as an easy way for me to upload files I want to share, simply by uploading them to a shared directory. However over the years I have done much more stuff with the server, and it was showing. My Apache config was a horrible mess, and various services weren't quite running as they should (not to mention it was probably not very secure). So when I got an e-mail from the interest-group at the university kind enough to host my server that the university was rebuilding that part and all the servers would have to go I was a bit disheartened. Fortunately the same people were also able to offer a Virtual Machine hosting solution were all the servers previously running as physical machines would now be able to be run on the main rack as virtual machines instead. The only problem was that during the time of the move I was busy finishing off my master thesis. So it wasn't until now I've found time to actually migrate my server.

The first misguided attempt I tried was to simply create a virtual image of the physical machine. This wasn't optimal as I mentioned above the server was quite a mess and should be reconfigured anyways. However I wanted to get the server back up as quickly as possible and this seemed like the way to do it. Only problem was that the old drive was larger than the amount of space I had on the Virtual Machine so it wasn't a simple matter of just dd-ing the drive to an image. After figthing with LVM for a couple hours I finally had an image with all the data and took it over to the university to get uploaded. Lo and behold however, it didn't boot. I was fearing as much, but honestly I had given up hope long before this.

After a couple more days I finally found time to get the server configured from scratch. Armed with a backup of my old drive I got going. What follows is my notes for the day on how I pieced everything back up.

• First order of business, only allow log-in through SSH. Uploading my public key and disabling password only log-ins
• Then we need some security, arno-iptables-firewall has been configured to only allow the small sub-set of ports required to go through
• Next was Apache to get this site back up. Disabled the default site
• Then MySQL to hold the database for this and other projects requiring a DB. I initially set up a new user for this site, but removed this later
• Certbot was next, this time I've set up the entire thing to use properly signed certificates from the start. This was a bit of a pain to retrofit the last time
• Checking the configuration on https://www.ssllabs.com's ssltest showed some areas that could be improved, but otherwise a good score
• Now it was time to actually get the site up and running. I had forgotten to take a proper backup of the MySQL database so I decided to try and hack it. After disabling MySQL and backing up it's data directory I uploaded and overwrote the entire thing with my previous files.
• Initially this didn't seem to work, but it turned out to only be an issue with file permissions (MySQL logs to the rescue, was about to scrap the entire idea).
• As I've mentioned earlier this site is set up through git. All the content is stored in a DB and managed with CouchCMS, but the PHP, HTML, CSS, and JS is under git. So it was time to set up gitolite. Created a user and cloned their repository into it as intstructed on their site (this was previously done with a PPA, not quite sure what I liked best)
• Copying all the old repository data directly into the new git user almost didn't seem to work, but again it was just a configuration mishap. This time it was because the gitolite-shell binary referenced from the authorized-keys file had moved.
• I quickly changed the hook for the devlog repository around to point it to it's new location on the server. In the old setup all the files were in a single directory which was a mess. The new system uses Apache's sites functionality to split each site into it's own directory. More flexible and easier to keep track of.
• Cloning the repo and adding an empty commit allowed me to push the repository for this site back and trigger the git-hook. Partially done to check that everything was working as it should.
• Had to install some PHP packages for couch to work properly, namely the XML and MySQL packages.
• The git hook tries to run a compression on CSS and JS files, this didn't work however (probably because the compressor runs on Java which isn't currently installed) so they ended up empty. Thought I had made a configuration error and wasted a lot of time trying to figure out why my Apache config was failing, which it wasn't. Disabled the compressor for now.

And here we are, took almost the entire day on and off but now everything should finally be back up and working. During the downtime and over the summer I've encountered some cool things that I want to write about here, so more content is hopefully coming very soon.