My First Nix Derivation: OpenStreetMap Overpass Server

Browsing the nixpkgs repository and looking at random packages reveals a common structure to the attribute set passed to mkDerivation. First I started with the pname and version, which was the easiest part as those can come from the overpass archive name. The first roadblock was actually successfully downloading that archive.

Downloading the source archive

There were two issues downloading the source archive, so let’s tackle them in order.

The first was making sure the version numbers don’t drift out of sync between version = and the version in the url string. We can do that by using interpolation. But there’s a problem, if you try to reference version from in the string, you get this error:

error: undefined variable 'version'

       at /home/danielperry/repos/nix-scripts/openstreetmaps_overpass/osm-3s.nix:10:58:

            9|   src = pkgs.fetchurl {
           10|     url = "http://dev.overpass-api.de/releases/osm-3s_v${version}.tar.gz";
             |                                                          ^
           11|     sha256 = "02jk3rqhfwdhfnwxjwzr1fghr3hf998a3mhhk4gil3afkmcxd40l";

The trick is to make the attribute set you pass into mkDerivation recursive, by adding rec before the {:

pkgs.stdenv.mkDerivation rec {

The second issue I ran into was trying to come up with the sha256 value. At some point in my learning I read that “most people just throw in 0’s and let Nix tell you the correct value”, however I must have been mixing that advice up with advice for some other part of Nix, as that doesn’t work here.

If we go with that approach:

  src = pkgs.fetchurl {
    url = "http://dev.overpass-api.de/releases/osm-3s_v${version}.tar.gz";
    sha256 = "0000000000000000000000000000000000000000000000000000";
  };

error: hash mismatch in fixed-output derivation '/nix/store/n5qkklx87s1fqccxi9dw12y8iy4igr8q-osm-3s_v0.7.59.tar.gz.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-FJDWWZ1ODRofmRDWoVBKDo4Mnwv5c9m5dbBxB3EeUwo=

You can see the hash in ‘specified’ isn’t the 0’s we supplied. This hash is clearly in a different format than what we entered, so we can’t expect the ‘got’ value to work inside the nix script. We’ll need to try something else.

Instead, the suggestion is to use nix-prefetch-url to download the tar.gz and let it tell me what the sha is:

$ nix-prefetch-url http://dev.overpass-api.de/releases/osm-3s_v0.7.58.tar.gz
path is '/nix/store/alvjh48vhar8js00vrmb56q0qnb4ws9m-osm-3s_v0.7.58.tar.gz'
1ipig0w3327nhl7dldcgkcm3fyhrib6h82n030rkb32bj9zr0g92

And indeed, this (1ipig...) is the value that needs to be given for Nix to be happy with the downloaded gzip archive.

With these changes, we now have something for Nix to build. But the build fails pretty quickly due to missing dependencies.

error: builder for '/nix/store/v1bxx04iix74x6xqx75sb8s195ipdy1m-osm-3s-0.7.59.drv' failed with exit code 2;
       last 10 log lines:
       > overpass_api/data/../statements/../dispatch/../../template_db/zlib_wrapper.h:22:10: fatal error: zlib.h: No such file or directory
       >    22 | #include "zlib.h"
       >       |          ^~~~~~~~

Side Note: I’m recreating most of these errors long after the fact, Nix’s reproducibility has been a boon here to help simulate the experience 😊

These errors aren’t unexpected. After all, the Overpass Installation wiki page clearly lists the dependencies needed to build, and we haven’t done that yet:

Install the following packages: g++, make, expat, libexpat1-dev and zlib1g-dev.

Determining Build Inputs

g++ and make are used by mkDerivation by default, so all that was needed was to figure out how to make expat and zlib available during the build. Luckily, other packages in nixpkgs already reference both of these libraries, so it was as simple as figuring out how they did that and copying it into my script:

  buildInputs = [pkgs.expat pkgs.zlib];

We complete that, and it turns out that’s all we needed! We have a building derivation:

$ time nix-build ./osm-3s.nix
...
/nix/store/sbj1c6cjjbh97gn9zrhz2g2b4hns0v2i-osm-3s-0.7.58

real    8m29.319s
user    0m0.135s
sys     0m0.032s

But…it’s kinda slow, though. It took my Ryzen 5800X over 8 full minutes to compile everything. When I built this outside of Nix, it was much faster. What gives? A quick look at htop shows the issue: compiling only uses a single thread. There must be a way to enable multithreaded compiling.

A quick google search shows the answer, we just need to give the mkDerivation function enableParallelBuilding = true. Re-run the build and now:

/nix/store/sbj1c6cjjbh97gn9zrhz2g2b4hns0v2i-osm-3s-0.7.58

real    1m26.673s
user    0m0.139s
sys     0m0.025s

Great! We now have a reproducible build of the Overpass server binaries. The only final thing remaining is the Overpass wiki suggests compiling with CXXFLAGS=O2. A quick search on Nix documentation suggests that this flag might be supplied to g++ by default, however you can also explicitly supply it by simply adding CXXFLAGS = "02" into the attribute set you pass to mkDerivation.

You can view the current version of that script on my GitHub here.

The most helpful resource I could find for examples making Docker images from Nix came, probably unsurprisingly, from the nixpkgs repository. The Docker examples.nix file was invaluable in trying to figure out how to get the right parameters to pkgs.dockerTools.buildLayeredImage.

The first step is to generate a Docker image based on httpd, containing the Overpass binaries.

First Steps

Using that repository as a resource, I was able to get an image that contained the Overpass binaries, and had an idea on how to pull httpd. The tricky part was, as always, getting the sha256.

Starting simply with only x86_64 support, I had the docker image ready to pass to pullImage. I could fill in most of the blanks, but how do I get the sha256? There’s no url to give to nix-prefetch-url this time!

Google to the rescue, there’s a tool called nix-prefetch-docker you can use for Docker images. How convenient! How do we run it? The same way you run any Nix-specific tool, with nix-shell.

$ nix-shell -p nix-prefetch-docker

$ nix-prefetch-docker --image-name httpd --image-tag 2.4.54 --image-digest sha256:15515209fb17e06010fa5af6fe15fa0351805cc12acfe82771c7724f06c34ae4

...
{
  imageName = "httpd";
  imageDigest = "sha256:15515209fb17e06010fa5af6fe15fa0351805cc12acfe82771c7724f06c34ae4";
  sha256 = "1r3zvfas5nb757z26gjmmdkk4hzbrglmj2q9ckhkhdjf77c29qzr";
  finalImageName = "httpd";
  finalImageTag = "2.4.54";
}

Simple enough. And it works! Well, it builds. We get a .tar.gz that contains a docker image split out into intelligent layers, which contains the Overpass binaries and Apache. However, nothing runs in the docker image yet. Until you specify config.Cmd in the nix script, Docker won’t automatically run anything. That’s okay for now, though. One step at a time.

At this point, the call to buildLayeredImage looks roughly (not exactly) like this:

pkgs.dockerTools.buildLayeredImage {
  name = "osm-3s-static";
  tag = "latest";
  contents = [
    osm3s
  ];
  fromImage = {
      imageName = "httpd";
      imageDigest = "sha256:15515209fb17e06010fa5af6fe15fa0351805cc12acfe82771c7724f06c34ae4";
      sha256 = "1r3zvfas5nb757z26gjmmdkk4hzbrglmj2q9ckhkhdjf77c29qzr";
      finalImageName = "httpd";
      finalImageTag = "2.4.54";
    }
}

Now, let’s add multi-architecture support!

Arm64 Support

First off, we need the info for the arm64 version of httpd, which is simple enough to retrieve. Just swap the image digest you pass to nix-prefetch-docker to the arm64 version of the same image. It’ll spit out another attribute set for that version.

Next, how do we make Nix choose appropriately? For now, just to learn how this works, I decided that this script will build the docker image for the architecture you’re running the script on. If you want an arm64 image, run the script on arm64.

Note: It’s important to know that cross-building is possible with Nix. There’s no reason you couldn’t make an arm64 docker image from an x86_64 processor. This script doesn’t do that, though. Again, this is more of a journal than a tutorial 😊.

I managed to accomplish this through the variable pkgs.stdenv.hostPlatform.linuxArch. As expected, this variable contains a string with the architecture of the host platform. I use this string to index into another attribute set I define with all of the platform docker images.

Abbreviated to just the relevant code, this is what it looks like in Nix:

let
  basePlatformImages = {
    "x86_64" = {
      imageName = "httpd";
      imageDigest = "sha256:15515209fb17e06010fa5af6fe15fa0351805cc12acfe82771c7724f06c34ae4";
      sha256 = "1r3zvfas5nb757z26gjmmdkk4hzbrglmj2q9ckhkhdjf77c29qzr";
      finalImageName = "httpd";
      finalImageTag = "2.4.54";
    };
    "arm64" = {
      imageName = "httpd";
      imageDigest = "sha256:8b449db91d13460b848b60833cad68bd7f7076358f945bddf14ed4faf470fee4";
      sha256 = "1a0b23pk5lf0fa2z1shggzmcskmj378rafdpfppwg8id6kfwfcgj";
      finalImageName = "httpd";
      finalImageTag = "2.4.54";
    };
  };
  currentBasePlatformImage = basePlatformImages."${pkgs.stdenv.hostPlatform.linuxArch}";
in
pkgs.dockerTools.buildLayeredImage {
  fromImage = pkgs.dockerTools.pullImage currentBasePlatformImage;
}

With this, I can make both x86_64 and arm64 images with the same Nix script. Now, let’s make it run!

Getting Overpass running

When you run an Overpass server for yourself, you first need to seed the database somehow. Let’s ignore this initial step for now, and assume there exists a database ready to go somewhere on disk. Thanks to my prior experience with Overpass, I know bootstrapping is as simple as running a single shell script.

Let’s assume the database is available under /mnt/osm/db. I chose this location because I imagined this database should be contained within a Docker volume, and I like to keep mount points under /mnt/ as a matter of personal preference. In addition, let’s assume logs should be written to /mnt/log.

Note: The Overpass build process generates a reboot.sh script you can use to start all the overpass binaries automatically. I chose not to use this script, as I wanted to have a more ‘hands-on’ knowledge of how the server runs. It’s possible this script might run as-is without many major modifications, however I haven’t tried it.

To run a full self-updating Overpass server, we need to run multiple OSM binaries in addition to the Apache httpd process. For this simple static Overpass server, we only need to start a single Overpass binary + Apache. Either way, we should make a shell script inside the image to hold the logic for starting everything, then we can invoke that script in the config.Cmd attribute.

But how do we make a shell script in the container? Initially, I used the attribute extraCommands passed to pkgs.dockerTools.buildLayeredImage to run some shell commands in the docker container to create a shell script that did what I needed. However, that’s excessively complicated. It’s hard to keep track of where and when everything is executing, and it’s hard to imagine what the final state of things looks like.

Instead, we can have Nix create a script file for us with pkgs.writeTextFile. We can make that file executable, then we can include that script in the image just by referencing it in the config.Cmd.

Again an abbreviated Nix script to illustrate the point:

let
  startupScript = pkgs.writeTextFile {
    name = "start_server.sh";
    executable = true;
    text = ''
    echo "Starting OSM Dispatcher..."
    rm /mnt/osm/db/osm3s_v* || true
    ${osm3s}/bin/dispatcher --osm-base --db-dir=/mnt/osm/db 1>/mnt/log/dispatcher.log 2>&1 &

    echo "Starting httpd..."
    /usr/local/bin/httpd-foreground
    '';
  };
in
pkgs.dockerTools.buildLayeredImage {
  config.Cmd = ["${pkgs.bash}/bin/bash" "-c" startupScript];
}

And that’s about it for OSM startup. Now let’s come back around to the idea of starting a new server.

Database Bootstrapping

The Overpass Wiki provides a script you can run to clone the database files from a server: bin/download_clone.sh --db-dir=/mnt/osm/db --source=http://dev.overpass-api.de/api_drolbr/ --meta=no.

If we add osm to the contents = [ of the nix script, then the mentioned script download_clone.sh will exist in the docker image under the root bin folder, under /bin/download_clone.sh.

Let’s run that script in our docker image:

$ docker run --rm -v /mnt/ext/osm_db/:/mnt/osm osm-3s-static:latest /bin/download_clone.sh --db-dir=/mnt/osm/db/ --source=...
/bin/download_clone.sh: line 82: wget: command not found
/bin/download_clone.sh: line 82: wget: command not found

Oh, well that’s simple enough to fix. Just add pkgs.wget to the contents as well and we’re up and running!

Well, except for the Apache configuration.

Configuring Apache Modules

The Overpass Wiki mentions some setup for Apache. Let’s go one by one and see if we can’t replicate the steps.

First up, we need to enable some modules:

sudo a2enmod cgi ext_filter

a2enmod doesn’t seem to exist on this docker image. But after much googling, I found out you can enable these modules by changing the config file /usr/local/apache2/conf/httpd.conf. I created a temporary container and ran docker cp to copy that file out of the container and onto my dev machine to modify it locally. To enable the cgi module, you need to uncomment the line in the httpd.conf that looks like this:

<IfModule !mpm_prefork_module>
	#LoadModule cgid_module modules/mod_cgid.so
</IfModule>

The ext_filter module looks like this:

#LoadModule ext_filter_module modules/mod_ext_filter.so

So I uncommented those two lines and saved it under a similar directory tree relative to my nix script. Alongside the docker Nix script I created a image_root folder, which emulates the root of the docker container. I can then tell Nix to copy that folder to the root of the docker container in the build script.

In other words, in my docker folder, I have this directory tree:

- osm_static.nix
- image_root/
  - usr/
    - local/
      - apache2/
        - conf/
          - httpd.conf

And in my osm_static.nix script, I have this (abbreviated)

pkgs.dockerTools.buildLayeredImage {
  contents = [
    osm3s
    pkgs.wget
    ./image_root
  ]
}

And it works as expected, Nix creates a new layer specifically for these files, and it gets included in the Docker image.

One final step from the Wiki and we’ve got ourselves an Overpass server!

Configuring Apache Virtual Host

The Overpass Wiki provides a virtual host file:

<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	ExtFilterDefine gzip mode=output cmd=/bin/gzip
	DocumentRoot [YOUR_HTML_ROOT_DIR]

	# This directive indicates that whenever someone types http://www.mydomain.com/api/ 
	# Apache2 should refer to what is in the local directory [YOUR_EXEC_DIR]/cgi-bin/
	ScriptAlias /api/ [YOUR_EXEC_DIR]/cgi-bin/


	# This specifies some directives specific to the directory: [YOUR_EXEC_DIR]/cgi-bin/
	<Directory "[YOUR_EXEC_DIR]/cgi-bin/">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                # For Apache 2.2:
                #  Order allow,deny
                # For Apache >= 2.4:  
                Require all granted
                #SetOutputFilter gzip
                #Header set Content-Encoding gzip
	</Directory>

	ErrorLog /var/log/apache2/error.log

	# Possible values include: debug, info, notice, warn, error, crit, alert, emerg
	LogLevel warn

	CustomLog /var/log/apache2/access.log combined

</VirtualHost>

For our usage, [YOUR_EXEC_DIR] can be removed completely, as the overpass binaries get installed into the root /bin and /cgi-bin folders. And [YOUR_HTML_ROOT_DIR] can be just the httpd docker image default: "/usr/local/apache2/htdocs" (or whatever you like, as Overpass doesn’t use it).

I also changed the ErrorLog and CustomLog to point to /mnt/log instead.

Save that under the path image_root/usr/local/apache2/conf/extra/httpd-vhosts.conf. Next, we need to change the httpd.conf to read the vhosts file we made. Uncomment the line:

# Virtual hosts
#Include conf/extra/httpd-vhosts.conf

Save, build and deploy. And that’s about it, a fully functional Overpass server! I’m sure more than a few things could be improved about the scripts, but I’m pretty proud that I was able to get to this point.