Poudriere in a Jail

At the time of writing, this blog along with many other applications is hosted on a relatively small VM. Hence, the old-school approach of building every port yourself is tiresome and the omnipresence of Perl accompanied by tons of modules makes it a pain every time I fire up a new jail and need to install basic software.

The cool kids today use binary packages, which are of course available on FreeBSD. But sometimes, you still want to compile yourself, mostly when a maintainer has chosen a configuration that doesn’t fit your needs. Too bad mixing self-compiled ports and packages can lead to problems when upgrading, so let’s get things right.

Poudriere to the Rescue

Poudriere is the build system used for the official FreeBSD binary repositories. What I want on my machine is the following:

  • Always use FreeBSD binary packages on the host system.
  • Run poudriere in a jail, serving compiled packages via HTTP (because it’s easy) on a local interface
    • I’ll only build packages that actually need modification. Everything else will come from the offical repos.
  • Have all the other jails use both FreeBSD binary packages and the poudriere-jail packages with higher priority for the latter repository.

This enables me to run pkg upgrade in a tmux synchronize-panes session which is just how it should be.
Of course, you can use this guide for more complex setups, too.

Setup the Jail and Permissions

I use ezjail on my system, but the setup should work similarly on a regular setup. Plus: you get the advantage of using the new jail.conf configuration format.

Create an IP address for the poudriere-jail. Remember to persist this in rc.conf.

ifconfig em0 inet 10.0.0.2 alias

Create the jail:

ezjail-admin create poudriere 10.0.0.2

Regardless of whether your ezjail-installation assigns a ZFS dataset to every jail or not: You need want a dedicated ZFS dataset for the poudriere-jail. Letting a jail manage a dataset is represented through the jailed property of a dataset.

zfs create -o jailed=on zroot/poudriere

Poudriere does some fancy stuff to speed things up, e.g. building on a RAM disk and creating nested jails for each platform you want to build on. Hence, that jail needs a lot of privileges.
After an hour of trying, I got things working with the following configuration for the jail.

Configure the Jail

Make the following adjustments in /usr/local/etc/ezjail/poudriere.

Assign the loopback-interface to the jail. According to this mailing list post, this is necessary to let poudriere create the nested build-jails.

export jail_poudriere_ip="10.0.0.2,127.0.0.1"

Allow access to the dataset we created for poudriere.

export jail_poudriere_zfs_datasets="zroot/poudriere"

Allow creating up to ten child jails, mounting special filesystems and the ZFS build dataset.

export jail_poudriere_parameters="children.max=10 \
allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs \
allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Load kernel modules needed by poudriere and make sure they are loaded after reboot.

kldload tmpfs linux linprocfs nullfs procfs fdescfs
# persist these changes
echo 'kld_list="tmpfs linux linprocfs nullfs procfs fdescfsu"' >> /etc/rc.conf

Start the jail and make yourself comofortable there.

Setting up Poudriere

From now on, we operate in the poudriere-jail.

Install poudriere:

pkg install poudriere

Configure poudriere’s /usr/local/etc/poudriere.conf.
If you have enough resources, the default settings should suffice.
On my machine, I disabled TMPFS because I have limited RAM.
The config is well-commented and should get you through.

One thing: you want to sign the packages, even though at the moment, you have everything on the same machine. But this might change one day.
Create a public-private key pair…

mkdir -p /usr/local/etc/pki/poudriere
sudo openssl genrsa -out /usr/local/etc/pki/poudriere/poudriere.key 4096
sudo openssl rsa -in /usr/local/etc/pki/poudriere/poudriere.key -pubout -out /usr/local/etc/pki/poudriere/poudriere.crt

… and update the configuration file:

# /usr/local/etc/poudriere.conf
PKG_REPO_SIGNING_KEY=/usr/local/etc/pki/poudriere/poudriere.key

Poudriere Web Status Page

Poudriere_Jail__listing

Since you’ll most likely serve the packages via HTTP, you will need an HTTP server on the machine anyways. So why not give something to look at while your machine is sweating?

You can set-up a read-only status page of the poudriere build-process using nginx.

pkg install nginx

Relevant part of /usr/local/etc/nginx/nginx.conf:

server {

        listen       80;
        server_name  localhost;

        # Point to the web-fronted
        location / {
            root /usr/local/share/poudriere/html/;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }


        # This location is used by the web-interface
        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;
        }

        # Use this as the base URL to serve packages via http
        location /packages {
            root /usr/local/poudriere/data/;
            index index.html;
        }

    }

Note: You probably still want to protect access to this web-server if you run it on the public internet. With an RFC1918 address on the jail interface, this is not the case.

Additional resources:

Flattr this!