Poudriere in a Jail

At the time of writing, this blog along with many other applications is hosted on a relatively small VM. Hence, the old-school approach of building every port yourself is tiresome and the omnipresence of Perl accompanied by tons of modules makes it a pain every time I fire up a new jail and need to install basic software.

The cool kids today use binary packages, which are of course available on FreeBSD. But sometimes, you still want to compile yourself, mostly when a maintainer has chosen a configuration that doesn’t fit your needs. Too bad mixing self-compiled ports and packages can lead to problems when upgrading, so let’s get things right.

Poudriere to the Rescue

Poudriere is the build system used for the official FreeBSD binary repositories. What I want on my machine is the following:

  • Always use FreeBSD binary packages on the host system.
  • Run poudriere in a jail, serving compiled packages via HTTP (because it’s easy) on a local interface
    • I’ll only build packages that actually need modification. Everything else will come from the offical repos.
  • Have all the other jails use both FreeBSD binary packages and the poudriere-jail packages with higher priority for the latter repository.

This enables me to run pkg upgrade in a tmux synchronize-panes session which is just how it should be.
Of course, you can use this guide for more complex setups, too.

Setup the Jail and Permissions

I use ezjail on my system, but the setup should work similarly on a regular setup. Plus: you get the advantage of using the new jail.conf configuration format.

Create an IP address for the poudriere-jail. Remember to persist this in rc.conf.

ifconfig em0 inet alias

Create the jail:

ezjail-admin create poudriere

Regardless of whether your ezjail-installation assigns a ZFS dataset to every jail or not: You need want a dedicated ZFS dataset for the poudriere-jail. Letting a jail manage a dataset is represented through the jailed property of a dataset.

zfs create -o jailed=on zroot/poudriere

Poudriere does some fancy stuff to speed things up, e.g. building on a RAM disk and creating nested jails for each platform you want to build on. Hence, that jail needs a lot of privileges.
After an hour of trying, I got things working with the following configuration for the jail.

Configure the Jail

Make the following adjustments in /usr/local/etc/ezjail/poudriere.

Assign the loopback-interface to the jail. According to this mailing list post, this is necessary to let poudriere create the nested build-jails.

export jail_poudriere_ip=","

Allow access to the dataset we created for poudriere.

export jail_poudriere_zfs_datasets="zroot/poudriere"

Allow creating up to ten child jails, mounting special filesystems and the ZFS build dataset.

export jail_poudriere_parameters="children.max=10 \
allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs \
allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Load kernel modules needed by poudriere and make sure they are loaded after reboot.

kldload tmpfs linux linprocfs nullfs procfs fdescfs
# persist these changes
echo 'kld_list="tmpfs linux linprocfs nullfs procfs fdescfsu"' >> /etc/rc.conf

Start the jail and make yourself comofortable there.

Setting up Poudriere

From now on, we operate in the poudriere-jail.

Install poudriere:

pkg install poudriere

Configure poudriere’s /usr/local/etc/poudriere.conf.
If you have enough resources, the default settings should suffice.
On my machine, I disabled TMPFS because I have limited RAM.
The config is well-commented and should get you through.

One thing: you want to sign the packages, even though at the moment, you have everything on the same machine. But this might change one day.
Create a public-private key pair…

mkdir -p /usr/local/etc/pki/poudriere
sudo openssl genrsa -out /usr/local/etc/pki/poudriere/poudriere.key 4096
sudo openssl rsa -in /usr/local/etc/pki/poudriere/poudriere.key -pubout -out /usr/local/etc/pki/poudriere/poudriere.crt

… and update the configuration file:

# /usr/local/etc/poudriere.conf

Poudriere Web Status Page


Since you’ll most likely serve the packages via HTTP, you will need an HTTP server on the machine anyways. So why not give something to look at while your machine is sweating?

You can set-up a read-only status page of the poudriere build-process using nginx.

pkg install nginx

Relevant part of /usr/local/etc/nginx/nginx.conf:

server {

        listen       80;
        server_name  localhost;

        # Point to the web-fronted
        location / {
            root /usr/local/share/poudriere/html/;
            index  index.html index.htm;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;

        # This location is used by the web-interface
        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;

        # Use this as the base URL to serve packages via http
        location /packages {
            root /usr/local/poudriere/data/;
            index index.html;


Note: You probably still want to protect access to this web-server if you run it on the public internet. With an RFC1918 address on the jail interface, this is not the case.

Additional resources:

Working on a Geli-Encrypted ZFS Pool from a Live-CD or Memstick

Imagine you are experimenting with ZFS. You might render your system unbootable, e.g. because you misconfigured a mountpoint property. Let’s fix this.

Note: In this post, I’ll refer to the following setup which is the one bsdinstall generates if you tick the encryption checkbox during installation.

For the sake of simplicity, we’ll assume that you have set-up a stripe on just one disk /dev/ada0. If you have set-up a mirror, you’ll have to decrypt the mirrored partitions of the ZFS pool as well to operate on it.

Who’s who

You can show the partitions on your drive using gpart:

gpart show ada0
### output ###
=>       34  419430333  ada0  GPT  (200G)
         34       1024     1  freebsd-boot  (512K)
       1058    4194304     2  freebsd-zfs  (2.0G)
    4195362    8388608     3  freebsd-swap  (4.0G)
   12583970  406846397     4  freebsd-zfs  (194G)

The number in the ada0 column is the partition you can find ad /dev/ada0p<number>.

  • ada0p1: bootstrapping code to boot off bootpool.
  • ada0p2: the ZFS pool bootpool. This one is not encrypted because Geli needs the kernel’s crypto framework. Hence, bootpool contains
    • The kernel with modules to mount ZFS, support geli, etc.
    • The geli decryption key for the partition containing zroot. Don’t worry, it’s protected by your passphrase through symetric encryption.
  • ada0p3: swap space – probably also encrypted if you checked that checkbox, too.
  • ada0p4: the geli-encrypted ZFS pool zroot.
    zroot contains at least one dataset with / as mountpoint (usually zroot/ROOT/default).

More detailed and accurate information can be found on the man-page section about BOOTSTRAPPING.

Decrypting the pool

Assuming you want to fix things on zroot, you’ll need to decrypt ada0p4 first. For decryption, you need the key-file. So let’s mount bootpool to /tmp/bootpool.

mkdir /tmp/bootpool
zfs import # lists the available pools to import. bootpool should be listed there
zfs import -N -f bootpool # forcefully import bootpool, but don't mount it.
zfs set mountpoint=/tmp/bootpool bootpool
zfs mount bootpool

The keyfile should be located at /tmp/bootpool/boot/encryption.key Now you are able to decrypt ada0p4:

geli attach -k /tmp/bootpool/boot/encryption.key /dev/ada0p4

Note: If you have a ZFS mirror or raidz set-up, you’ll have to decrypt the other partitions before importing the pool.

Importing the pool

zfs import -N -f zroot # import zroot without mounting it

Note: Be careful when mounting datasets of zroot. They have their mountpoints set to locations that aleady contain files on the live-system, e.g. zroot/ROOT/default‘s mountpoint is / but the livesystem is mounted at /, too. If you screw it up, reboot and start over.

Congratulations, you can now operate on your pool from the live-system!

SSH Into a FreeBSD Live System

One of the worst nightmares: you screwed up and your system won’t boot anymore. One approach to fix things is booting from a Live-CD. You probably don’t want to repair you system using a VNC console or similar.

Luckily, the FreeBSD Live-CD comes with an OpenSSH server. (And with dhclient which I needed in my special case.). So let’s do the repair from the comforting warmth of your local machine’s terminal emulator.

  • Boot from the Live-CD
  • Choose “Shell” on the first screen
  • Configure your network interface
ifconfig em0 inet netmask ... # if you are using static addresses
-- OR --
dhclient em0 # if you are using DHCP
  • “Make /etc writable” and configure sshd and start the service.
    (This is the first time I knowingly used UnionFS but the approach sounds reasonable.)
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
vi /etc/ssh/sshd_config # allow root login
passwd root # Set a root password
service sshd onestart
  • You can now ssh into the live-system from your local machine and fix things.
    (Please check out my separate post about fixing ZFS from a Live-CD system.)

Note: The host keys are generated right when you run service sshd onestart. I am not sure how much entropy exists at this point.


From the reddit thread:

The short answer is “enough”. The long answer is that the random subsystem blocks until it has enough entropy to initialize. You can check dmesg for the line random: unblocking device if you want to make sure it is unblocked. Usually it unblocks somewhere between the tail-end of device probing and beginning of filesystem mounts.

Keeping FreeBSD up-to-date

I have to admit: my FreeBSD server had not been updated for quite a while. But I fixed that today and make the daring promise that I will keep my boxes on top of the line from now on. To internalize the commands for upating the system, I have created this little guide.

IMPORTANT: Yes, bold letters. If you don’t do it already, you should definitely run all / critical server operations in screen or preferrably tmux. This way you can let stuff run in the background and don’t have to worry about loosing network connection, etc.

Updating the system

IMPORTANT #2: Whenever you update the base system, update the jails using ezjail before reboot. This is because ezjail-admin’s update functionality relies on uname -r which would return the latest system version after reboot.

# Fetch new binary patches and apply them
freebsd-update fetch
freebsd-update install
# Update the jails using ezjail
ezjail-admin update -u

More info here.

Updating ports

What is a system without software? You’ll most certainly want to install from ports or packages and keep those up-to-date.


I keep track of new vulnerabilities affecting my system by using jailaudit. It is part of the periodic security mails I get from my server. If there is a vulnerablity, I try to fix it as soon as possible.

But I have also decided to do updates on a regular interval. I always feared this because of the famous Never change a running system but I feel confident enough to fix things in reasonable time when they break.

As a fallback, I do a zfs snapshot zroot/ezjail/thejail@pre_update as a fallback and destroy it after some time of everything running smoothly.

→ I recommend putting live data into a jail different than your web-stuff jail. I do so because the setup web-stuff jail is far more complex (nginx, php, ruby, node, perl, …) than the data jail (just a database server). When the shit hits the fan and I can’t fix stuff in web-jail fast enough, I always have the option to do zfs rollback on the web jail with the data being untouched (unless migrations happened in which case I’ll consult my regular backups).

Give me the code

I use portmaster to update the ports. and I keep a tmux pane with /usr/ports/UPDATING open all the time. It would be awesome if there was a tool that filters UDPATING down to which software will be affected by a portmaster operation.

Update the ports tree

##for the main system
portsnap fetch update
##jail ports tree
ezjail-admin update -P

Main host & per jail

#check for updates
portmaster -L
#using the pkg2ng tool (= = no update, < = needs update)
pkg version [-vl "<"]

#actual updating
portmaster  -aD --packages
portmaster --clean-distfiles
  • -a Updates all ports that require updates
  • -D Disables cleaning of distfiles at the end of each port-installation. This disables the annoying Do you want to delete xyz_0.0.1.tar.gz?.
  • --packages Uses a package if available and the configuration matches
  • --clean-distfiles Cleans the distfiles → more disk space

Important #3: Read the pkg-messages at then end of the installation and read them carefully. Often there are hints / notices that might effect your configuration, i.e. a different path to the Passenger app server, etc.

More info here.

If things don’t work

  • Google is your friend
  • → Mailinglists
  • FreeBSD bug tracking
  • → IRC: #freebsd on irc.freenode.net

Notes on ZFS incremental snapshots

On my production server, there is a cronjob that does snapshots in regular intervals (daily, weekly, monthly) and removes old ones following a certain policy.

I replicate the filesystems on my production server to a backup server using zfs send / receive.

The initial backup of a filesystem is done using zfs send -R (replication stream), so that I have an exact copy of the filesystem and snapshots on the backup server.

Following backups are done using an incremental stream:

[latest common snapshot] → [latest snapshot]

But what happens if the snapshot rotation on the server removes the latest common snapshot?

Or in other terms:

Will zfs receive apply an incremental stream if the target filesystem’s latest snapshot is not a direct ancestor of the streams oldest snapshot?

The answer:

You cannot apply an incremental stream on a filesystem whose latest snapshot is not the direct ancestor of the first snapshot in the incremental stream.

ZFS will tell you:

 cannot receive incremental stream: most recent snapshot of zroot/targetfs does not match incremental source