FreeBSD: Taming Periodic

When I started using FreeBSD, it didn't take long for mails with subject daily run output started to fill my inbox.

While I like the idea of a system maintenance / status report, I think it doesn't make sense to flood an admin with long emails indicating that everything is just fine.

The admin will first skim and later auto-delete the mail and hence not recognize when things actually go wrong (Alarm fatigue).

I use the following settings in /etc/periodic.conf to mute success-only periodic reports:




# Tame security reports

A word about about security_status_passwdless_enable: While this may be a useful check, this script doesn't report the diffs but the current list of passwordless accounts – I couldn't calm it down. Depending on your security requirements, you might want to keep this report enabled.

SpamAssassin: Debugging, SPF and DKIM

A little post-mortem of last night's admin / debug session.

More Logs

SpamAssassin has a humongous amount of debug output – which is disabled by default and to the best of my knowledge is not sent to syslog.

However, by starting spamd without daemonization, you can specify so called debug areas via the -D <comma,separated,areas> flag.

The other options used to start SpamAssassin need to be retrieved from your init script or /etc/defaults/spamassassin.

The command to be run in Terminal:

spamd <options from init script> -D debug,areas,...

Debug areas are documented in the SpamAssassin wiki.
Note: Debug areas are the prefix of of the strings passed to the dbg(<Perl string>) function in the SpamAssassin sources.

SPF Validation

SpamAssassin can perform SPF validation.
After stepping through the (terrible) source code, I learned that there are two modes:

  • Check SPF headers added by a preceding MTA.
  • Do the SPF validation on its own.

Both are used to increase the spam score.

If SpamAssassin is integrated as a milter and you want to validate SPF, it is mandatory that SpamAssassin knows about the SMTP converstation its MTA had. Otherwise, you get the following log errors:

$ spamd <...> -D spf
dbg: spf: relayed through one or more trusted relays, cannot use header-based Envelope-From, skipping
dbg: spf: spf_whitelist_from: could not find useable envelope sender
dbg: spf: checking to see if the message has a Received-SPF header that we can use
dbg: spf: no suitable relay for spf use found, skipping SPF check
dbg: spf: already checked for Received-SPF headers, proceeding with DNS based checks
dbg: spf: no suitable relay for spf use found, skipping SPF-helo check
dbg: spf: def_spf_whitelist_from: already checked spf and didn't get pass, skipping whitelist check

To fix this error in Postfix, add the Underscore _ to the macros in

milter_connect_macros = j {daemon_name} v {if_name} _

In your /etc/spamassassin/local.pre, add

loadplugin Mail::SpamAssassin::Plugin::SPF

Now restart spamd et voilà, broken SPF now adds to the Spam score.

Note: In a setup with spamass-milter, you can use the Shortcircuit module to reject mail with high spam scores.

DKIM Validation

To validate DKIM signatures and make broken signatures add to the Spam score, you need to install the Perl module: Mail::DKIM::Verifier. Afterwards, restart spamd and DKIM signatures are validated.

Poudriere in a Jail

At the time of writing, this blog along with many other applications is hosted on a relatively small VM. Hence, the old-school approach of building every port yourself is tiresome and the omnipresence of Perl accompanied by tons of modules makes it a pain every time I fire up a new jail and need to install basic software.

The cool kids today use binary packages, which are of course available on FreeBSD. But sometimes, you still want to compile yourself, mostly when a maintainer has chosen a configuration that doesn’t fit your needs. Too bad mixing self-compiled ports and packages can lead to problems when upgrading, so let’s get things right.

Poudriere to the Rescue

Poudriere is the build system used for the official FreeBSD binary repositories. What I want on my machine is the following:

  • Always use FreeBSD binary packages on the host system.
  • Run poudriere in a jail, serving compiled packages via HTTP (because it’s easy) on a local interface
    • I’ll only build packages that actually need modification. Everything else will come from the offical repos.
  • Have all the other jails use both FreeBSD binary packages and the poudriere-jail packages with higher priority for the latter repository.

This enables me to run pkg upgrade in a tmux synchronize-panes session which is just how it should be.
Of course, you can use this guide for more complex setups, too.

Setup the Jail and Permissions

I use ezjail on my system, but the setup should work similarly on a regular setup. Plus: you get the advantage of using the new jail.conf configuration format.

Create an IP address for the poudriere-jail. Remember to persist this in rc.conf.

ifconfig em0 inet alias

Create the jail:

ezjail-admin create poudriere

Regardless of whether your ezjail-installation assigns a ZFS dataset to every jail or not: You need want a dedicated ZFS dataset for the poudriere-jail. Letting a jail manage a dataset is represented through the jailed property of a dataset.

zfs create -o jailed=on zroot/poudriere

Poudriere does some fancy stuff to speed things up, e.g. building on a RAM disk and creating nested jails for each platform you want to build on. Hence, that jail needs a lot of privileges.
After an hour of trying, I got things working with the following configuration for the jail.

Configure the Jail

Make the following adjustments in /usr/local/etc/ezjail/poudriere.

Assign the loopback-interface to the jail. According to this mailing list post, this is necessary to let poudriere create the nested build-jails.

export jail_poudriere_ip=","

Allow access to the dataset we created for poudriere.

export jail_poudriere_zfs_datasets="zroot/poudriere"

Allow creating up to ten child jails, mounting special filesystems and the ZFS build dataset.

export jail_poudriere_parameters="children.max=10 \
allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs \
allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Load kernel modules needed by poudriere and make sure they are loaded after reboot.

kldload tmpfs linux linprocfs nullfs procfs fdescfs
# persist these changes
echo 'kld_list="tmpfs linux linprocfs nullfs procfs fdescfsu"' >> /etc/rc.conf

Start the jail and make yourself comofortable there.

Setting up Poudriere

From now on, we operate in the poudriere-jail.

Install poudriere:

pkg install poudriere

Configure poudriere’s /usr/local/etc/poudriere.conf.
If you have enough resources, the default settings should suffice.
On my machine, I disabled TMPFS because I have limited RAM.
The config is well-commented and should get you through.

One thing: you want to sign the packages, even though at the moment, you have everything on the same machine. But this might change one day.
Create a public-private key pair…

mkdir -p /usr/local/etc/pki/poudriere
sudo openssl genrsa -out /usr/local/etc/pki/poudriere/poudriere.key 4096
sudo openssl rsa -in /usr/local/etc/pki/poudriere/poudriere.key -pubout -out /usr/local/etc/pki/poudriere/poudriere.crt

… and update the configuration file:

# /usr/local/etc/poudriere.conf

Poudriere Web Status Page


Since you’ll most likely serve the packages via HTTP, you will need an HTTP server on the machine anyways. So why not give something to look at while your machine is sweating?

You can set-up a read-only status page of the poudriere build-process using nginx.

pkg install nginx

Relevant part of /usr/local/etc/nginx/nginx.conf:

server {

        listen       80;
        server_name  localhost;

        # Point to the web-fronted
        location / {
            root /usr/local/share/poudriere/html/;
            index  index.html index.htm;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;

        # This location is used by the web-interface
        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;

        # Use this as the base URL to serve packages via http
        location /packages {
            root /usr/local/poudriere/data/;
            index index.html;


Note: You probably still want to protect access to this web-server if you run it on the public internet. With an RFC1918 address on the jail interface, this is not the case.

Additional resources:

Angela Merkels Sprache arbeitet mit einer permanenten Immunisierung gegen jede Berührung, jeden Zweifel, sie agitiert gegen alles Offene, alles Unfertige – und damit gegen das, was eine freie Gesellschaft und eine deliberative Demokratie gerade ausmacht. Was hat sie gesagt? (1.2.2015, 15:40)

Fun With Apple-ID Renaming Continues

I had a curious iOS support incident I want to share. Two weeks ago, a friend asked me for help with her iPhone.

The device was given to her by a relative who had apparently reset it by restoring a backup he made almost three years ago.

The backup contained a log-in to the relative’s iCloud account with Find my iPhone turned on. To put things in perspective: Find my iPhone was first launched in June 2010 and integrated into iCloud on its launch in October 2011.

Nevertheless, Activation Lock, the feature that prevents re-activation of an iPhone as long as it is marked locked by its owner, was not a thing until iOS 7. But – purposedly in the endeavor to make the transition seamless – Apple kept the label Find my iPhone for the switch in which now enables / disables Find my iPhone and Activation Lock simulatenously.

And this lead to the following problem: My friend’s device could not be restored (neither by iTunes nor from the iPhone itself) because it assumed Activation Lock was still enabled.

In case you did not know: Apple offers a web-form where you can check the Activation Lock status of your device. This page reported Activation Lock to be disabled.

Looking at iCloud settings on the device, the switch was enabled. Attempts to toggle it off failed with a try again error after the relative entered his password. It was not the kind of the occassional 503 HTTP status codes you get when Apple activation servers are down.

Also, on none of the relative’s devices were listed in the Find my iPhone webinterface.

I started to go through my catalog of questions I use to identify possible user errors. And indeed, after fifteen minutes of interrogation, there was the answer:

Oh, yeah, one more thing: I renamed my Apple-ID since I did that backup I restored on the phone.

To be clear: He had not just changed his primary e-mail address but indeed altered the username of the Apple-ID.

Please don’t do this. Just don’t. Absolutely. Never. I speak from experience.

So apparently, activation lock is out of sync with the server-side.

Luckily, putting the device into DFU mode and connecting it to iTunes solved the issue for this particular case.

However, it is simply embarrassing that Apple still has not figured out their user account system. Come on guys, this is known territory.


Comment: The Functional High Ground

In case you did not notice: in the last months, torches and pitchforks have been held steady in the Apple developer community. People have been furious about the inexplicable and arbitrary rejections of high-profile and / or innovative apps, e.g. Transmit, PCalc and Nintype.

While this particular turmoil seemed to cool down in the last week(s), another topic that has been on the radar for quite some time has been in discussion: The recent decline in Apple’s software quality. Marco Arment hit the nail on the head for a lot of people, myself included, although Daniel Jalkut puts some statements in a more rational perspective.
Additional link: Craig Hockenberry has written another great piece.

Since the launch of iOS 7, the number of people asking me for help or complaining about their Apple-devices has increased significantly.
People are pissed.

This post is written on OS X Mavericks, because I still hear about problems with WiFi on Yosemite. And because the new features just are not worth the risk of the slightest instability to me.

The #1 reason I am not using FreeBSD or PCBSD on my laptop is OS X’s reliability I learned to love since I switched to the Mac around five years ago. And the software I use on a daily basis, which is in my opinion far superior to everything I have seen on other platforms. In short, OS X still is my favorite UNIX desktop.

I would not consider myself an iOS power user. Sure, I use the device quite a lot but it’s nothing all-too fancy: podcasts, OmniFocus, Twitter, navigation and occassionally messaging although I hate typing on the phone. And some annoying bugs along with the restrictions you still have on iOS make me feel tired of the platform.

I tend to agree that Apple’s hardware is amazing – my laptop delights me every day and so does my phone and my tablet.

But it is not hardware quality that convinced me to buy Apple products in the first place. And it is definitely not the attempt to build a platform with inferior Maps, windy Clouds, locked-down devices and rushed-through software releases.

I do not care about Apple doing well economically. As a company, profit will always be the highest priority for them. No matter what. However, it is important to have a long-term strategy for your products.

I want need that strategy to be: making the most reliable operating system there is with the most advanced technologies available that are suitable for the job.

Everything else, I’m out. Sooner or later.


31C3 CCH outside view
[Foto: CC-BY 2.0 Robert]

After all the years I could not go because of limited travelling and legal capabilities (minors can’t rent hotel rooms, etc.), things worked out in 2014: I went to the Chaos Communication Congress for the first time.

What a great event, so many great talks. So much Mate. So little sleep. So cool projects. So impressive people.

What stands out in the aftermath of (re)watching some of the talks.

You will find a bullet in my head before finding a back door in Tor.

These words sound pathetic, but you could clearly feel that Jacob Applebaum is dead-serious about it.Watch the talk State of the Onion.


A new dawn is breaking. And if we move on with the momentum I felt during Congress, the sun might never set.

Happy 2015, everyone!

Working on a Geli-Encrypted ZFS Pool from a Live-CD or Memstick

Imagine you are experimenting with ZFS. You might render your system unbootable, e.g. because you misconfigured a mountpoint property. Let’s fix this.

Note: In this post, I’ll refer to the following setup which is the one bsdinstall generates if you tick the encryption checkbox during installation.

For the sake of simplicity, we’ll assume that you have set-up a stripe on just one disk /dev/ada0. If you have set-up a mirror, you’ll have to decrypt the mirrored partitions of the ZFS pool as well to operate on it.

Who’s who

You can show the partitions on your drive using gpart:

gpart show ada0
### output ###
=>       34  419430333  ada0  GPT  (200G)
         34       1024     1  freebsd-boot  (512K)
       1058    4194304     2  freebsd-zfs  (2.0G)
    4195362    8388608     3  freebsd-swap  (4.0G)
   12583970  406846397     4  freebsd-zfs  (194G)

The number in the ada0 column is the partition you can find ad /dev/ada0p<number>.

  • ada0p1: bootstrapping code to boot off bootpool.
  • ada0p2: the ZFS pool bootpool. This one is not encrypted because Geli needs the kernel’s crypto framework. Hence, bootpool contains
    • The kernel with modules to mount ZFS, support geli, etc.
    • The geli decryption key for the partition containing zroot. Don’t worry, it’s protected by your passphrase through symetric encryption.
  • ada0p3: swap space – probably also encrypted if you checked that checkbox, too.
  • ada0p4: the geli-encrypted ZFS pool zroot.
    zroot contains at least one dataset with / as mountpoint (usually zroot/ROOT/default).

More detailed and accurate information can be found on the man-page section about BOOTSTRAPPING.

Decrypting the pool

Assuming you want to fix things on zroot, you’ll need to decrypt ada0p4 first. For decryption, you need the key-file. So let’s mount bootpool to /tmp/bootpool.

mkdir /tmp/bootpool
zfs import # lists the available pools to import. bootpool should be listed there
zfs import -N -f bootpool # forcefully import bootpool, but don't mount it.
zfs set mountpoint=/tmp/bootpool bootpool
zfs mount bootpool

The keyfile should be located at /tmp/bootpool/boot/encryption.key Now you are able to decrypt ada0p4:

geli attach -k /tmp/bootpool/boot/encryption.key /dev/ada0p4

Note: If you have a ZFS mirror or raidz set-up, you’ll have to decrypt the other partitions before importing the pool.

Importing the pool

zfs import -N -f zroot # import zroot without mounting it

Note: Be careful when mounting datasets of zroot. They have their mountpoints set to locations that aleady contain files on the live-system, e.g. zroot/ROOT/default‘s mountpoint is / but the livesystem is mounted at /, too. If you screw it up, reboot and start over.

Congratulations, you can now operate on your pool from the live-system!