FreeBSD: Taming Periodic

When I started using FreeBSD, it didn't take long for mails with subject daily run output started to fill my inbox.

While I like the idea of a system maintenance / status report, I think it doesn't make sense to flood an admin with long emails indicating that everything is just fine.

The admin will first skim and later auto-delete the mail and hence not recognize when things actually go wrong (Alarm fatigue).

I use the following settings in /etc/periodic.conf to mute success-only periodic reports:




# Tame security reports

A word about about security_status_passwdless_enable: While this may be a useful check, this script doesn't report the diffs but the current list of passwordless accounts – I couldn't calm it down. Depending on your security requirements, you might want to keep this report enabled.

SpamAssassin: Debugging, SPF and DKIM

A little post-mortem of last night's admin / debug session.

More Logs

SpamAssassin has a humongous amount of debug output – which is disabled by default and to the best of my knowledge is not sent to syslog.

However, by starting spamd without daemonization, you can specify so called debug areas via the -D <comma,separated,areas> flag.

The other options used to start SpamAssassin need to be retrieved from your init script or /etc/defaults/spamassassin.

The command to be run in Terminal:

spamd <options from init script> -D debug,areas,...

Debug areas are documented in the SpamAssassin wiki.
Note: Debug areas are the prefix of of the strings passed to the dbg(<Perl string>) function in the SpamAssassin sources.

SPF Validation

SpamAssassin can perform SPF validation.
After stepping through the (terrible) source code, I learned that there are two modes:

  • Check SPF headers added by a preceding MTA.
  • Do the SPF validation on its own.

Both are used to increase the spam score.

If SpamAssassin is integrated as a milter and you want to validate SPF, it is mandatory that SpamAssassin knows about the SMTP converstation its MTA had. Otherwise, you get the following log errors:

$ spamd <...> -D spf
dbg: spf: relayed through one or more trusted relays, cannot use header-based Envelope-From, skipping
dbg: spf: spf_whitelist_from: could not find useable envelope sender
dbg: spf: checking to see if the message has a Received-SPF header that we can use
dbg: spf: no suitable relay for spf use found, skipping SPF check
dbg: spf: already checked for Received-SPF headers, proceeding with DNS based checks
dbg: spf: no suitable relay for spf use found, skipping SPF-helo check
dbg: spf: def_spf_whitelist_from: already checked spf and didn't get pass, skipping whitelist check

To fix this error in Postfix, add the Underscore _ to the macros in

milter_connect_macros = j {daemon_name} v {if_name} _

In your /etc/spamassassin/local.pre, add

loadplugin Mail::SpamAssassin::Plugin::SPF

Now restart spamd et voilà, broken SPF now adds to the Spam score.

Note: In a setup with spamass-milter, you can use the Shortcircuit module to reject mail with high spam scores.

DKIM Validation

To validate DKIM signatures and make broken signatures add to the Spam score, you need to install the Perl module: Mail::DKIM::Verifier. Afterwards, restart spamd and DKIM signatures are validated.

Poudriere in a Jail

At the time of writing, this blog along with many other applications is hosted on a relatively small VM. Hence, the old-school approach of building every port yourself is tiresome and the omnipresence of Perl accompanied by tons of modules makes it a pain every time I fire up a new jail and need to install basic software.

The cool kids today use binary packages, which are of course available on FreeBSD. But sometimes, you still want to compile yourself, mostly when a maintainer has chosen a configuration that doesn’t fit your needs. Too bad mixing self-compiled ports and packages can lead to problems when upgrading, so let’s get things right.

Poudriere to the Rescue

Poudriere is the build system used for the official FreeBSD binary repositories. What I want on my machine is the following:

  • Always use FreeBSD binary packages on the host system.
  • Run poudriere in a jail, serving compiled packages via HTTP (because it’s easy) on a local interface
    • I’ll only build packages that actually need modification. Everything else will come from the offical repos.
  • Have all the other jails use both FreeBSD binary packages and the poudriere-jail packages with higher priority for the latter repository.

This enables me to run pkg upgrade in a tmux synchronize-panes session which is just how it should be.
Of course, you can use this guide for more complex setups, too.

Setup the Jail and Permissions

I use ezjail on my system, but the setup should work similarly on a regular setup. Plus: you get the advantage of using the new jail.conf configuration format.

Create an IP address for the poudriere-jail. Remember to persist this in rc.conf.

ifconfig em0 inet alias

Create the jail:

ezjail-admin create poudriere

Regardless of whether your ezjail-installation assigns a ZFS dataset to every jail or not: You need want a dedicated ZFS dataset for the poudriere-jail. Letting a jail manage a dataset is represented through the jailed property of a dataset.

zfs create -o jailed=on zroot/poudriere

Poudriere does some fancy stuff to speed things up, e.g. building on a RAM disk and creating nested jails for each platform you want to build on. Hence, that jail needs a lot of privileges.
After an hour of trying, I got things working with the following configuration for the jail.

Configure the Jail

Make the following adjustments in /usr/local/etc/ezjail/poudriere.

Assign the loopback-interface to the jail. According to this mailing list post, this is necessary to let poudriere create the nested build-jails.

export jail_poudriere_ip=","

Allow access to the dataset we created for poudriere.

export jail_poudriere_zfs_datasets="zroot/poudriere"

Allow creating up to ten child jails, mounting special filesystems and the ZFS build dataset.

export jail_poudriere_parameters="children.max=10 \
allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs \
allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Load kernel modules needed by poudriere and make sure they are loaded after reboot.

kldload tmpfs linux linprocfs nullfs procfs fdescfs
# persist these changes
echo 'kld_list="tmpfs linux linprocfs nullfs procfs fdescfsu"' >> /etc/rc.conf

Start the jail and make yourself comofortable there.

Setting up Poudriere

From now on, we operate in the poudriere-jail.

Install poudriere:

pkg install poudriere

Configure poudriere’s /usr/local/etc/poudriere.conf.
If you have enough resources, the default settings should suffice.
On my machine, I disabled TMPFS because I have limited RAM.
The config is well-commented and should get you through.

One thing: you want to sign the packages, even though at the moment, you have everything on the same machine. But this might change one day.
Create a public-private key pair…

mkdir -p /usr/local/etc/pki/poudriere
sudo openssl genrsa -out /usr/local/etc/pki/poudriere/poudriere.key 4096
sudo openssl rsa -in /usr/local/etc/pki/poudriere/poudriere.key -pubout -out /usr/local/etc/pki/poudriere/poudriere.crt

… and update the configuration file:

# /usr/local/etc/poudriere.conf

Poudriere Web Status Page


Since you’ll most likely serve the packages via HTTP, you will need an HTTP server on the machine anyways. So why not give something to look at while your machine is sweating?

You can set-up a read-only status page of the poudriere build-process using nginx.

pkg install nginx

Relevant part of /usr/local/etc/nginx/nginx.conf:

server {

        listen       80;
        server_name  localhost;

        # Point to the web-fronted
        location / {
            root /usr/local/share/poudriere/html/;
            index  index.html index.htm;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;

        # This location is used by the web-interface
        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;

        # Use this as the base URL to serve packages via http
        location /packages {
            root /usr/local/poudriere/data/;
            index index.html;


Note: You probably still want to protect access to this web-server if you run it on the public internet. With an RFC1918 address on the jail interface, this is not the case.

Additional resources:

SSH Into a FreeBSD Live System

One of the worst nightmares: you screwed up and your system won’t boot anymore. One approach to fix things is booting from a Live-CD. You probably don’t want to repair you system using a VNC console or similar.

Luckily, the FreeBSD Live-CD comes with an OpenSSH server. (And with dhclient which I needed in my special case.). So let’s do the repair from the comforting warmth of your local machine’s terminal emulator.

  • Boot from the Live-CD
  • Choose “Shell” on the first screen
  • Configure your network interface
ifconfig em0 inet netmask ... # if you are using static addresses
-- OR --
dhclient em0 # if you are using DHCP
  • “Make /etc writable” and configure sshd and start the service.
    (This is the first time I knowingly used UnionFS but the approach sounds reasonable.)
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
vi /etc/ssh/sshd_config # allow root login
passwd root # Set a root password
service sshd onestart
  • You can now ssh into the live-system from your local machine and fix things.
    (Please check out my separate post about fixing ZFS from a Live-CD system.)

Note: The host keys are generated right when you run service sshd onestart. I am not sure how much entropy exists at this point.


From the reddit thread:

The short answer is “enough”. The long answer is that the random subsystem blocks until it has enough entropy to initialize. You can check dmesg for the line random: unblocking device if you want to make sure it is unblocked. Usually it unblocks somewhere between the tail-end of device probing and beginning of filesystem mounts.

Microsoft Sculpt Ergonomic Keyboard on OS X

A few weeks ago, my Apple Wireless Keyboard’s K-key broke and I was looking for a replacement. As a good Apple customer, I was already about to order a new one when I remembered all the positive reviews of certain ergonomic keyboards – in particular the Microsoft Sculpt.

So I ordered one from Amazon: there was none with US-Layout offered in the German store so I went with the UK-one instead – not realizing the subtle differences between the layouts.

Anyways, here are my impressions one week after the switch from the Apple-Keyboard to the Sculpt.

First contact

Man, that thing is bulky. Not as bulky as the normal keyboards with a big useless fat num-block to the right but still, the device takes a lot more of (visual) space than the humble Apple products. However, you get used to it pretty fast and since you can easily grab your favorite pointing device without waving your arm, the most annoying issue with Windows keyboards is already resolved.

When switching, you’ll most certainly hit many wrong keys in the beginning – as you’d do with any new keyboard.

If you think you are pretty OK typing blind on the Apple keyboard, you’ll be frustrated to feel your index finger fall into the space caused by the split layout when attempting to type b.

If you are used to not having the Delete Forward key on the small Apple notebook / wireless keyboards and use the fn key for this, you’ll be annoyed the Sculpt does not have such one.

If you are a heavy user of Vim, the Escape key will most certainly be a no-go for you.

And if you use the arrow keys heavily, you’ll notice how nice it is not having to move your wrist to jump a few lines up and and down on the Apple keyboard.


All this is fairly annoying and I wouldn’t use the Sculpt if it were not for the following modifications.

I found an open source tool callee Karabiner that comes in handy when dealing with the weak points of the Sculpt. While the interface is crappy, it is a very powerful tool that holds tons of options to customize the behavior of your keyboard. If you are not satisfied with shitload of tweaks already shipping with it, you can add your own using XML-formatted configuration files.

Please note that the following customizations have been made using the U.S. With Umlauts via Option Key keyboard layout that facilitates typing German text while using a US layout for programming, etc. You can download it here.

Since using the arrow keys on the Sculpt requires lifting your right wrist from the cushion, you might want Vim-style navigation sometimes. In Karabiner, there is an option for that: hold s + hjkl to use the corresponding arrow keys and you will have to lift your hands far less. However, this comes with a little brainfuck since you are virtually changing editing modes far less explicitly than in Vim (when using the Escape key).

Other mapping are:

  • Alt-Gr to CMD_R
  • Application-key to OPTION_L (Didn’t find a checkbox to use OPTION_R which should not make a difference anyway)
  • Ctrl-Backspace as a poor man’s Delete Forward (I don’t want to accustom to having a dedicated key for that only on the Sculpt)
  • Backquote to Escape and Section to Backquote. Sounds crazy but it’s fine since the British keyboard gives you an extra key you wouldn’t have on the US-Version anyways. Using these options gives you an easily accessible Escape key that you can punch on when stuck in some weird mode in Vim.

The result looks like this:


Final words

Putting it all together, I am not really happy with the steps you have to go through to get an OK experience with the Sculpt on the Mac. However, typing feels like walking on clouds which you’ll appreciate when returning to you MacBook’s internal keyboard.

Karabiner uses standard Accessibility APIs which I don’t expect Apple to break soon. It is a good sign that the software is already compatible with Yosemite which is not released yet.

It took me about two days and several hours of practice in TIPP10 but I feel comfortable now. Temporarily switching back to the Apple keyboard is not much of a problem either since you’ll most cetainly be better at the 10-finger-system than ever before.

I consider the Sculpt a net-win for me although it comes with some major drawbacks that were clearly demotivating in the first days of use.

iTunes Backup Lock

I am just installing the iOS 8 and wanted my old iOS 7 backup to be preserved in case I need to go back.

While at first, I thought about copying the backup directory located in ~/Library/Application Support/MobileSync/Backup/<UIID> to be sure iTunes doesn’t overwrite it, I discovered that this bloated piece of software actually has something like this built-in.

Just Ctrl-click on a backup in preferences and choose Archive to lock your backup – indicated by the little icon.


Running ZNC on FreeBSD

ZNC is an IRC bouncer. When installing the software on FreeBSD using pkg install znc, you’re not able to just run the service znc (one)start – it’ll throw an error like Could not open configuration file.

Let’s check the rc.d script installed at /usr/local/etc/rc.d/znc

# znc_enable:       Set to NO by default. Set it to YES to enable it.
# znc_conf_dir:     Directory where znc configuration
#                   data is stored.
#                   Default: /usr/local/etc/znc
# znc_user:         The user account znc runs as what
#                   you want it to be. It uses 'znc' user by
#                   default. Do not sets it as empty or it will run
#                   as root.

. /etc/rc.subr



load_rc_config ${name}

: ${znc_enable:="NO"}
: ${znc_user:="znc"}
: ${znc_conf_dir="/usr/local/etc/znc"}

What do we learn from this? The script runs znc as user znc by default using the configuration directory in /usr/local/etc/znc.

Setup user and permissions

At the time of writing, neither the user nor the directory are created by the package. To fix this, follow the percedure below:

  1. Run adduser -D (-D for disabling creation of the home directory) and create a user znc. Don’t add the user to any special groups or the like.
  2. Run mkdir /usr/local/etc/znc
  3. Run chown znc:znc /usr/local/etc/znc

The user znc now has access to the configuration directory specified in the packages rc-script.

Create znc configuration

Now it’s time to create the znc configuration file.

Create the directory for znc configs using sudo -u znc mkdir /usr/local/etc/znc/configs

Run sudo -u znc znc --makeconf, follow the instructions and specify /usr/local/etc/znc/configs/znc.conf as destination for the config file when you’re asked for this.
Check the znc wiki for more information !

Custom SSL certificate

ZNC supports SSL and has a webinterface-module for easier configuration. (Hence it needs write-permissions to its configuration directory).

You probably want to use your own certificate. tl;dr: You need to put all certificate data (i.e. private key, cert and CA certificates) into one file at /usr/local/etc/znc/znc.pem.

More information here.

Recommendation: Soulver

Have you ever felt the need to do some calculations using variables, mathematical functions, unit conversions, etc? Have you ever felt the need to punch into the face?

Soulver might be worth a try. It’s a mix of spreadsheets and writing pseudo-code.

I’ve been using it for several years and it’s what suits my thought-process best.

TinyTinyRSS: Migration hickup

I use TinyTiny RSS for keeping up-to-date with my ~200 RSS feeds. Together with the Fever API comptability plugin, this has been a sweet solution for me since Google Reader closed its doors.

However, yesterday, I was migrating the MySQL server that is accessed by TTRSS to a separate jail. At first look, everything seemed to work after changing config.php:

define('DB_HOST', 'localhost');
define('DB_HOST', '<The MySQL-Server>');

However, this morning I noticed that the feeds had not been updated. update.php is usually run by a cronjob, but when executing it manually, you saw error messages in certain feeds. In my case, it’s the famous iOS Dev Weekly feed.

 Fatal error: Query INSERT INTO ttrss_error_log
(errno, errstr, filename, lineno, context, owner_uid, created_at) VALUES
(256, 'Query INSERT INTO ttrss_entries\n (title,\n guid,\n
link,\n updated,\n content,\n content
_hash,\n no_orig_date,\n date_updated,\n date_entered,\n
comments,\n num_comments,\n plugin_data,\n lang,\n
author)\n VALUES\n (\'Issue 153 - 4th July 2014\',\n \'SHA1:c4acc
25a37b992b33491f0d8c5b49e2008e470a3\',\n \'\',
[Post content]
in /path/to/ttrss/classes/db/mysqli.php line 33

Quite cryptic. Looking at the database, I learned that TTRSS is storing the error logs themselves in the database. Does not make any sense for storing databse-related errors, but ok, it worked so far…. As the errors stored there were fairly up-to-date, I decided it was not an issue with access permissions.

Fiddling around a little more, I noticed the following configuration option for config.php:

define('MYSQL_CHARSET', 'utf8');
// Connection charset for MySQL. If you have a legacy database and/or experience
// garbage unicode characters with this option, try setting it to a blank string.

I fixed my issue by setting MYSQL_CHARSET  to a blank string first and afterwards to utf8 again. If you don’t reenable it, the web interface does not render certain feeds.

I suppose this was an encoding issue although I cannot explain why: the database is setup to use UTF-8 and the feeds contain only standard characters (would fit into ASCII).

To the poor guy who is searching the web for a solution to this apparently rather exotic problem for two hours: you’re welcome.


After all, I think I have figured out the encoding issue: The fix above stopped working after a few hours – I guess because some new item in a feed caused an encoding error. My current solution is to force utf8mb4 for communication betweeen mysql client and server.

#server's my.cnf
#ttrss config.php
define('MYSQL_CHARSET', 'utf8');