FreeBSD: Taming Periodic

When I started using FreeBSD, it didn't take long for mails with subject daily run output started to fill my inbox.

While I like the idea of a system maintenance / status report, I think it doesn't make sense to flood an admin with long emails indicating that everything is just fine.

The admin will first skim and later auto-delete the mail and hence not recognize when things actually go wrong (Alarm fatigue).

I use the following settings in /etc/periodic.conf to mute success-only periodic reports:

daily_show_empty_output='NO'
daily_show_success='NO'
daily_show_info='NO'

weekly_show_empty_output='NO'
weekly_show_success='NO'
weekly_show_info='NO'

monthly_show_info_show_empty_output='NO'
monthly_show_success='NO'
monthly_show_info='NO'

# Tame security reports
security_status_passwdless_enable='NO'
daily_status_security_inline='NO'
weekly_status_security_inline='NO'
monthly_status_security_inline='NO'
security_show_success='NO'

A word about about security_status_passwdless_enable: While this may be a useful check, this script doesn't report the diffs but the current list of passwordless accounts – I couldn't calm it down. Depending on your security requirements, you might want to keep this report enabled.

Flattr this!

SpamAssassin: Debugging, SPF and DKIM

A little post-mortem of last night's admin / debug session.

More Logs

SpamAssassin has a humongous amount of debug output – which is disabled by default and to the best of my knowledge is not sent to syslog.

However, by starting spamd without daemonization, you can specify so called debug areas via the -D <comma,separated,areas> flag.

The other options used to start SpamAssassin need to be retrieved from your init script or /etc/defaults/spamassassin.

The command to be run in Terminal:

spamd <options from init script> -D debug,areas,...

Debug areas are documented in the SpamAssassin wiki.
Note: Debug areas are the prefix of of the strings passed to the dbg(<Perl string>) function in the SpamAssassin sources.

SPF Validation

SpamAssassin can perform SPF validation.
After stepping through the (terrible) source code, I learned that there are two modes:

  • Check SPF headers added by a preceding MTA.
  • Do the SPF validation on its own.

Both are used to increase the spam score.

If SpamAssassin is integrated as a milter and you want to validate SPF, it is mandatory that SpamAssassin knows about the SMTP converstation its MTA had. Otherwise, you get the following log errors:

$ spamd <...> -D spf
...
dbg: spf: relayed through one or more trusted relays, cannot use header-based Envelope-From, skipping
dbg: spf: spf_whitelist_from: could not find useable envelope sender
dbg: spf: checking to see if the message has a Received-SPF header that we can use
dbg: spf: no suitable relay for spf use found, skipping SPF check
dbg: spf: already checked for Received-SPF headers, proceeding with DNS based checks
dbg: spf: no suitable relay for spf use found, skipping SPF-helo check
dbg: spf: def_spf_whitelist_from: already checked spf and didn't get pass, skipping whitelist check

To fix this error in Postfix, add the Underscore _ to the macros in main.cf:

milter_connect_macros = j {daemon_name} v {if_name} _

In your /etc/spamassassin/local.pre, add

loadplugin Mail::SpamAssassin::Plugin::SPF

Now restart spamd et voilà, broken SPF now adds to the Spam score.

Note: In a setup with spamass-milter, you can use the Shortcircuit module to reject mail with high spam scores.

DKIM Validation

To validate DKIM signatures and make broken signatures add to the Spam score, you need to install the Perl module: Mail::DKIM::Verifier. Afterwards, restart spamd and DKIM signatures are validated.

Flattr this!

Poudriere in a Jail

At the time of writing, this blog along with many other applications is hosted on a relatively small VM. Hence, the old-school approach of building every port yourself is tiresome and the omnipresence of Perl accompanied by tons of modules makes it a pain every time I fire up a new jail and need to install basic software.

The cool kids today use binary packages, which are of course available on FreeBSD. But sometimes, you still want to compile yourself, mostly when a maintainer has chosen a configuration that doesn’t fit your needs. Too bad mixing self-compiled ports and packages can lead to problems when upgrading, so let’s get things right.

Poudriere to the Rescue

Poudriere is the build system used for the official FreeBSD binary repositories. What I want on my machine is the following:

  • Always use FreeBSD binary packages on the host system.
  • Run poudriere in a jail, serving compiled packages via HTTP (because it’s easy) on a local interface
    • I’ll only build packages that actually need modification. Everything else will come from the offical repos.
  • Have all the other jails use both FreeBSD binary packages and the poudriere-jail packages with higher priority for the latter repository.

This enables me to run pkg upgrade in a tmux synchronize-panes session which is just how it should be.
Of course, you can use this guide for more complex setups, too.

Setup the Jail and Permissions

I use ezjail on my system, but the setup should work similarly on a regular setup. Plus: you get the advantage of using the new jail.conf configuration format.

Create an IP address for the poudriere-jail. Remember to persist this in rc.conf.

ifconfig em0 inet 10.0.0.2 alias

Create the jail:

ezjail-admin create poudriere 10.0.0.2

Regardless of whether your ezjail-installation assigns a ZFS dataset to every jail or not: You need want a dedicated ZFS dataset for the poudriere-jail. Letting a jail manage a dataset is represented through the jailed property of a dataset.

zfs create -o jailed=on zroot/poudriere

Poudriere does some fancy stuff to speed things up, e.g. building on a RAM disk and creating nested jails for each platform you want to build on. Hence, that jail needs a lot of privileges.
After an hour of trying, I got things working with the following configuration for the jail.

Configure the Jail

Make the following adjustments in /usr/local/etc/ezjail/poudriere.

Assign the loopback-interface to the jail. According to this mailing list post, this is necessary to let poudriere create the nested build-jails.

export jail_poudriere_ip="10.0.0.2,127.0.0.1"

Allow access to the dataset we created for poudriere.

export jail_poudriere_zfs_datasets="zroot/poudriere"

Allow creating up to ten child jails, mounting special filesystems and the ZFS build dataset.

export jail_poudriere_parameters="children.max=10 \
allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs \
allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Load kernel modules needed by poudriere and make sure they are loaded after reboot.

kldload tmpfs linux linprocfs nullfs procfs fdescfs
# persist these changes
echo 'kld_list="tmpfs linux linprocfs nullfs procfs fdescfsu"' >> /etc/rc.conf

Start the jail and make yourself comofortable there.

Setting up Poudriere

From now on, we operate in the poudriere-jail.

Install poudriere:

pkg install poudriere

Configure poudriere’s /usr/local/etc/poudriere.conf.
If you have enough resources, the default settings should suffice.
On my machine, I disabled TMPFS because I have limited RAM.
The config is well-commented and should get you through.

One thing: you want to sign the packages, even though at the moment, you have everything on the same machine. But this might change one day.
Create a public-private key pair…

mkdir -p /usr/local/etc/pki/poudriere
sudo openssl genrsa -out /usr/local/etc/pki/poudriere/poudriere.key 4096
sudo openssl rsa -in /usr/local/etc/pki/poudriere/poudriere.key -pubout -out /usr/local/etc/pki/poudriere/poudriere.crt

… and update the configuration file:

# /usr/local/etc/poudriere.conf
PKG_REPO_SIGNING_KEY=/usr/local/etc/pki/poudriere/poudriere.key

Poudriere Web Status Page

Poudriere_Jail__listing

Since you’ll most likely serve the packages via HTTP, you will need an HTTP server on the machine anyways. So why not give something to look at while your machine is sweating?

You can set-up a read-only status page of the poudriere build-process using nginx.

pkg install nginx

Relevant part of /usr/local/etc/nginx/nginx.conf:

server {

        listen       80;
        server_name  localhost;

        # Point to the web-fronted
        location / {
            root /usr/local/share/poudriere/html/;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }


        # This location is used by the web-interface
        location /data {
            alias /usr/local/poudriere/data/logs/bulk;
            autoindex on;
        }

        # Use this as the base URL to serve packages via http
        location /packages {
            root /usr/local/poudriere/data/;
            index index.html;
        }

    }

Note: You probably still want to protect access to this web-server if you run it on the public internet. With an RFC1918 address on the jail interface, this is not the case.

Additional resources:

Flattr this!

Working on a Geli-Encrypted ZFS Pool from a Live-CD or Memstick

Imagine you are experimenting with ZFS. You might render your system unbootable, e.g. because you misconfigured a mountpoint property. Let’s fix this.

Note: In this post, I’ll refer to the following setup which is the one bsdinstall generates if you tick the encryption checkbox during installation.

For the sake of simplicity, we’ll assume that you have set-up a stripe on just one disk /dev/ada0. If you have set-up a mirror, you’ll have to decrypt the mirrored partitions of the ZFS pool as well to operate on it.

Who’s who

You can show the partitions on your drive using gpart:

gpart show ada0
### output ###
=>       34  419430333  ada0  GPT  (200G)
         34       1024     1  freebsd-boot  (512K)
       1058    4194304     2  freebsd-zfs  (2.0G)
    4195362    8388608     3  freebsd-swap  (4.0G)
   12583970  406846397     4  freebsd-zfs  (194G)

The number in the ada0 column is the partition you can find ad /dev/ada0p<number>.

  • ada0p1: bootstrapping code to boot off bootpool.
  • ada0p2: the ZFS pool bootpool. This one is not encrypted because Geli needs the kernel’s crypto framework. Hence, bootpool contains
    • The kernel with modules to mount ZFS, support geli, etc.
    • The geli decryption key for the partition containing zroot. Don’t worry, it’s protected by your passphrase through symetric encryption.
  • ada0p3: swap space – probably also encrypted if you checked that checkbox, too.
  • ada0p4: the geli-encrypted ZFS pool zroot.
    zroot contains at least one dataset with / as mountpoint (usually zroot/ROOT/default).

More detailed and accurate information can be found on the man-page section about BOOTSTRAPPING.

Decrypting the pool

Assuming you want to fix things on zroot, you’ll need to decrypt ada0p4 first. For decryption, you need the key-file. So let’s mount bootpool to /tmp/bootpool.

mkdir /tmp/bootpool
zfs import # lists the available pools to import. bootpool should be listed there
zfs import -N -f bootpool # forcefully import bootpool, but don't mount it.
zfs set mountpoint=/tmp/bootpool bootpool
zfs mount bootpool

The keyfile should be located at /tmp/bootpool/boot/encryption.key Now you are able to decrypt ada0p4:

geli attach -k /tmp/bootpool/boot/encryption.key /dev/ada0p4

Note: If you have a ZFS mirror or raidz set-up, you’ll have to decrypt the other partitions before importing the pool.

Importing the pool

zfs import -N -f zroot # import zroot without mounting it

Note: Be careful when mounting datasets of zroot. They have their mountpoints set to locations that aleady contain files on the live-system, e.g. zroot/ROOT/default‘s mountpoint is / but the livesystem is mounted at /, too. If you screw it up, reboot and start over.

Congratulations, you can now operate on your pool from the live-system!

Flattr this!

SSH Into a FreeBSD Live System

One of the worst nightmares: you screwed up and your system won’t boot anymore. One approach to fix things is booting from a Live-CD. You probably don’t want to repair you system using a VNC console or similar.

Luckily, the FreeBSD Live-CD comes with an OpenSSH server. (And with dhclient which I needed in my special case.). So let’s do the repair from the comforting warmth of your local machine’s terminal emulator.

  • Boot from the Live-CD
  • Choose “Shell” on the first screen
  • Configure your network interface
ifconfig em0 inet 1.2.3.4 netmask ... # if you are using static addresses
-- OR --
dhclient em0 # if you are using DHCP
  • “Make /etc writable” and configure sshd and start the service.
    (This is the first time I knowingly used UnionFS but the approach sounds reasonable.)
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
vi /etc/ssh/sshd_config # allow root login
passwd root # Set a root password
service sshd onestart
  • You can now ssh into the live-system from your local machine and fix things.
    (Please check out my separate post about fixing ZFS from a Live-CD system.)

Note: The host keys are generated right when you run service sshd onestart. I am not sure how much entropy exists at this point.

Update

From the reddit thread:

The short answer is “enough”. The long answer is that the random subsystem blocks until it has enough entropy to initialize. You can check dmesg for the line random: unblocking device if you want to make sure it is unblocked. Usually it unblocks somewhere between the tail-end of device probing and beginning of filesystem mounts.

Flattr this!

Talk: FreeBSD – The Next 10 Years

Jordan Hubbard – FreeBSD: The Next 10 Years

I agree with all of the points he makes.

I got into FreeBSD less than one year ago and (as well as on Linux) still have the feeling that it’s hard to tell with abolute certainty what service is starting / running at which time.

An entertaining talk although the recording is a little rough.

Flattr this!

FreeBSD interim

I don’t actually know how long it has been since I run my own servers. Probably two and a half years. That’s not long, but hey, everyone has got to start some day.

Since Februrary, my main system has become FreeBSD. I switched from Linux because of a vague annoyance of the OS, hukl’s great screencast on FreeBSD, a great podcast episode (German) and several personal recommendations.

The old days

I stumbled around for a long time. My maxim was simple: Never touch a running system. Once I had punched something together, I wouldn’t touch the config files and treat them like magic knobs on a giant machine where one wrong action could blow the whole thing up.

Updates where horror, because when things broke, I would barely read the log files. The setup was always just following a tutorial withouth understanding what actually happened.

Gosh, these VPSses must have been more than inviting for any serious attacker.

What has changed

I feel more confident than ever before. Just today, I came home from vacation and some updates broke my configuration.

While solving problems still feels like poking around most of the time, I feel like I know where to poke now.

I have discovered the #freebsd IRC channel on freenode. Sometimes you have to ask twice, but usually, there is someone more experienced than you willing to help.

I have begun to use manpages properly, have learned so much from configuring and compiling the software from the ports tree and feel safer than ever with a backup script that sends incremental snapshots of my entire ZFS system to a backup server. I even filed some valid bugreports. FreeBSD feels like the right balance between Arch or Gentoo and a stable Linux distribution.

No doubt: I have broken my system more times than ever before. But I guess that is what you have to go through if you want to get better.

Flattr this!

Keeping FreeBSD up-to-date

I have to admit: my FreeBSD server had not been updated for quite a while. But I fixed that today and make the daring promise that I will keep my boxes on top of the line from now on. To internalize the commands for upating the system, I have created this little guide.

IMPORTANT: Yes, bold letters. If you don’t do it already, you should definitely run all / critical server operations in screen or preferrably tmux. This way you can let stuff run in the background and don’t have to worry about loosing network connection, etc.

Updating the system

IMPORTANT #2: Whenever you update the base system, update the jails using ezjail before reboot. This is because ezjail-admin’s update functionality relies on uname -r which would return the latest system version after reboot.

# Fetch new binary patches and apply them
freebsd-update fetch
freebsd-update install
# Update the jails using ezjail
ezjail-admin update -u

More info here.

Updating ports

What is a system without software? You’ll most certainly want to install from ports or packages and keep those up-to-date.

Strategy

I keep track of new vulnerabilities affecting my system by using jailaudit. It is part of the periodic security mails I get from my server. If there is a vulnerablity, I try to fix it as soon as possible.

But I have also decided to do updates on a regular interval. I always feared this because of the famous Never change a running system but I feel confident enough to fix things in reasonable time when they break.

As a fallback, I do a zfs snapshot zroot/ezjail/thejail@pre_update as a fallback and destroy it after some time of everything running smoothly.

→ I recommend putting live data into a jail different than your web-stuff jail. I do so because the setup web-stuff jail is far more complex (nginx, php, ruby, node, perl, …) than the data jail (just a database server). When the shit hits the fan and I can’t fix stuff in web-jail fast enough, I always have the option to do zfs rollback on the web jail with the data being untouched (unless migrations happened in which case I’ll consult my regular backups).

Give me the code

I use portmaster to update the ports. and I keep a tmux pane with /usr/ports/UPDATING open all the time. It would be awesome if there was a tool that filters UDPATING down to which software will be affected by a portmaster operation.

Update the ports tree

##for the main system
portsnap fetch update
##jail ports tree
ezjail-admin update -P

Main host & per jail

#check for updates
portmaster -L
#using the pkg2ng tool (= = no update, < = needs update)
pkg version [-vl "<"]

#actual updating
portmaster  -aD --packages
portmaster --clean-distfiles
  • -a Updates all ports that require updates
  • -D Disables cleaning of distfiles at the end of each port-installation. This disables the annoying Do you want to delete xyz_0.0.1.tar.gz?.
  • --packages Uses a package if available and the configuration matches
  • --clean-distfiles Cleans the distfiles → more disk space

Important #3: Read the pkg-messages at then end of the installation and read them carefully. Often there are hints / notices that might effect your configuration, i.e. a different path to the Passenger app server, etc.

More info here.

If things don’t work

  • Google is your friend
  • → Mailinglists
  • FreeBSD bug tracking
  • → IRC: #freebsd on irc.freenode.net

Flattr this!

Running ZNC on FreeBSD

ZNC is an IRC bouncer. When installing the software on FreeBSD using pkg install znc, you’re not able to just run the service znc (one)start – it’ll throw an error like Could not open configuration file.

Let’s check the rc.d script installed at /usr/local/etc/rc.d/znc

[...]
# znc_enable:       Set to NO by default. Set it to YES to enable it.
# znc_conf_dir:     Directory where znc configuration
#                   data is stored.
#                   Default: /usr/local/etc/znc
# znc_user:         The user account znc runs as what
#                   you want it to be. It uses 'znc' user by
#                   default. Do not sets it as empty or it will run
#                   as root.
#

. /etc/rc.subr

name=znc
rcvar=${name}_enable

command=/usr/local/bin/znc

load_rc_config ${name}

: ${znc_enable:="NO"}
: ${znc_user:="znc"}
: ${znc_conf_dir="/usr/local/etc/znc"}
[...]

What do we learn from this? The script runs znc as user znc by default using the configuration directory in /usr/local/etc/znc.

Setup user and permissions

At the time of writing, neither the user nor the directory are created by the package. To fix this, follow the percedure below:

  1. Run adduser -D (-D for disabling creation of the home directory) and create a user znc. Don’t add the user to any special groups or the like.
  2. Run mkdir /usr/local/etc/znc
  3. Run chown znc:znc /usr/local/etc/znc

The user znc now has access to the configuration directory specified in the packages rc-script.

Create znc configuration

Now it’s time to create the znc configuration file.

Create the directory for znc configs using sudo -u znc mkdir /usr/local/etc/znc/configs

Run sudo -u znc znc --makeconf, follow the instructions and specify /usr/local/etc/znc/configs/znc.conf as destination for the config file when you’re asked for this.
Check the znc wiki for more information !

Custom SSL certificate

ZNC supports SSL and has a webinterface-module for easier configuration. (Hence it needs write-permissions to its configuration directory).

You probably want to use your own certificate. tl;dr: You need to put all certificate data (i.e. private key, cert and CA certificates) into one file at /usr/local/etc/znc/znc.pem.

More information here.

Flattr this!

Nginx & Passenger

Another short post on my babystep-by-babystep server administration: I run nginx with Phusion Passenger as an app server for a Redmine installation. As I am always running into trouble when updating Passenger, I think it is time to write my setup & updating percedure down.

Installation

We need nginx and Phusion Passenger which is pretty straightforward:

#nginx with PASSENGER support enabled
portmaster --force-config www/nginx
#passenger with NGINX enabled.
portmaster --force-config www/rubygem-passenger

Nginx configuration

Your nginx needs to know where the passenger is installed. I defined this in nginx.conf:

passenger_root => $(passenger-config --root)
passenger_ruby => $(sudo -u www passenger-config --ruby-command)

Updating passenger

I follow this percedure when updating the app server:

  • Update rubygem-passenger
  • Update/Reinstall nginx
  • Get the new variables for nginx’s configuration. While passenger_ruby should be fairly stable, passenger_root will change for each version as it inlcudes the version name.

Flattr this!