Working on a Geli-Encrypted ZFS Pool from a Live-CD or Memstick

Imagine you are experimenting with ZFS. You might render your system unbootable, e.g. because you misconfigured a mountpoint property. Let’s fix this.

Note: In this post, I’ll refer to the following setup which is the one bsdinstall generates if you tick the encryption checkbox during installation.

For the sake of simplicity, we’ll assume that you have set-up a stripe on just one disk /dev/ada0. If you have set-up a mirror, you’ll have to decrypt the mirrored partitions of the ZFS pool as well to operate on it.

Who’s who

You can show the partitions on your drive using gpart:

gpart show ada0
### output ###
=>       34  419430333  ada0  GPT  (200G)
         34       1024     1  freebsd-boot  (512K)
       1058    4194304     2  freebsd-zfs  (2.0G)
    4195362    8388608     3  freebsd-swap  (4.0G)
   12583970  406846397     4  freebsd-zfs  (194G)

The number in the ada0 column is the partition you can find ad /dev/ada0p<number>.

  • ada0p1: bootstrapping code to boot off bootpool.
  • ada0p2: the ZFS pool bootpool. This one is not encrypted because Geli needs the kernel’s crypto framework. Hence, bootpool contains
    • The kernel with modules to mount ZFS, support geli, etc.
    • The geli decryption key for the partition containing zroot. Don’t worry, it’s protected by your passphrase through symetric encryption.
  • ada0p3: swap space – probably also encrypted if you checked that checkbox, too.
  • ada0p4: the geli-encrypted ZFS pool zroot.
    zroot contains at least one dataset with / as mountpoint (usually zroot/ROOT/default).

More detailed and accurate information can be found on the man-page section about BOOTSTRAPPING.

Decrypting the pool

Assuming you want to fix things on zroot, you’ll need to decrypt ada0p4 first. For decryption, you need the key-file. So let’s mount bootpool to /tmp/bootpool.

mkdir /tmp/bootpool
zfs import # lists the available pools to import. bootpool should be listed there
zfs import -N -f bootpool # forcefully import bootpool, but don't mount it.
zfs set mountpoint=/tmp/bootpool bootpool
zfs mount bootpool

The keyfile should be located at /tmp/bootpool/boot/encryption.key Now you are able to decrypt ada0p4:

geli attach -k /tmp/bootpool/boot/encryption.key /dev/ada0p4

Note: If you have a ZFS mirror or raidz set-up, you’ll have to decrypt the other partitions before importing the pool.

Importing the pool

zfs import -N -f zroot # import zroot without mounting it

Note: Be careful when mounting datasets of zroot. They have their mountpoints set to locations that aleady contain files on the live-system, e.g. zroot/ROOT/default‘s mountpoint is / but the livesystem is mounted at /, too. If you screw it up, reboot and start over.

Congratulations, you can now operate on your pool from the live-system!

SSH Into a FreeBSD Live System

One of the worst nightmares: you screwed up and your system won’t boot anymore. One approach to fix things is booting from a Live-CD. You probably don’t want to repair you system using a VNC console or similar.

Luckily, the FreeBSD Live-CD comes with an OpenSSH server. (And with dhclient which I needed in my special case.). So let’s do the repair from the comforting warmth of your local machine’s terminal emulator.

  • Boot from the Live-CD
  • Choose “Shell” on the first screen
  • Configure your network interface
ifconfig em0 inet 1.2.3.4 netmask ... # if you are using static addresses
-- OR --
dhclient em0 # if you are using DHCP
  • “Make /etc writable” and configure sshd and start the service.
    (This is the first time I knowingly used UnionFS but the approach sounds reasonable.)
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
vi /etc/ssh/sshd_config # allow root login
passwd root # Set a root password
service sshd onestart
  • You can now ssh into the live-system from your local machine and fix things.
    (Please check out my separate post about fixing ZFS from a Live-CD system.)

Note: The host keys are generated right when you run service sshd onestart. I am not sure how much entropy exists at this point.

Update

From the reddit thread:

The short answer is “enough”. The long answer is that the random subsystem blocks until it has enough entropy to initialize. You can check dmesg for the line random: unblocking device if you want to make sure it is unblocked. Usually it unblocks somewhere between the tail-end of device probing and beginning of filesystem mounts.

1Password and Touch-ID

AgileBits recently updated their famous 1Password to version 5.0 taking advantage of the LocalAuthentication.framework on iOS. Users can now identify themselves using Touch-ID – the master password is stored in the iOS keychain.

1Password_5_touch_id_unlock

Let me stress the word identification here: your fingerprint is not a password – it does at most identify you. What I want from my password manager is authentication before opening its gates to the user. In fact, what I want is two-factor-authentication.

Using the master password is inconvenient and probably leads to users disabling lock-on-exit.
Using the quick-unlock code is convenient enough for me, but I won’t leave my passwords protected by a four-digit code that could be easily watched by someone walking next / behind me.
Using Touch-ID is very convenient but there’s no authentication.

I would like to see quick-unlock code and Touch-ID combined. It’s one of the most straight-forward implementations of two-factor-authentication. And while I’m still uncomfortable with storign my password on the iOS keychain, it would be a big step forward compared to the current situation.

Keeping FreeBSD up-to-date

I have to admit: my FreeBSD server had not been updated for quite a while. But I fixed that today and make the daring promise that I will keep my boxes on top of the line from now on. To internalize the commands for upating the system, I have created this little guide.

IMPORTANT: Yes, bold letters. If you don’t do it already, you should definitely run all / critical server operations in screen or preferrably tmux. This way you can let stuff run in the background and don’t have to worry about loosing network connection, etc.

Updating the system

IMPORTANT #2: Whenever you update the base system, update the jails using ezjail before reboot. This is because ezjail-admin’s update functionality relies on uname -r which would return the latest system version after reboot.

# Fetch new binary patches and apply them
freebsd-update fetch
freebsd-update install
# Update the jails using ezjail
ezjail-admin update -u

More info here.

Updating ports

What is a system without software? You’ll most certainly want to install from ports or packages and keep those up-to-date.

Strategy

I keep track of new vulnerabilities affecting my system by using jailaudit. It is part of the periodic security mails I get from my server. If there is a vulnerablity, I try to fix it as soon as possible.

But I have also decided to do updates on a regular interval. I always feared this because of the famous Never change a running system but I feel confident enough to fix things in reasonable time when they break.

As a fallback, I do a zfs snapshot zroot/ezjail/thejail@pre_update as a fallback and destroy it after some time of everything running smoothly.

→ I recommend putting live data into a jail different than your web-stuff jail. I do so because the setup web-stuff jail is far more complex (nginx, php, ruby, node, perl, …) than the data jail (just a database server). When the shit hits the fan and I can’t fix stuff in web-jail fast enough, I always have the option to do zfs rollback on the web jail with the data being untouched (unless migrations happened in which case I’ll consult my regular backups).

Give me the code

I use portmaster to update the ports. and I keep a tmux pane with /usr/ports/UPDATING open all the time. It would be awesome if there was a tool that filters UDPATING down to which software will be affected by a portmaster operation.

Update the ports tree

##for the main system
portsnap fetch update
##jail ports tree
ezjail-admin update -P

Main host & per jail

#check for updates
portmaster -L
#using the pkg2ng tool (= = no update, < = needs update)
pkg version [-vl "<"]

#actual updating
portmaster  -aD --packages
portmaster --clean-distfiles
  • -a Updates all ports that require updates
  • -D Disables cleaning of distfiles at the end of each port-installation. This disables the annoying Do you want to delete xyz_0.0.1.tar.gz?.
  • --packages Uses a package if available and the configuration matches
  • --clean-distfiles Cleans the distfiles → more disk space

Important #3: Read the pkg-messages at then end of the installation and read them carefully. Often there are hints / notices that might effect your configuration, i.e. a different path to the Passenger app server, etc.

More info here.

If things don’t work

  • Google is your friend
  • → Mailinglists
  • FreeBSD bug tracking
  • → IRC: #freebsd on irc.freenode.net

Recommendation: Soulver

Have you ever felt the need to do some calculations using variables, mathematical functions, unit conversions, etc? Have you ever felt the need to punch Calculator.app into the face?

Soulver might be worth a try. It’s a mix of spreadsheets and writing pseudo-code.

I’ve been using it for several years and it’s what suits my thought-process best.

TinyTinyRSS: Migration hickup

I use TinyTiny RSS for keeping up-to-date with my ~200 RSS feeds. Together with the Fever API comptability plugin, this has been a sweet solution for me since Google Reader closed its doors.

However, yesterday, I was migrating the MySQL server that is accessed by TTRSS to a separate jail. At first look, everything seemed to work after changing config.php:

define('DB_HOST', 'localhost');
define('DB_HOST', '<The MySQL-Server>');

However, this morning I noticed that the feeds had not been updated. update.php is usually run by a cronjob, but when executing it manually, you saw error messages in certain feeds. In my case, it’s the famous iOS Dev Weekly feed.

 Fatal error: Query INSERT INTO ttrss_error_log
(errno, errstr, filename, lineno, context, owner_uid, created_at) VALUES
(256, 'Query INSERT INTO ttrss_entries\n (title,\n guid,\n
link,\n updated,\n content,\n content
_hash,\n no_orig_date,\n date_updated,\n date_entered,\n
comments,\n num_comments,\n plugin_data,\n lang,\n
author)\n VALUES\n (\'Issue 153 - 4th July 2014\',\n \'SHA1:c4acc
25a37b992b33491f0d8c5b49e2008e470a3\',\n \'http://iosdevweekly.com/issues/153\',
[Post content]
in /path/to/ttrss/classes/db/mysqli.php line 33

Quite cryptic. Looking at the database, I learned that TTRSS is storing the error logs themselves in the database. Does not make any sense for storing databse-related errors, but ok, it worked so far…. As the errors stored there were fairly up-to-date, I decided it was not an issue with access permissions.

Fiddling around a little more, I noticed the following configuration option for config.php:

define('MYSQL_CHARSET', 'utf8');
// Connection charset for MySQL. If you have a legacy database and/or experience
// garbage unicode characters with this option, try setting it to a blank string.

I fixed my issue by setting MYSQL_CHARSET  to a blank string first and afterwards to utf8 again. If you don’t reenable it, the web interface does not render certain feeds.


I suppose this was an encoding issue although I cannot explain why: the database is setup to use UTF-8 and the feeds contain only standard characters (would fit into ASCII).

To the poor guy who is searching the web for a solution to this apparently rather exotic problem for two hours: you’re welcome.


UPDATE:

After all, I think I have figured out the encoding issue: The fix above stopped working after a few hours – I guess because some new item in a feed caused an encoding error. My current solution is to force utf8mb4 for communication betweeen mysql client and server.

#server's my.cnf
[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
#ttrss config.php
define('MYSQL_CHARSET', 'utf8');

Reading List

Für mehr als zwei Jahre war ich eingeschworener Instapaper Subscriber und habe den Service zumeist mit dem Hintergrund benutzt, ein Archiv von all dem zu haben, was ich mal gelesen habe – es könnte ja sein, dass man das irgendwann nochmal herauskramen möchte.

Wie ich festgestellt habe: Dem war nicht so. Vor allem die Full-Text-Suche, die man als Subscriber bekommt, habe ich höchstens zwei Mal benutzt.

Mit iOS 7 ist nun für Entwickler die Option dazugekommen, Links zur Reading List hinzuzufügen, und genau das hat noch für den Umstieg gefehlt: Tweetbot 3 kann es, Reeder hoffentlich bald auch – viel mehr brauche ich in der täglichen Nutzung nicht.

Was Reading List besser macht

Safari Reading List hat standardmäßig keinen Textparser. Dieser hat bei Instapaper eine Zeit lang ganz OK funktioniert, fangen die Seiten aber mit JavaScript-Spielereien an, kann man das Feature direkt vergessen. Die Reader-Ansicht macht bei Bedarf aber aus fast allem gut lesbaren Text.

Bookmarking wird von mit jetzt erstmals intensiv genutzt – als Ersatz für Instapapers Archiv. Natürlich findet sich in den Bookmarks nicht ein durchgetagger Full-Record von allem, was ich gelesen habe, aber Artikel, die ich irgendwo zusätzlich zur recht unübersichtlichen Reading-List-History speichern möchte, haben hier genug Platz. Für den Rest vertraue ich ab jetzt auf die Suchmaschine meines geringsten Misstrauens.

Was ich mir noch wünsche

Eine (von mir aus auch write-only) Web-API für Reading List wäre wünschenswert, wird aber wahrscheinlich in Zeiten der Platform-Kriege nicht kommen.