Introduction: Kuiper, PFSense Firewall

This system is a 1U quarter-depth Super Micro, Intel Atom-based system. Its name is slightly irregular considering my server naming scheme is planets orbiting Sol, but I figure “Kuiper” isn’t too hard to associate with a firewall, as the Kuiper belt encapsulates our solar system.

Low power consumption and threading was important when deciding on which hardware to get for this sytem. I expect it to push 1gbps (inter-VLAN) without too much trouble. It has five ethernet ports, where one is dedicated for IPMI. WAN has a dedicated ethernet port, and I’ll be setting up LAGG between at least two ports for the LAN side, for more efficient routing between VLANs, once the 16-port switch arrives.

System Parts
Chassis SuperChassis CSE-505-203B
Mainboard Super Micro A1SAI-2550F
CPU Intel Atom C2550 (14W, embedded)
RAM 1x 8GB DDR3-1600 ECC SODIMM
SSD 1x Innodisk 32GB InnoLite LP Vertical (for 1U) SATADOM MLC

I did a very simple network benchmark (iperf -c hostname -p 20000 -t 60 -P 1 -L 20000 -w 1m -t 30 -i 5), which showed about 600 Mbit/s  bandwidth. Kuiper’s CPU load was hovering around 10% during the test. I’ll have to do more scientific tests later, as I haven’t made any effort to find the actual bottleneck for this test.

FreeBSD, NGINX, SSL and the ChaCha20 cipher suites

In this post, I’ll be describing the journey of enabling the stronger ChaCha20 cipher suites on my FreeBSD NGINX reverse proxy. I’m using SSL Labs SSLTest to get the details on what is being offered, and for sanity-testing the SSL configuration in general. I’m using FreeBSD 10.2.

In nginx.conf:

http {
  (...)
  ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers         EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!MD5;
  ssl_prefer_server_ciphers on;
  (...)
}

This clearly shows the server prefers the EECDH+CHACHA20 cipher suite. But according to the SSL test, it’s not being offered. Why could this be? I’m using whatever OpenSSL version is in the FreeBSD 10.2 base system, so I go poke it to find which ciphers it supports:

# /usr/bin/openssl version
OpenSSL 1.0.1p-freebsd 9 Jul 2015
# /usr/bin/openssl ciphers | grep -i chacha
#

Apparently, it doesn’t support the ChaCha cipher suites. Ok, then it’s obvious why NGINX isn’t offering those cipher suites. So I go over to my package build server and change its configuration around a little. I add security/libressl to the build list, and I instruct it to build the reverse proxy’s packages with the following make.conf options:

# Build ports against security/libressl
WITH_OPENSSL_PORT=      yes
OPENSSL_PORT=           security/libressl

I instruct the package builder to run a build for the reverse proxy, and it seems to be doing what I wanted. All ports linking to OpenSSL (including NGINX) are being rebuilt, and LibreSSL is built too.

Once the build server is done, I tell the reverse proxy to force an update and upgrade. (pkg update -f and pkg upgrade -f), restart NGINX, and test again.

It’s still not serving CHACHA20. WHAT!? I check if NGINX actually linked against LibreSSL:

# /usr/local/bin/openssl version
LibreSSL 2.2.4
# ldd /usr/local/bin/openssl
/usr/local/bin/openssl:
        libthr.so.3 => /lib/libthr.so.3 (0x800888000)
        libssl.so.35 => /usr/local/lib/libssl.so.35 (0x800aac000)
        libcrypto.so.35 => /usr/local/lib/libcrypto.so.35 (0x800d12000)
        libc.so.7 => /lib/libc.so.7 (0x801110000)

# ldd /usr/local/sbin/nginx
/usr/local/sbin/nginx:
        libthr.so.3 => /lib/libthr.so.3 (0x8008db000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x800aff000)
        libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x800d1f000)
        libssl.so.35 => /usr/local/lib/libssl.so.35 (0x800f96000)
        libcrypto.so.35 => /usr/local/lib/libcrypto.so.35 (0x8011fc000)
        libz.so.6 => /lib/libz.so.6 (0x8015fa000)
        libc.so.7 => /lib/libc.so.7 (0x801810000)

Yes, it is linked against LibreSSL. And /usr/local/bin/openssl ciphers does list CHACHA20. This is very strange. While trying to figure out what’s going on here, I strike up a conversation with Allan Jude on IRC and casually mention my troubles. He mentions something about ChaCha20 being slower (not a bad thing), citing https://wiki.freebsd.org/SSHPerf, and mention some black magic that is using the OpenSSL client to check the SSL session handshake. So I do that.

# /usr/local/bin/openssl s_client -host HOSTNAME -port 443
(...)
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
(...)

What? This is saying it’s using CHACHA20! Why isn’t the SSL test saying so? And then Allan casually says the SSL report does state the preferred cipher is CHACHA20. I go refresh the page, and indeed it does.

So to make a long story short, I had to link NGINX with LibreSSL to get CHACHA20 support, once that was done, I checked the now stale report to see CHACHA20 wasn’t supported, even though it actually was.

Lesson of the story: Be thorough. Very thorough. Messing up a simple step can make you stare at the screen wondering if you’ve lost your sanity. Also, sharing the trouble can be entertaining for those you share it with, and save you time. Win-win. :)

Let’s Encrypt on a FreeBSD NGINX reverse proxy

This is a write-up on how I set up “Let’s Encrypt” on the reverse proxy sitting in front of the various VM’s serving a few of my websites. I looked at a guide which was very helpful, but I had to fill in on some gaps and tweak the configuration slightly. I’ll be outlining every step of the way here.

Let’s Encrypt let people enable HTTPS with a trusted certificate, for free. You can even get multiple-domain certificates, in case you run multiple websites behind a single IP address.

First of all, I installed the Let’s Encrypt package.

I then configured nginx to serve the magic folder for verification (/usr/local/etc/nginx/sites-enabled/letsencrypt.conf), and made my “real” vhosts only listen for SSL traffic. (You may have to temporarily disable them or add the magic stuff to each of them, if you didn’t have a SSL configuration at all before this.) I then reloaded nginx.

This is the catch-all ‘magic’ vhost for verification. It will redirect real traffic to the https version of the site.

server {
  server_name example.com something.example.com anotherdomain.example;
  listen 80;
  location '/.well-known/acme-challenge' {
    default_type "text/plain";
    root /tmp/letsencrypt-auto;
  }
  location / {
    return 301 https://$host$request_uri;
  }
}

I then executed the following commands to create my certificate:
export DOMAINS="-d example.com -d something.example.com -d anotherdomain.example"
export DIR=/tmp/letsencrypt-auto
mkdir -p $DIR
letsencrypt certonly --server https://acme-v01.api.letsencrypt.org/directory \
-a webroot --webroot-path=$DIR --agree-dev-preview $DOMAINS

This command outputs the path to the directory containing the certificate files. “privkey.pem” is the private key, and “fullchain.pem” is the file you want to use as certificate.

I updated the nginx configuration to use these certificates, in /usr/local/etc/nginx/nginx.conf:

http {
  (...)
  ssl_certificate /usr/local/etc/letsencrypt/live/example.com/fullchain.pem;
  ssl_certificate_key /usr/local/etc/letsencrypt/live/example.com/privkey.pem;
  (...)
}

I then created a script [letsencrypt_renew.sh] which renews the certificate when it’s 14 days or less from expiring. I set up crontab to call it once a day, and only report about any errornous output:

13 2 * * * /root/letsencrypt_renew.sh /usr/local/etc/letsencrypt/live/example.com/fullchain.pem > /dev/null

And that’s it. My websites which are hosted at home are now served over HTTPS with a trusted certificate. For free.

The Let’s Encrypt public beta will start 3rd December 2015, good luck!

Manually compiling your own FUSE file system on FreeBSD

This is a combined rant and tutorial. The tutorial is available further down, under its own subheading. :)

It started Saturday, when I decided to jump in and get my hands dirty with the FUSE API. I started looking for the API documentation, but couldn’t find any which were relevant for my needs. I found some describing the internal kernel API, but nothing describing how to USE that API.

I found some example code with instructions describing how to compile it. These instructions state “gcc -Wall hello.c pkg-config fuse –cflags –libs -o hello”. I got errors about directories not found and flags not recognized. Oh well, not too surprising. I was maybe a little bit optimistic in thinking it was as easy as replacing gcc with clang!

So I went on to google for cmd switch replacements, etc, to no avail. After banging my head on this for over a day, I figured it’s better to leave this problem for another day, and just go with gcc for now. I installed gcc, and… same problem!

I then tweeted, hoping someone would have some suggestions for me. Almost immediately, @badboy_ replied, saying it was displayed wrong, and some of it should be wrapped in ticks. Okay, so I tried gcc48 -Wall hello.c `pkg-config fuse --cflags --libs` -o hello. Now it was complaining about pkg-config not being a valid command. Okay, one step further. In a combination of frustration and joy I immediately ask where to find “pkg-config” for FreeBSD. @FreeBSDHelp mentioned pkgconf. I tried the substitution trick on this, and the command line went gcc48 -Wall hello.c `pkgconf fuse --cflags --libs` -o hello. Okay one step further. It’s now complaining about a missing package for fuse. Digging around the ports tree, I find sysutils/fusefs-libs. Install it, try again.. and voila! It compiles, and works.

I once again try the clang-for-gcc substitution trick, and it works as a charm now. I immediately uninstalled gcc. ;)

How to Compile Your Own FUSE File System on FreeBSD?

  1. Install the package sysutils/fusefs-libs
  2. Have some code which uses FUSE (let’s assume hello.c from http://fuse.sourceforge.net/doxygen/hello_8c.html)
  3. Execute: clang -Wall hello.c `pkg-config fuse --cflags --libs` -o hello

That’s how easy it is, really. Now if only someone could have written that somewhere it could be found. :)

Changelog:
2015-10-28: Replaced ‘pkgconf’ with ‘pkg-config’ according to @myfreeweb‘s tweet.
2015-10-28: Added freshports.org link to package name.
2015-10-28: Cleaned up some links, adjusted text to reflect the post wasn’t published the day I started writing it.
2015-10-28: Added profile links for twitter handles.
2015-10-28: Made the link to @badboy_’s tweet more visible and added link to @FreeBSDHelp’s tweet.

ELI5: FreeBSD Accept Filters

Five months ago, I wrote the following as a response to a Redditor who asked how accept filters worked in FreeBSD, and wanted to have it explained like they were five years old. I’m posting it here, because it’s a recurring question, and I’d like it somewhere easy to find. Original thread.

Without accept filters: Imagine if someone were to send you a message by letter. They’d send one sentence the first day, the second sentence the second day, and so forth. You’d go check that mail box every day, because the ‘new mail’ flag was up. You piece the sentences together, and after a number of days you have the full message.

With accept filters: Imagine the above example, but the mail box scanned the contents of your letters, and wouldn’t raise the ‘new mail’ flag until there are enough sentences to form a full paragraph (request). You’d spend less time checking the mail box, and you’d still process the message at the earliest possible time.

The advantage to this is more noticeable when you have to check many mail boxes at the same time, and can skip the ones which don’t have a full paragraph yet.

Error ID 10-T: IPMI Entering Coma

Today, I had an “accident” configuring my new firewalls IPMI. Okay, I’ll admit. It was a PEBKAC situation: I wanted to configure the firewalls IPMI to use a dedicated network interface, but the option was disabled in the web UI. So I used Chromes debug mode to force-enable the drop-down menu, selected ‘Dedicated’, and saved. After a few seconds, there was an error dialog box on the web interface stating it failed to save the settings. And bam, the remote KVM console was disconnected, and the web interface went dead. A bit of investigation revealed that the IPMI had dropped off the network.

Continue reading

Converting a FreeBSD MySQL server to jail host with MySQL in jail

I have a FreeBSD 10.0 server which currently only runs Percona MySQL server 5.6 backed by ZFS. The SQL server doesn’t have a high enough load to justify dedicated hardware, but I also don’t want to run it as a virtual machine as I want to use local ZFS storage, and because of virtualization overhead. The server is dual-homed (DMZ and LAN).

The solution is to convert the server into a jail host, and run MySQL inside a jail. The overhead should be minimal to non-existing as I won’t be using VNET.
Continue reading