Host Configuration

This sections describes which configuration steps we need to take to be able to install our services in jails.

While we are working on the system, we will be using the root user. After logging in with your general user, change into root with

su -

Any further commands are expected to be run as root.

System Update

First things first. Lets install any security patches released for the system. This is done with the following two commands

freebsd-update fetch
freebsd-update install

The fetch subcommand downloads any applicable patches. install will install all downloaded patches.

Basic Tools

We want some basic tools on the host. These are optional, but they might make life quite a bit easier. We will install the following

• micro - as our main editor (note: I am also very fond of helix, but helix has a bug which affects file ownership, so we opt for micro)
• bat - alternative to cat
• curl - always usefull

Install them with the following command

pkg install micro bat curl

SSHD

We want to use key-based auth when login in to the host with our username and disallowing login as root. To change the root user, one still needs the root password, but this is only possible after login in as a regular user with a key.

From you local shell transfer you public key to the freebsd host.

ssh-copy-id psykon@ez1.ezdk.org 

Enter your password and see if you can login afterwards without one. If this has worked and you are back on the freebsd host, change again to root with su - and follow the remaining guide.

Let’s adapt our SSHD configuration:

micro /etc/ssh/sshd_config

Make sure, the following options are set

PermitRootLogin no
HostbasedAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no

And restart sshd

service sshd restart

Bastille

We will use bastille to manage the jails. We need to install and bootstrap it.

pkg install bastille

# enable bastille services
sysrc bastille_enable=YES
sysrc bastille_rcorder=YES

Automatic Setup

Bastille comes with an easy setup to adjust our system for jails that communicate over an internal loopback interface, with NAT to the WAN an storage managed by ZFS.

We can start this setup with bastille setup. You should see something like this:

bastille setup
bastille_enable: YES -> YES
Configuring bastille0 loopback interface
cloned_interfaces:  -> lo1
ifconfig_lo1_name:  -> bastille0
Bringing up new interface: bastille0
Created clone interfaces: lo1.
Determined default network interface: (vtnet0)
/etc/pf.conf does not exist: creating...
pf_enable: NO -> YES
pf ruleset created, please review /etc/pf.conf and enable it using 'service pf start'.
bastille_zfs_enable: NO -> YES
bastille_zfs_zpool:  -> zroot

Check the output for the following values:

• Is the new interface (lo1_name) called bastille0?
• Has bastille detected the interface which we configured in the FreeBSD setup as the default network interface?

Finally, check the content of /etc/pf.conf with bat /etc/pf.conf. It should contain something like this, except for a different interface name on the second line:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

## generated by bastille setup
ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state

If everything looks like this, activate pf with service pf start. You will probably be disconnectec from your SSH session. Simply reconnect. If you find, that you cannot reconnect, access the machine through your hosters console interface and disable pf with

sysrc pf_enable="NO"
service pf stop

If you run into trouble with the automatic setup of bastille, you can follow the manual guide in the next chapter, otherwise, jump directly to Bootstrapping

Manual Setup

Add a new loopback interface an start it. The new interface should be named “bastille0”.

# setup networking
sysrc cloned_interfaces+=lo1 # we add a new loopback network interface, lo1
sysrc ifconfig_lo1_name="bastille0" # this interface will have the name "bastille0". We will use this name in our jail configuratins
service netif cloneup # start the interface

Next we need to enable and configure our firewall - pf. We want it to block any requests from the internet, except on the port on which sshd is running. Requests from inside the jails should always be allowed (so that we can download software, etc). We also want to allow specific ports to be forwarded to a jail, so that requests on 433 will arrive at our reverse-proxy. bastille does most of that for us, but we still need to do the following steps to enable it:

Enable pf (but not starting it yet)

sysrc pf_enable="YES"

Add the following to the pf configuration file at /etc/pf.conf make sure to replace vnet0 with the name of the network interface through which the host is connected to the internet.

micro /etc/pf.conf

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state

Also make sure, that sshd is only listening on the “outward facing” network interface and not on lo/lo1/bastille0 or other clones.

Next, enable ZFS support for bastille.

# -f tells sysrc to write to another file instead of /etc/rc.conf. We tell it to write to the bastille configuration.
sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_enable=YES
sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_zpool=zroot # Name of te zpool which bastille should use for jails

Bootstrapping

Now we can bootstrap a FreeBSD Release for use in containers. The bootstrap command will create a zfs dataset, which then can be cloned to create new jails.

bastille bootstrap 14.2-RELEASE

On using Jails

bastille offers multiple commands to modify the configuratin and content of the jails. The syntax is usually bastille <command> <jailname> <subcommand/values>. E.g. to install micro in the caddy jail you can execute bastille pkg caddy install micro. This allows us to set up and configure our jails from the comfort of our hosts root user.

At anytime you can also enter the jail with bastille console <jailname>. This will log you in as root in the jail. You can then use the jail just like another FreeBSD system. Pretty handy if you need to debug something.

Fail2Ban

While we have set up sshd to only allow key based auth, we still want to be able to ban clients which try to brute-force our host. So we setup Fail2Ban. This will also come in handy further down the road, to ban clients that try to bruteforce various services.

First, we have to install it:

pkg install py311-fail2ban

and enable it

sysrc fail2ban_enable=YES

We’ll use fail2ban together with pf. For this to work, we need to add a few rules to /etc/pf.conf that instructs our firewall to block IPs which Fail2Ban has selected for blocking.

The following three lines are required

table <f2b> persist
anchor "f2b/*"
block drop in log quick on $ext_if from <f2b> to any

The last line - which actually blocks the request, we’ll put at the very end of our pf.conf file, so that the whole thing has the following content:

ext_if="igc0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

table <f2b> persist
anchor "f2b/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state

block drop in log quick on $ext_if from <f2b> to any

Now we need to adjust the fail2ban configuration to actually detect missbehaving ips and write them to the files read by pf. Start by creating a new file at /usr/local/etc/fail2ban/jail.local

micro /usr/local/etc/fail2ban/jail.local

and add the following content

[DEFAULT]
banaction = pf[actiontype=<allports>]
banaction_allports = pf[actiontype=<allports>]

[sshd]
enabled = true
findtime = 3600
maxretry = 2

Now we can load the new pf configuration and start the fail2ban service.

pfctl -f /etc/pf.conf
service fail2ban start

And check if it is running

fail2ban-client status sshd

To see, if any ips have been banned, you can also use

fail2ban-client banned

To check if the same ips are acually blocked by pf, execute

pfctl -a 'f2b/sshd' -t 'f2b-sshd' -Ts

Snapshot Management

One of the nice things about ZFS is that we can easily take snapshots of each of the individual datasets. Since bastille creates datasets for each jail automatically for us, this gives us the option to rollback the whole machine or individual jails at any time.

Shedule

We’ll use the following shedule for the snapshots.

• One Snapshot every Hour, which we will keep for one Day
• One Snapshot every Day, which we will keep for one Week
• One Snapshot every Week, which we will keep for one Month
• One Snapshot every Month, which we will keep for one Year
• One Snapshot every Six Months, which we will keep for four Years

Setup

We make our lifes a bit easier and use a wrapper over zfs to create and delete our snapshots which is called zfsnap. To install it, execute

pkg install zfsnap2

After that, we can add the following entries to /etc/crontab

# ZFS Snapshots
#
# Create Snapshot every 6 Months - Keep for 4 years
10  1  1  */6  *  root  zfsnap snapshot -a 4y -r zroot
# Create Snapshot every Month - Keep for 1 year
20  1  1  1  *  root  zfsnap snapshot -a 1y -r zroot
# Create Snapshot every Week - Keep for 1 month
30  1  *  *  0  root  zfsnap snapshot -a 1m -r zroot
# Create Snapshot every Day - Keep for 1 Week
40  1  *  *  *  root  zfsnap snapshot -a 1w -r zroot
# Create Snapshot every Hour - Keep for 1 Day
50  *  *  *  *  root  zfsnap snapshot -a 1d -r zroot 
#
# Delete all Snapshots which have exeeded their TTl daily
10  2  *  *  *  root  zfsnap destroy -r zroot

You can list all the datasets including their snapshots with

zfs list -t all

Backups

Snapshots are great to restore the system to an earlier state in case of misconfiguration, data deletion and so on. But we still need a solid automated backup of our data in another location in case of lost of the host-system, a failure at the hoster or any other catastrophic issue. We’ll use restic to create, encrypt and transfer our backups and cron to automate it. Restic takes incremental backups and only saves data that has changed. It can work with local storage, s3 compatible storage and everything thats reachable via sftp. So as long as we have some disk space reachable from the host that is either writeable throug s3 or sftp, we can setup our remote backup and from there, transfer the backup-repository onto further locations (or something like a external HDD that is kept in a locker most of the time) and be able to verify and restore this backup as long as we have the encryption key.

Setting up Backups with restic

Log onto the host and change to root. Copy the SSH keypair used to connect to the remote host/storage space to /home/root/.ssh/. You might also want to add a config file in that location to specify which user/port/idendity are to be used with the remote host. After that, install restic with

pkg install restic

Now we can initiate the repository (the structure into which restic writes all the backed up data) on the remote host with the following command. This will create the repository in a directory called restic-repo in the home directory of <user> on <host.tld>.

restic -r sftp:<user>@<host.tld>:/restic-repo init

Enter and confirm the password for the repository. This will be the encryption-key of the repository - it is not asking for the users password on the remote host (we use key based authentication for that, since we want to automate it).

To automate the backup, restic needs to be provided with the address to the repository and it’s password. For this, we will create two files in the roots home directory.

mkdir ~/.restic
touch ~/.restic/repo
touch ~/.restic/secret

Write the full connection string to the repository (E.G. sftp:@<host.tld>:/restic-repo) into ~/.restic/repo and the password into ~/.restic/secret. We will reference these two files in the restic commands with –repository-file and –password-file. This allows us to move to some other storage provider in the future and we’ll only have to change the values in these two files.

The last thing we need to do is to modify the /etc/crontab file to include restic commands for weekly backups. There are three directory-structures we want to backup:

• /etc To save all the host-specific configuration
• /home To save all the files in users home directories
• /usr/local To save all the bastille config and jails (under /usr/local/bastille), software configuration, and directories we mounted in jails

To set this up, add the following lines to /etc/crontab

# restic Backup
# weekly /usr/local
30	4  *  *  0  root  restic --repository-file ~/.restic/repo --password-file ~/.restic/secret backup /usr/local --skip-if-unchanged
# weekly homedirs
30  5  *  *  0  root  restic --repository-file ~/.restic/repo --password-file ~/.restic/secret backup /home --skip-if-unchanged
# weekly /etc
50  5  *  *  0  root  restic --repository-file ~/.restic/repo --password-file ~/.restic/secret backup /etc --skip-if-unchanged

Now where all set and all the data needed to restore the system is savely stored on a remote system. To check the state of the remote repository, execute

restic --repository-file ~/.restic/repo --password-file ~/.restic/secret snapshots

To see which files are actually contained in a snapshot, use

restic --repository-file ~/.restic/repo --password-file ~/.restic/secret ls <hash-of-snapshot>

To replicate the backup onto other hosts, simply use sftp/rsync to regularly copy the restic-repo from the remote-host onto other systems.