Core Services

This section describes how to install, configure and expose the servies on which other applications will depend on. Such as identity providers.

Reverse Proxy

The first service we are gonna set up is a reverse proxy. This is a central component since any user-traffic must go through the reverse proxy, as it is the only service that is exposed to the internet. All other services are only reachable trough this proxy. Just as any other service running on the host, it will be isolated in a jail.

We are using Caddy for all our webserver and proxy-needs. It’s easy to configure and has many nice features built in.

Jail creation

After having bootstrapped a release. We can create a new Jail. The bastille command for creating a new jail is bastille create NAME RELEASE IP [INTERFACE]. We’re gonna call this jail simply caddy. Because some of our services will be bound to the externally exposed interface, and not the bastille loopback interface, the caddy jail will need to ‘inherit’ the full network definition from the host. We can do this by specifying the inherit keywork instead of an ip/interface when creating the jail. Execute the following command to create the new jail

bastille create caddy 14.2-RELEASE inherit

Caddy Installation

Afterwards we want to install caddy and enable its service and start it. We also install micro in the jail.

bastille pkg caddy install caddy
bastille sysrc caddy caddy_enable=YES
bastille service caddy caddy start

Caddy will bind to ports 80 and 443 (http & https). For it to be reachable, we will need to adjust the firewall rules to allow traffic to pass throug these ports.

pass in inet proto tcp from any to any port {ssh,http,https} flags S/SA keep state

and reload the config

pfctl -f /etc/pf.conf

Caddy Configuration

Since we currently have no service to expose, we will only define a hard coded response to see if caddy works. We will configure caddy trough its default Caddyfile. This file is located in the caddy jail at /usr/local/etc/caddy/Caddyfile. We can access the jails filesystem at /usr/local/bastille/jails/<jailname>/root/. So to edit the Caddyfile in the caddy jail, execute

micro /usr/local/bastille/jails/caddy/root/usr/local/etc/caddy/Caddyfile

Delete the whole content of the file and replace it with this:

test.ezdk.org {
  respond * 200 {
  body "Hello World"
  }
}

Replace ezdk.org with you own domain and set up a DNS record for test.<domain> that points to the public ip of the machine.

Tell caddy to reload the configuration with bastille service caddy caddy reload and test if you get the correct response by running curl -L test.<domain> on your local machine

curl -L test.ezdk.org
Hello World%

Certificate Management

We want to be able to give other services read access to selected certificates which caddy pulls via ACME. To do that, we’ll move the directory which caddy uses to store the certificates to the hosts filesystem at /usr/local/certificates and then mount it into the caddy jail (as well as into other jails which need access). From the host (not inside the jail) execute the following commands

cp -r /usr/local/bastille/jails/caddy/root/var/db/caddy/data/caddy/certificates /usr/local/
bastille stop caddy
rm -r /usr/local/bastille/jails/caddy/root/var/db/caddy/data/caddy/certificates
bastille mount caddy /usr/local/certificates /var/db/caddy/data/caddy/certificates nullfs rw 0 0
bastille start caddy

Then check that our test endpoint still works over HTTPS.

LDAP

We want centralized user management, and LDAP is still the best way to facilitate it. But on this scale, most LDAP implementations would be overkill. Thankfully, there is lldap which implements only the essentials of LDAP, while still providing a nice web interface and some selfservice features for users (e.g. changing their passwords).

Jail Creation

We want to isolate lldap by running it in its own jail. Let’s create it and give it the ip 10.0.0.10:

bastille create lldap 14.2-RELEASE 10.0.0.10/8 bastille0

lldap setup

Next, we want to setup lldap in the jail. The setup for FreeBSD requires some extra steps (such as downloading an rc.d file for the service). For this, it is easier to to execute the commands directly in the jail instead of using bastille. We enter the jail with the console commmand:

bastille console lldap

Now we can enter the following commands to install and start lldap.

# Start by downloading the lldap binaries and the rc.d script
# The fetch command is a part of the  FreeBSD userland and
# made for this task.
fetch https://github.com/n-connect/rustd-hbbx/raw/refs/heads/main/x86_64-freebsd_lldap-v0.6.1.tar.gz
fetch https://github.com/lldap/lldap/raw/refs/heads/main/example_configs/freebsd/rc.d_lldap

# Unpack the binaries to /usr/local and rename the directory to lldap_server
tar -xvf x86_64-freebsd_lldap-v0.6.1.tar.gz -C /usr/local/
mv /usr/local/x86_64-freebsd/ /usr/local/lldap_server

# Move the rc.d script to /usr/local/etc/rc.d/lldap
# make it executable and enable the service
mkdir -p /usr/local/etc/rc.d/
mv rc.d_lldap /usr/local/etc/rc.d/lldap
chmod +x /usr/local/etc/rc.d/lldap
sysrc lldap_enable="YES"

# Start the lldap service
service lldap start

# You can check if lldap is running with
service lldap status

Exit the jail, by exiting its shell.

exit

Now we are back in our hosts shell.

We have to adjust our reverse proxy to expose the web-interface of lldap. Edit the Caddyfile in the caddy jail.

micro /usr/local/bastille/jails/caddy/root/usr/local/etc/caddy/Caddyfile

Add the following block to the Caddyfile. Replace ezdk.org with the domain pointing to this host

lldap.ezdk.org {
  reverse_proxy 10.0.0.10:17170
}

Let caddy reload the configuration:

bastille service caddy caddy reload

Open a browser, go to lldap. and login with user admin and password password. Open Account Details and immediately change the password to something secure.

Users & Groups

To prepare our ldap directory for use with other services, especially NextCloud, lets add an user that nextcloud will use to query the directory and a group which will identify users which should be able to log into nextcloud.

Create a new user through Users > Create a user. Give it the name ro_admin, Display Name can be ReadOnly Admin, enter an email address you have access to and set a complex password. After you have created the user, open its details by clicking on it in the users list and under Group emberships select the group lldap_strict_readonly from the drop-down list. Then press the Add to group button. This will give the ro_admin user access to the directory, but only in readonly mode.

Add a new group through Groups > Create a group and call it nextcloud_users. Users in this group will be able to log into nextcloud. Also add a user for yourself (so you can test the functionality). Add an user through Users > Create a user. You can name it however you want. Be sure to add a display name and an email address you have access to. After you have created the user, add membership of the group nextcloud_users to it.

Email

Host Configuration

We need to enable linux binary support on the host, to be able to run the linux stalwart binary in a jail.

sysrc linux_enable="YES"
service linux start

Create a jail for stalwart

bastille create stalwart 14.2-RELEASE inherit

Stop the jail with bastille stop stalwart

Edit the new jails jail.conf and add the lines containing allow.mount:

stalwart {
  enforce_statfs = 2;
  devfs_ruleset = 4;
  exec.clean;
  exec.consolelog = /var/log/bastille/stalwart_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = stalwart;
  mount.devfs;
  mount.fstab = /jails/jails/stalwart/fstab;
  path = /jails/jails/stalwart/root;
  securelevel = 2;
  osrelease = 14.2-RELEASE;

  allow.mount;
  allow.mount.devfs;
  allow.mount.fdescfs;
  allow.mount.procfs;
  allow.mount.linprocfs;
  allow.mount.linsysfs;
  allow.mount.tmpfs;

  ip4 = inherit;

  ip6 = disable;
}

Now start the jail again with bastille start stalwart and change into the jail with bastille console stalwart.

We need to install a linux baselayer in the jail. Use the following command to install the rockylinux9 layer:

pkg install linux_base-rl9

Stalwart installation

Get the latest linux release archive from https://github.com/stalwartlabs/stalwart/releases and extract it

fetch https://github.com/stalwartlabs/stalwart/releases/download/v0.13.1/stalwart-x86_64-unknown-linux-gnu.tar.gz
mkdir /usr/local/stalwart
tar xf stalwart-x86_64-unknown-linux-gnu.tar.gz -C /usr/local/stalwart

Change into the stalwart dir. Make the binary executable and initialize it in the current directory.

cd /usr/local/stalwart
chmod +x stalwart
./stalwart --init /usr/local/stalwart

Note down the admin passwort.

Service

Add the following to the file /usr/local/bastille/jails/stalwart/root/usr/local/etc/rc.d/stalwart

#!/bin/sh

# PROVIDE: stalwart
# REQUIRE: DAEMON NETWORKING
# KEYWORD: shutdown

# Add the following lines to /etc/rc.conf to enable stalwart:
# stalwart_enable : set to "YES" to enable the daemon, default is "NO"

. /etc/rc.subr

name=stalwart
rcvar=stalwart_enable

stalwart_chdir="/usr/local/stalwart"

load_rc_config $name

stalwart_enable=${stalwart_enable:-"NO"}

logfile="/var/log/${name}.log"

procname=/usr/local/stalwart/stalwart
command="/usr/sbin/daemon"
command_args="-u root -o ${logfile} -t ${name} /usr/local/stalwart/stalwart -c ./etc/config.toml"

run_rc_command "$1"

And make it executable

chmod +x /usr/local/bastille/jails/stalwart/root/usr/local/etc/rc.d/stalwart

Then enable and start the service.

bastille sysrc stalwart stalwart_enable=YES
bastille service stalwart stalwart start

Caddy & Certificates

Now adjust our Caddyfile to proxy traffic to this jail by adding the following lines to /usr/local/bastille/jails/caddy/root/usr/local/etc/caddy/Caddyfile

mail.ezdk.org {
  tls {
      reuse_private_key
  }
	reverse_proxy 127.0.0.1:1443 {
		transport http {
			proxy_protocol v2
			tls_server_name mail.ezdk.org
		}
	}
}

stalwart.ezdk.org {
	reverse_proxy 127.0.0.1:8080
}

Reload the caddy config

bastille service caddy caddy reload

And send a request to the mail.<domain.tld> endpoint (so that caddy fetches the certificates).

Stalward needs access to these certificates. So we mount the directory in which caddy writes these certs into the stalwart jail:

bastille mount stalwart /usr/local/certificates/acme-v02.api.letsencrypt.org-directory/mail.ezdk.org /usr/local/stalwart/certs nullfs ro 0 0

Additionally, we need to adapt the pf config, to expose the imap and smtp ports.

pass in inet proto tcp from any to any port {ssh,http,https,smtp,smtps,imaps} flags S/SA keep state

and reload the config

pfctl -f /etc/pf.conf

Stalwart Config

There are multiple adjustments and additions we need to make in stalwarts configuration file located at /usr/local/bastille/jails/stalwart/root/usr/local/stalwart/etc/config.toml.

• Remove unwanted (non-essential) listeners
• Bind all listeners to an explicit interface
• Add paths to the certificates

With those modifications, the file looks as follows:

[server.listener.smtp]
bind = "<host-ip>:25"
protocol = "smtp"

[server.listener.submissions]
bind = "<host-ip>:111465"
protocol = "smtp"
tls.implicit = true

[server.listener.imaptls]
bind = "<host-ip>:993"
protocol = "imap"
tls.implicit = true

[server.listener.https]
protocol = "http"
bind = "127.0.0.1:1443"
tls.implicit = true

[server.listener.http]
protocol = "http"
bind = "127.0.0.1:8080"

[storage]
data = "rocksdb"
fts = "rocksdb"
blob = "rocksdb"
lookup = "rocksdb"
directory = "internal"

[store.rocksdb]
type = "rocksdb"
path = "/usr/local/stalwart/data"
compression = "lz4"

[directory.internal]
type = "internal"
store = "rocksdb"

[tracer.log]
type = "log"
level = "info"
path = "/usr/local/stalwart/logs"
prefix = "stalwart.log"
rotate = "daily"
ansi = false
enable = true

[authentication.fallback-admin]
user = "admin"
secret = "<PWHASHL>"

# Certificate from Caddy
server.tls.certificate = "default"
certificate.default.cert = "%{file:/usr/local/stalwart/certs/mail.ezdk.org.crt}%"
certificate.default.default = true
certificate.default.private-key = "%{file:/usr/local/stalwart/certs/mail.ezdk.org.key}%"

Restart stalwart:

bastille service stalwart stalwart restart

Now you can point your browser to stalwart.ezdk.org and log in with admin and the password which was generated during the init setup.

SSO

TODO