1st post in a series about self-hosting

I self-host some stuff, using Podman. There are a whole bunch of guides online, and it feels like most of them have consolidated on docker-compose. This is understandable, considering how Docker was the first in the field, and docker compose provides a nice way to do multi-container infrastructure as code-type deployments with just a single file.

When I started experimenting with self-hosting, I transitioned quickly from Docker to Podman. While I didn’t quite understand the implications at the time, the running rootless by default sounded good. The first thing I set up was Nextcloud, running with Docker. Thinking it makes things a whole lot easier, I added my user to the docker group, to allow running Docker commands without sudo as a regular user.

Running the Docker daemon in rootless mode is perfectly possible, but by default the daemon runs as root. This means you can do sudo docker run -v /:/mnt alpine ash and ta-da, you have full access to the entire filesystem. Combine that with the ability to run docker without sudo, you can see where this is going.

So, in comes Podman. There are plenty of technical articles discussing the differences of Podman vs Docker, I’m not going to go there. Suffice to say that using Podman for some years, I’m quite sure most self-hosters would be just fine running everything with rootless Podman. My opinion is if some service requires the extra privileges Docker provides by default, it’s probably a bit dubious anyway.

Podman kube play

The truly powerful feature of Podman is the ability to use Kubernetes-style YAML files to define multi-container deployments. Without going in too much detail, the smallest unit in Kubernetes is a pod, a unit containing one or more containers.

Inside a pod, if there are multiple containers, they can access ports that other containers’ services expose using localhost. This communication is possible without exposing any ports to the host system, in fact without explicitly forwarding ports from the pod, the services are not accessible from the host at all. But the services inside the pod can talk to each other.

This allows for great isolation of different services, and is in my opinion far easier to understand than the docker-compose way of relying on DNS on a dedicated container network.

Podman can read a subset of the Kuberneres YAML spec, and deploy pods using those definitions. The format is slightly more complex than the docker-compose spec, but the upside is that you get your foot between the door for Kubernetes. For most self-hosters Kubernetes is overkill, but learning about it is still a big plus in my opinion.

Quadlet

Quadlet is Red Hat’s component for executing containerized workloads. Using it, systemd acts kind of as an orchestrator like Kubernetes, while Podman is used for the execution of these services.

Quadlet is already quite featureful, and I’ve only scratched the surface. The main reason I use it is it comes bundled with later versions of Podman, at least on the RHELative distros. Keeping extra dependencies to mininum simplifies maintenance.

One thing I don’t like about Quadlet is the need to remember the systemd unit syntax in addition to Podman Kube YAML.

ZFS

If only the licensing complications between Linux and ZFS could be just forgotten. Meanwhile, ZFS is in my non-expert opinion the most robust filesystem for storing data.

In the self-hosted context, I store all persistent data (bind mounts) on ZFS. Podman provides some native integration with ZFS. For rootful stuff I think it could be directly used as the storage driver, but overlayfs is simpler, and the performance is more than enough for the home server scale.

There’s the common misconception of requiring a ton of RAM with ZFS. While extra RAM speeds things up a lot, more so than with other filesystems, it’s not the necessity it was before. What ZFS does provide, is the ability to do backups with minimal downtime while keeping the backups consistent. You can achieve the same with other snapshotting mechanisms, but ZFS’s tooling is really straightforward while having all the bells and whistles. More on backups later on a separate post.

Borg for off-site backups

While ZFS would provide a very effective way to replicate datasets to remote hosts by using zfs send | zfs receive over SSH, I’ve come to the conclusion that having all backups dependent on the filesystem is a bit restrictive. Not to mention I’ve been using Borg from the start, so sticking with it means continuing with the existing setup.

Borg provides deduplicated, incremental, encrypted and compressed backups, and is very good at doing what it does. A remote backup site can be almost any *nix host with SSH and borgbackup installed.

I used to execute the daily and automated remote backups using shell scripts and cron. Those scripts are still around and work, but they were getting a bit elaborate to maintain. I’ve since moved to borgmatic, to get a bit more declarative approach to backups. This required a bit of fiddling and perhaps even abusing borgmatic’s ability to run pre and post scripts, but it works well. Borgmatic also includes backup configuration files in the backups, so they are easier to transfer to other hosts too.

Wrapping up

These are the basic building blocks for what is in my mind a somewhat reliable setup for running important services. For example, my personal Nextcloud hosts calendar and contacts, which replace the respective Google services on Android on my phone. I wouldn’t want that going out of business all of a sudden.

In the next post on the series, I’ll be covering the layout of my setup. The main idea is that it’s quite easy to lift and shift it elsewhere. Some may consider it a useless perk, but when self-hosting is also a hobby, keeping the services as transferrable as possible is a big advantage.