The goal of this guide is to document its namesake, as well as the reasoning and common caveats behind it. It follows the process of how I went about configuring a preinstalled Debian image to run a quark server on a Raspberry Pi.

Headless setup
Server setup
Hardening

Headless setup

Obtain the initial image

Download a preinstalled image from raspi.debian.net. Be aware that this requires you to put your trust in just one person who builds the images (who happens to be a Debian developer). Working from preinstalled images is common in this scenario. The reason for this mainly lies in only having one bootable SD card slot, which means you would have to plug in another SD card reader to install to (then swap them around once it’s ready to boot), or an SSD/HDD the pi can reliably power (in which case the hassle is in doing the firmware update which makes booting possible from the microUSB port).

How to find your SD card’s device

BEWARE, for all who don’t verify their device IDs before using it as an output of dd, are subject to the whims of fate. Make sure that when you pass /dev/sdX to any command that writes to it, it is exactly the device you believe it to be, so that you don’t accidentally wipe your system.

To find the name of your SD card, you can either list your block devices with lsblk or devices with ls /dev, before and after your plug in your SD card. Compare their outputs, and choose the new name in its topmost/simplest form (not any partitions, but the device itself).

Get a shell into the installed environment

One method is to run the OS on the raspberry itself. In this case, we’ll need to copy the image to bootable media. As always, be careful when using /dev/sdX as the destination of a write operation. A safer way is to use disk IDs directly.

# dd if=<preinstalled>.img of=/dev/sdX

If you have a .img.xz compressed archive, you can either extract it to an image with xz --decompress <your-image>.img.xz, or you can pipe the output of xzcat into dd, like so.

# xzcat <your-image>.img.xz | sudo dd of=/dev/<your-device> bs=64k oflag=dsync status=progress

Another method that avoids booting the target hardware is to chroot into the image using an emulator such as qemu, because you might be using a host system with a different architecture from arm. In my case, armv9 emulation for the RPi4 seems to be in the works still, but this could be feasible later on.

Grab a cup of your beverage of choice while this completes, and pray to your system administrator that you entered the correct SCSI drive name. Contemplate struggle, futility and fortune, and how computers will eventually ruthlessly dominate all reality. Copying will be done aaany day now. After completion, plug in your device to your target hardware, hook up your peripherals, and boot it.

Set up the hostname

Set the hostname by editing /etc/hostname. For sudo to be able to resolve the host as well, the same name should be listed in /etc/hosts after the loopback address 127.0.0.1.

Setting up the root user, creating a new user

Controversially, the preinstalled images ships with an empty user password, which allows you to log in locally (not through the network).

Let’s change the root user’s password using # passwd root.

Also, we should create a new user with # adduser <username>. If you log in as this user before you get to the point of setting it up as a sudoer (for example, through ssh, since root logins are disabled through it by default), you can become root by su root.

If commands are not being found while you are root, add the following to /root/.bashrc.

export PATH=/usr/local/sbin:/usr/sbin:/sbin:$PATH

Update apt package cache initially

# apt update

If security is any concern, it is a good idea to update your system regularly with # sudo apt update and # sudo apt upgrade.

Changing the keyboard settings

To configure the keyboard for the Linux kernel and X, you’ll need the keyboard-configuration and console-setup packages.

Change the keyboard configuration with # dpkg-reconfigure keyboard-configuration, or edit /etc/defaults/keyboard manually, then use # service keyboard-setup restart to apply the changes.

If you wish to change the console font, use # dpkg-reconfigure console-setup.

Change timezone

If you happen to not be located in UTC+0, list the available timezones with timedatectl list-timezones, scrolling with F and B, and once you find yours, set it using # timedatectl set-timezone <region/place>.

Setting up SSH

In this case, the preinstalled image’s configuration for SSH will work out of the box, and we don’t have to bother with installing or enabling it.

Running a headless setup

For many purposes, running the device headless (without a screen and keyboard) is necessary, however, it is easier the perform the setup with those attached. First, install the package dhcpcd.

Connecting to a wired ethernet network is straight-forward, and the device will be assigned an IP address by the DHCP server automatically. To connect to the device using this lease address, we’ll have to find out the specific one it was assigned using ip addr. Look for an interface that is BROADCAST, MULTICAST and UP. It’s IP address on the local network will be on the second row after “inet”. At this point, you can log into the device using ssh <username>@<lease-addr>.

To find out the lease IP address from the outside, you can also try pinging addresses on the local network using ping 192.168.0.xx, and attempting to log in once you get a response.

The DHCP lease address may change as the network changes & the device loses & gains connection, so to access it over the network headlessly it would be better if it had a fixed IP address on the local network. This can be set up through the router, but it is easier to configure the device to try and lease the same address from the DHCP server every time, and hope that it won’t be leased to another device.

Edit /etc/dhcpcd.conf file to specify a static profile. We’ll also configure it to use the same DNS servers every time, and set it to point to Big G Megacorp’s servers. Add the following to the bottom of the file.

interface eth0
inform 192.168.0.222
static routers=192.168.0.1
static domain_name_servers=8.8.8.8 4.4.4.4

If everything went well, we can now reboot and ssh into the device using the informed address.

Gaining root privileges using sudo

Install the package sudo. There are multiple ways of adding a user to be a sudoer, we’ll do it through sudoer files. Create the directory containing information about how users can sudo using mkdir /etc/sudoers.d. Then, create a file called <username> under it, and add to it the contents <username> ALL=(ALL) PASSWD:ALL.

Note that we specify PASSWD:ALL because it’s a good idea on live systems to add an additional layer of protection against attackers who could only obtain a shell to your user, but have yet to gain root privilages. For configuring a locked system, you might want to specify NOPASSWD:ALL temporarily, for the sake of convenience.

Taking a breath and a backup

Now that we have set up a user with root access with which we can ssh onto the machine, it might be a good idea to clone this configuration back into a .img file using dd the other way around, before doing more complex configuration.

# dd if=/dev/sdx | gzip > <backup>.img.gz

We can recover our image later using the following.

# gunzip --stdout <backup>.img.gz | dd of=/dev/sdx

A hat trick that lets you peek inside backups

Let’s say you’re irresponsible like me, and completely forgot your username that you use to log into your device. More realistically, let’s just say that you need to peek inside a backup .img file for some more plausible reason.

First, we’ll check the partition table of the image using a loopback device.

# losetup /dev/loop0 <backup>.img
# fdisk -l /dev/loop0
# losetup -d /dev/loop0

From the output of fdisk, we’ll multiply the sector size (in bytes) with the starting offset (in sectors) of the partition we want to mount to get the byte offset of the partition. We’ll use this as an offset to mount the image (along with the loop option). Also, create a temporary directory to serve as a mount point.

# mount -o loop,offset=<sectorsize*partstart> <backup>.img <mountpoint>
# umount <mountpoint>

While the file was mounted, I just had to check my /etc/passwd, and lo, I could avoid redoing my configuration from scratch. Note that this whole ordeal is usually done with a tool called kpartx, so you could check it out if the method here does not work for you.

Server setup

Quark, a static site server

Install git and clone quark’s repository with git clone https://git.suckless.org/quark.

Install make and a C compiler of your choice, such as gcc or clang. Try building quark with make all. If successful, you can install it into the appropriate paths using make install. Be sure to check out man quark to see what the parameters do.

Daemonizing a process using a systemd unit

Add the following to /etc/systemd/system/<your-daemon>.service, filling in your own parameters.

[Unit]
Description=My daemon

[Service]
ExecStart=/usr/bin/mydaemon
Restart=on-failure

[Install]
WantedBy=multi-user.target

In my case, to start my quark server, ExecStart was /usr/bin/quark -h 0.0.0.0 -p 80 -d /home/<my-user>/<website-dir>.

To start the daemon, run systemctl start <your-daemon>.service.

To start the daemon at boot, enable it using systemctl enable <your-daemon>.service.

Dynamic DNS a la duck

First, find out if cron and curl are installed on your system by running ps -ef | grep cr and curl, or apt list --installed, and install them if that wasn’t the case.

Let’s make a directory to put our stuff into called duckdns and a file in it called duck.sh.

Copy the following text into it, and change the token and domain to be the one you were assigned.

echo url="https://www.duckdns.org/update?domains=exampledomain&token=a7c4d0ad-114e-40ef-ba1d-d217904a50f2&ip=" | curl -k -o ~/duckdns/duck.log -K -

We specify a hyphen after the -K option so that curl will read its command line options from stdin.

You can also pass a comma separated list of domains (no spaces allowed). The “ip” portion is for hardcoding an address if you need to, but usually you wouldn’t want to do this, they will detect your remote IP address.

Next, add execution privileges for the user to the file with either chmod u+x or more concretely, chmod 700.

Next, we want to edit the tables used to drive the cron daemon using crontab -e, where the argument specifies that we want to edit the current crontab with the editor specified by the VISUAL or EDITOR environment variables, and to automatically install the modified crontab after exiting that editor. Note that if you are running with su, it may confuse crontab, and it is best to specify the user whose crontab should be edited with the -u option.

This is where I have to recommend you to read up on cron if you are unfamiliar with it, as it’s out of scope here.

Add the following entry to the bottom of your user crontab.

*/5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1

Test the script manually, and check the response in ~/duckdns/duck.log.

Hardening

Disable wireless interfaces

If you are getting network connectivity to your RPi via a cable, you should consider disabling unused wireless functionality. While this isn’t a perfect solution to prevent sideways movement into a wireless network (if an attacker has root, re-enabling the wireless features is trivial), it does allow the creation of a high-quality alert that something unexpected is happening on your system.

There are multiple ways and layers to disable these, we’ll blacklist the related kernel modules & disable the related systemd services here.

Disable kernel modules

All kernel modules are listed in the /lib/modules directory system in .ko (kernel object) files by default. In my case, the kernel modules were a path such as /lib/modules/<kernel-version>-<arch>/kernel. Running find on this path, and grep on the results with the terms wireless or bluetooth. Information about modules can be gathered using modinfo <module>.

Disable HCI over UART (bluetooth transport) and the broadcom bluetooth driver by adding the following to /etc/modprobe.d/raspi-blacklist.conf.

# Disable bluetooth
blacklist hci_uart
blacklist btbcm

Disable Wi-Fi by blacklisting the broadcom driver modules.

# Disable wi-fi
blacklist brcmfmac
blacklist brcmutil

To check that the modules were not loaded after booting again, use modprobe --showconfig | grep blacklist.

SSH configuration

When changing SSH configurations, you should ensure that you either don’t test them on a live system, and that you have another way to access your machine, or you could end up locking yourself out otherwise.

If you are unacquainted with the basics of opsec, I strongly recommend that you read up on this section on your own time. In my case, I didn’t realize at first how bad of an idea is leaving a password-authenticated SSH port open on the internet.

There are some mitigations against brute force attacks, such as:

Using a port different than 22 to eliminate most unsophisticated worms,
Restricting access to known IP addresses,
Rate limiting SSH sessions using iptables,
Using an IPS (intrusion prevention system) such as fail2ban,
Disabling root logins and restricting access of the users that can log in this way.

However, all of these are susceptible to a man-in-the-middle attack (or some kind of DNS hijacking), if you don’t check the SSH trust-on-first-use-style ECDSA key fingerprint carefully every time you first try to connect from another computer. Another problem is that as long as password authentication is enabled, the knocking will forever continue, leeching resources such as bandwidth, disk writes & space. This is why disabling password authentication completely is recommended, switching to keypairs instead.

An SSH key passphrase is a secondary form of security that gives you a little time when your keys are stolen, so this option is still not immune to brute forcing in that case. If your RSA key has a strong passphrase, it might take your attacker a few hours to guess by brute force. That extra time should be enough to log in to any computers you have an account on, delete your old key from the ~/.ssh/authorized_keys file, and add a new key. This shifts the focus of security to the machines you keep your keys on. Note, however, that if your keys are stolen and you are unaware of it, the playing field is essentially reduced to brute-force (or dictionary) attacks again.

Configuration file

Edit /etc/ssh/sshd_config.d/<something>.conf, adding the following configuration changes, if you deem them appropriate. Restart the ssh service to apply the configuration using # systemctl restart ssh.

An empty password would not be very difficult to guess.

PermitEmptyPasswords no

Nobody should user the server as root (ideally), so nobody should log in as root via ssh.

PermitRootLogin no

Disable SSHv1 in favor of SSHv2

SSHv2 is usually the default, but it is worth making sure.

Protocol 2

Disable X11 forwarding

The security concern here is that X11 forwarding opens a channel from the server to the client. In an X11 session, the server can send specific X11 commands to the client, which can be dangerous if the server is compromised.

X11Forwarding no

Disable PAM

If you don’t need PAM (pluggable authentication module) authentication, disable it.

UsePAM no

Allow only a specific group of users to log in by creating a new group using # groupadd ssh-users, adding them to it with # usermod -a|--append -G|--groups ssh-users <username>. The supplementary group and your user in it should be visible in /etc/group.

AllowGroups ssh-users

Public key authentication

On the host system(s) where you will be connecting to the RPi system from via SSH you will need to generate a private and public keypair, and then add the public key of the host system to the RPi system. Generate a key using ssh-keygen, a tool that “generates, manages and converts authentication keys for ssh”. If # ssh-keygen is invoked without any arguments, it will generate an RSA key of 3072 bits with 16 rounds of key derivation when saving the private key. You’ll also need to use a passphrase to protect the key. Look into diceware for an easy way of generating easy-to-remember, cryptographically strong passwords.

Now add the key to the machine’s authorized keys using # ssh-copy-id -i <pubkey> <username>@<hostname>, where <pubkey> is the name of the public key on the host system, and <hostname> is where you originally ssh’d onto the machine, which in my case, was the 192.168.0.x local address.

To enable public key authentication, add the following to your ssh configuration file.

PubkeyAuthentication yes

If the key was added successfully to the machine (should be visible in ~/ssh/authorized_keys), test it out by attempting to log in using it by # ssh -i ~/<pubkey>/ <username>@<hostname>. If it all seems to work, disable password authentication by adding the following to your configuration.

PasswordAuthentication no

Changing the default port

Many suggest changing the default ssh port, but for my use case, I think I’ll just forward whatever port on my router to 22 if I ever need to log into the machine remotely. If you do, it’s a good way to reduce traffic from low-effort knocking. Here, the only complication is opening up that port in the firewall, and possibly reconfiguring fail2ban as well.

Firewall configuration

We’ll be using the uncomplicated firewall, so install the ufw package, which provides an easy interface for managing a netfilter firewall, iptables being the backend in this case. Throughout this section, you’ll be able to view the status of the firewall using # ufw status verbose.

Change the default policy to:

Deny incoming traffic by default with ufw default deny incoming.
Allow outgoing traffic by default with ufw default allow outgoing.

It is also possible to let the sender know that their traffic is being rejected instead of ignoring it, you can use reject instead of deny in the rule for this.

Rate-limit ssh traffic to deny more than 6 connections within 30 seconds by default (for a single IP address) by # ufw limit ssh.

Enable HTTP/S connectivity by # ufw allow http and # ufw allow https.

Enable per-rule logging using # ufw logging <level>. To log all blocked packets not matching the defined policy (with rate limiting), as well as packets matching logged rules, set <level> to low.

Finally, enable the firewall using # ufw enable. Reload the firewall using # ufw reload.

It is important to understand that the configuration of ufw logging (especially beyond the medium log level) may decrease the life of your RPi SD card, but may be needed if adding other security tools such as psad that require detailed logs to function. You can view your logs in /var/log/ufw.log.

Allow SSH only from the local network

Set the firewall to allow limited traffic from the local network on port 22 using # ufw limit from 192.168.0.0/24 to any port 22 proto tcp.

Then, we’ll have to delete the previous rule that allowed limited traffic from anywhere to port 22. List the firewall rules using # ufw status numbered, pick the corresponding rule, and delete it with # ufw delete <rule>.

Fail2ban configuration

Install the package fail2ban, which provides a set of server and client programs to limit brute force authentication attempts. The additional advantage beyond a rate-limiter is that fail2ban will count matches against rules, and ban IPs for longer times, while the default rate-limiting in ufw still allows roughly 600 retries per hour, which may be fine when requests are only coming from a few addresses every ~5 seconds, but a larger botnet could still swamp useful traffic.

The file /etc/fail2ban/jail.conf contains jail declarations. This file is read first, and jail.local second, which will contain our specific override settings. Inspect the log file /var/log/fail2ban.log.

Start/stop the fail2ban service using systemctl. View the status of the client with # fail2ban-client status. Note how sshd jail is enabled by default. Each time you make changes to the fail2ban configuration, restart the service using systemctl to apply them.

Hardening your specific application, and liftoff

Once you made sure that the general configuration of the server is reasonably safe for your purposes, it might be time to look into the security of whatever service it is going to provide. If you are serving a static website on port 80, that’s fairly simple, but if you’re running something more complicated, such as a game server, it might be time to look into the security of that.

I hope you’ve found this resource useful, and that you will configure your system for whatever cool idea you have in mind next.

created 2023-01-06

modified 2023-01-06