Welcome to the Café

Explore the world of Linux with clear, approachable guides. Whether you’re just starting out or looking to sharpen your skills, you’ll find something to brew your interest here.

When systemd‑journald ate my 2 GB SSD: how I capped it to 200 MB in a single config line

I was running a little home‑lab server on a 2 GB SSD. The board was an Orange Pi Zero‑like single‑board computer, and it was juggling a personal web server, a GitLab instance, and a handful of containerised apps. After a month of steady traffic, the SSD hit 100 % on /var. The culprit? systemd‑journald had quietly gobbled up the whole drive.

journald is the default logger on most modern distros. It keeps logs in binary form under /var/log/journal. By default it will keep everything until the filesystem is full, then start deleting the oldest entries. On a tiny SSD that can happen faster than you can say “disk full”.

[Read More]

Why My Home Server Keeps Crashing After Reboot: Fixing the Broken systemd Service for My Docker Compose Deployment

The mystery of the reboot crash

I had a single‑board server running Debian 12 in my living‑room. Every time I rebooted it, the Docker Compose stack that powers my home‑lab services would fail to start, and the machine would hang in a boot loop. The logs were a jumble of “failed to start” messages, but the root cause turned out to be a mis‑configured systemd unit. Below is the exact process I followed to diagnose and fix the issue, with a few trade‑offs and security notes that apply to any Docker‑based deployment on a small server.

[Read More]

Freeing a Full `/var/log` on a Home Lab Server with `journalctl --vacuum-time` and a One‑Line Bash Cleanup

When /var/log fills up

I’ve run a handful of services on my homelab box—Docker containers, an Nginx reverse proxy, a Samba share, and a few cron jobs that ping my home router. It’s a quiet machine most of the time, but every so often the disk starts choking on logs. When /var/log hits the filesystem limit, new log entries stop, services can silently die, and journalctl starts spitting out “no space left on device”. The fix is usually a quick vacuum of the journal and a tidy‑up of the plain‑text logs. Below is a step‑by‑step recipe that works on Debian, Ubuntu, Arch, and other systemd‑based distros.

[Read More]

Why I keep a weekly backup copy of my home directory on an external SSD using rsync and how I test the restore

Why I keep a weekly backup copy of my home directory on an external SSD using rsync

I’m not a “data‑hoarder” in the sense of storing every file I own. I keep a single, up‑to‑date snapshot of my ~/ on a 2 TB SSD that I plug into my workstation once a week. The copy is made with rsync, stored on a separate partition, and I run a quick restore test every month. The routine feels like a small ritual that gives me confidence without eating bandwidth or storage.

[Read More]

Why ssh keeps spawning new processes on every connect and how ControlPersist solves it

Why SSH keeps spawning new processes on every connect

Every time I run ssh host, the client forks a new process that talks to the remote sshd.
On the server side, sshd spawns a child for each accepted connection.
The model is simple and works fine for a handful of sessions, but it starts to feel like a drain when you:

  • run a script that opens dozens of SSH connections in a row,
  • use CI/CD pipelines that hit a remote host repeatedly, or
  • have a home‑lab with many small services that need quick SSH access.

Each fork eats a few kilobytes of RAM and a handful of file descriptors.
If you hit the per‑user process limit (ulimit -u) or the system’s max_user_processes, you’ll see “Too many users” or “Connection refused” errors.
Even on a modest machine, the cumulative CPU cost of negotiating the SSH handshake over and over can add up.

[Read More]

Repairing a broken initramfs that locked my Raspberry Pi in emergency mode after a kernel update

A Raspberry Pi stuck in emergency mode after a kernel update?

I hit this one last week while upgrading to the 6.6‑raspi kernel. The Pi rebooted, the splash screen faded, and I was greeted with the dreaded “Emergency mode” prompt. I had no idea what was wrong until I dug into the initramfs. Below is a step‑by‑step guide I used to recover, plus a few lessons that apply to any Debian‑based system.

[Read More]

How I added ionice ‑c3 to my nightly rsync backup and finally stopped my laptop from overheating

Nightly rsync and the heat problem

When I first set up a nightly cron job to copy my home directory to an external SSD, the laptop’s fan stayed at full blast all night. The CPU hit 90 °C, and the battery life dropped noticeably. I blamed the SSD’s sluggish write speed, but the real culprit turned out to be the I/O scheduler giving the backup process too much priority. Adding ionice -c3 (the idle class) to the rsync command brought the temperature back to sane levels without sacrificing backup reliability.

[Read More]

Running a nightly backup script after logout with systemd user units – no nohup needed

Running a nightly backup script after logout with systemd user units – no nohup needed

I set up a little home‑lab server last year and wanted a way to snap my home directory every night without tying the job to a terminal. The old nohup trick works, but it leaves orphaned processes and you end up chasing logs in the dark. Systemd user units give a cleaner, more reliable solution that plays nicely with the rest of the system.

[Read More]

I lost SSH access to my home server after accidentally setting /home/user to 777 – how I restored it with a live USB

I hit a wall yesterday when I accidentally set /home/user to 777 and lost SSH access.
The server was a 2026‑era Debian‑based homebox running systemd‑managed services.
SSH refuses to log in if any file under the user’s home directory is writable by others, so the key‑based login failed immediately.
The only way to fix it was to boot from a live USB, mount the root filesystem, and restore the correct ownership and permissions.

[Read More]