Linux Café

When /boot runs out of space and the machine drops into emergency mode

A few minutes after a kernel update, you power on the server and the console shows:

System boot failed.
Emergency mode - type "journalctl -xb" to view logs.

The root filesystem is mounted read‑only, the initramfs cannot be loaded, and the machine refuses to start. The culprit is almost always a full /boot partition. In this post I walk through the exact steps to diagnose, free space, and reboot safely. I’ll cover the most common distributions, show how to keep the system lean, and touch on security best‑practices that come with kernel housekeeping.

The Flicker Problem

I bought a 2022 Dell XPS 13 a few months ago and, like most people, I wanted to use an external 4K monitor via the single USB‑C port. The laptop ships with a Thunderbolt 4 controller that should handle 60 Hz 4K output without issue. On my Fedora 38 machine the monitor flickers when I connect it, and the flicker is worse on Wayland than on Xorg. The screen stutters every 1–2 seconds because the refresh rate drops in a periodic fashion. I spent a week trying to pin down the cause before finding a reliable fix.

USB plug‑in spam is a classic annoyance on modern Linux boxes. Every time you drop a keyboard, a mouse, or even a USB‑SD card in, the kernel goes into a burst‑mode and writes a wall of messages to journalctl. On a busy server or a home‑lab machine that logs everything, those lines can quickly fill the journal, slow down log‑rotation, and make it hard to spot real problems.

Below is a hands‑on guide to quiet those messages without losing the useful diagnostics you need. The techniques work on recent systemd‑based distributions (Debian 12, Ubuntu 24.04, Arch 2026, etc.) and are safe for a security‑aware setup because they only suppress noise, not real alerts.

I was running a little home‑lab server on a 2 GB SSD. The board was an Orange Pi Zero‑like single‑board computer, and it was juggling a personal web server, a GitLab instance, and a handful of containerised apps. After a month of steady traffic, the SSD hit 100 % on /var. The culprit? systemd‑journald had quietly gobbled up the whole drive.

journald is the default logger on most modern distros. It keeps logs in binary form under /var/log/journal. By default it will keep everything until the filesystem is full, then start deleting the oldest entries. On a tiny SSD that can happen faster than you can say “disk full”.

I’m ready to help polish the post, but I’ll need the article text first. Could you paste it here?

The mystery of the reboot crash

I had a single‑board server running Debian 12 in my living‑room. Every time I rebooted it, the Docker Compose stack that powers my home‑lab services would fail to start, and the machine would hang in a boot loop. The logs were a jumble of “failed to start” messages, but the root cause turned out to be a mis‑configured systemd unit. Below is the exact process I followed to diagnose and fix the issue, with a few trade‑offs and security notes that apply to any Docker‑based deployment on a small server.

When /var/log fills up

I’ve run a handful of services on my homelab box—Docker containers, an Nginx reverse proxy, a Samba share, and a few cron jobs that ping my home router. It’s a quiet machine most of the time, but every so often the disk starts choking on logs. When /var/log hits the filesystem limit, new log entries stop, services can silently die, and journalctl starts spitting out “no space left on device”. The fix is usually a quick vacuum of the journal and a tidy‑up of the plain‑text logs. Below is a step‑by‑step recipe that works on Debian, Ubuntu, Arch, and other systemd‑based distros.

Why I keep a weekly backup copy of my home directory on an external SSD using rsync

I’m not a “data‑hoarder” in the sense of storing every file I own. I keep a single, up‑to‑date snapshot of my ~/ on a 2 TB SSD that I plug into my workstation once a week. The copy is made with rsync, stored on a separate partition, and I run a quick restore test every month. The routine feels like a small ritual that gives me confidence without eating bandwidth or storage.

Why SSH keeps spawning new processes on every connect

Every time I run ssh host, the client forks a new process that talks to the remote sshd.
On the server side, sshd spawns a child for each accepted connection.
The model is simple and works fine for a handful of sessions, but it starts to feel like a drain when you:

• run a script that opens dozens of SSH connections in a row,
• use CI/CD pipelines that hit a remote host repeatedly, or
• have a home‑lab with many small services that need quick SSH access.

Each fork eats a few kilobytes of RAM and a handful of file descriptors.
If you hit the per‑user process limit (ulimit -u) or the system’s max_user_processes, you’ll see “Too many users” or “Connection refused” errors.
Even on a modest machine, the cumulative CPU cost of negotiating the SSH handshake over and over can add up.

A Raspberry Pi stuck in emergency mode after a kernel update?

I hit this one last week while upgrading to the 6.6‑raspi kernel. The Pi rebooted, the splash screen faded, and I was greeted with the dreaded “Emergency mode” prompt. I had no idea what was wrong until I dug into the initramfs. Below is a step‑by‑step guide I used to recover, plus a few lessons that apply to any Debian‑based system.