Problem

Linux servers can become unresponsive when the disk fills up. The OS cannot write logs, swap, or temporary files, causing services to crash and SSH access to be lost. This typically requires a hard stop/start cycle to recover.

Common Causes of Disk Full

1. Systemd Journal Logs (Highest Risk)

/var/log/journal can grow to several gigabytes if uncapped. By default, systemd-journald has no size limit and accumulates logs indefinitely.

2. Docker Container Logs

Docker default json-file log driver writes container stdout/stderr to log files with no size limit. A single busy container can generate gigabytes of logs.

3. Unused Docker Images

Every docker pull or docker build leaves old image layers on disk. These are never automatically removed.

4. Package Manager Cache

Yum/apt caches downloaded packages which grow with every update or install.

Prevention Steps

Step 1: Cap Systemd Journal Size

Edit /etc/systemd/journald.conf and add:

SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=30day

Apply the changes:

sudo systemctl restart systemd-journald
sudo journalctl --vacuum-size=500M

Step 2: Configure Docker Log Rotation

Create or edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

This caps each container log at 150 MB (3 x 50 MB). Restart Docker and recreate containers to apply.

Step 3: Set Up Daily Cleanup Cron

Create /etc/cron.daily/disk-cleanup:

#!/bin/bash
journalctl --vacuum-size=500M
docker image prune -f
find /var/log -name "*.gz" -mtime +30 -delete
find /var/log -name "*-20*" -mtime +30 -delete
yum clean all

Make it executable: sudo chmod +x /etc/cron.daily/disk-cleanup

Step 4: Clean Package Manager Cache

sudo yum clean all

Ensure keepcache=0 in /etc/yum.conf to prevent future accumulation.

Step 5: Set Up Disk Monitoring

Install CloudWatch Agent and create an alarm at 80% disk usage for early warning before the disk fills completely.

Disk Budget Planning

• System journal: 500 MB max

• Docker container logs: 150 MB per container

• Docker images (active): ~4 GB

• Audit logs: 40 MB

• System logs (logrotate): 50 MB

• Package cache: 100 MB

Recovery If Disk Already Full

1. Try soft reboot from cloud console

2. If OS is frozen, perform hard stop/start cycle

3. Once recovered, immediately run cleanup

4. Apply all prevention steps above

5. Consider increasing the volume size for extra safety margin

Key Takeaways

• Always set size limits on journal logs and Docker container logs

• Automate cleanup with a daily cron job

• Monitor disk usage with alerts at 80% threshold

• Prune unused Docker images regularly

• An ounce of prevention saves hours of downtime recovery