Docker No Space Left on Device: What's Eating Your Disk

Every house has a garage you can't park in.

Nobody decided to fill it. A box here, a bike there, the treadmill from that one January. Then one day the car stays in the driveway for good.

Your Docker server has one too. It's called /var/lib/docker.

Old images from every deploy. Build cache from every docker build. And container logs that grow a little every second, forever, because nobody told them to stop.

Then one morning your database refuses writes, and the error just says:

no space left on device

Here's how to find what filled it, clean it without deleting anything you need, and make sure it doesn't come back.

Short on time? Run docker system df, then sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail. Most of the time the answer is on one of those two screens. Clean top to bottom in the diagram below, and never prune volumes in a hurry.

Step 1: Confirm it's really the disk

df -h

Look at the filesystem mounted on / (or wherever /var/lib/docker lives). If it's at 100%, keep going.

If it shows free space and you still get the error, skip to the two sneaky causes below.

Step 2: Ask Docker what it's using

docker system df

You get four lines: Images, Containers, Local Volumes and Build Cache, each with a RECLAIMABLE column. That column is what Docker could free without touching anything in use.

For the per-image and per-volume detail:

docker system df -v

Note what docker system df does not show: container logs. Those live elsewhere, and they're the most common culprit.

Step 3: Check the container logs

Docker's default json-file log driver writes everything a container prints to a file, with no size limit:

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -5

A multi-gigabyte file here is your answer. Find which container it belongs to (the folder name is the container ID):

docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep <first-characters-of-id>

Empty it without deleting it:

sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log

Why not rm? Docker still holds the file open. Deleting it hides the name but keeps the space in use until the container restarts. truncate frees it immediately.

Step 4: Clean the safe stuff

Work from the top of this list down. Each line is safe to run on a live server:

# Build cache from past docker builds
docker builder prune
 
# Images not used by any container (they can be pulled again)
docker image prune -a
 
# Stopped containers (check the list first: some might be one-off jobs you still need)
docker container ls -a --filter status=exited
docker container prune

docker system prune -a does all of the above in one go. It does not touch volumes unless you add --volumes.

Step 5: Volumes, carefully

Volumes are where databases, uploads and app data live. Deleting the wrong one is not a cleanup, it's an outage with extra steps.

List what's not attached to any container:

docker volume ls -f dangling=true

On current Docker versions, docker volume prune removes only anonymous unused volumes. Named ones need docker volume prune -a. Before either, look at each name and ask: is there a database in here that a stopped container still needs? If you're not sure, back it up first:

docker run --rm -v <volume>:/data:ro -v "$(pwd)":/backup \
  alpine tar czf /backup/<volume>-$(date +%F).tgz -C /data .

When df shows free space

Two things produce "no space left" on a disk that isn't full:

Out of inodes. Millions of tiny files (cache folders, mail queues, session files) can use up the filesystem's file slots before the bytes run out:

df -i

If IUse% is at 100%, find the folder with the most files and clean that.

Deleted but still open. A process holds a deleted file, so the space isn't released:

sudo lsof +L1

Restarting the process named there frees the space.

While you're at it, check the system journal, which can quietly grow to gigabytes:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Stop it happening again

Cleaning is a one-off. These three changes make it permanent.

1. Cap container logs. Create or edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then sudo systemctl restart docker. The limit applies only to containers created after the change, so recreate yours (docker compose up -d --force-recreate). Per service, you can set the same thing in docker-compose.yml under logging:.

2. Prune on a schedule. A weekly cron job for the safe items:

#!/bin/sh
docker builder prune -f --filter until=168h
docker image prune -a -f --filter until=168h

3. Alert before 100%. A disk alert at 80% turns a 3 AM outage into a Tuesday afternoon task. Any uptime tool that can run a check or read node metrics will do it.

The bottom line

A full Docker disk is almost never mysterious. It's logs, old images, build cache, or a volume nobody remembers creating.

Clean from the safe end. Leave volumes for last. Then cap the logs, so you never have to read this post twice.

P.S. Paste me the output of docker system df and df -h. I'll reply with what I'd clean first, in one message. Free.