Kubernetes nodes with the DiskPressure condition set

What a full node filesystem does to its pods

The kubelet watches filesystem eviction signals: nodefs.available and nodefs.inodesFree for the node’s root filesystem, plus imagefs and containerfs equivalents when the node keeps images or container layers on a separate filesystem. The node-pressure eviction page lists the default hard thresholds as nodefs.available<10%, imagefs.available<15%, and 5% free inodes on Linux. Meeting any of them makes the kubelet report DiskPressure, and the control plane adds the node.kubernetes.io/disk-pressure taint so the scheduler sends new pods elsewhere.

Reclaiming comes next. On a node with a single filesystem, the kubelet garbage-collects dead pods and containers first, then deletes unused images. If space is still short, it evicts running pods, ranking them by how much disk they use across local volumes, logs and writable layers. Those evictions ignore PodDisruptionBudgets, and under a hard threshold the pod gets no graceful shutdown at all.

Spotting disk pressure across the fleet

kubectl get nodes -o json | jq -r '
  .items[]
  | select(any(.status.conditions[]; .type == "DiskPressure" and .status == "True"))
  | .metadata.name'

Then list what is running on a flagged node, since the heaviest writers are usually pods with busy logs or large emptyDir volumes:

kubectl get pods -A --field-selector spec.nodeName=<node-name> -o wide

The kubelet’s flag is the whole test

ZopNight reads the conditions it collected for each node and fires when a DiskPressure entry has status True. False and Unknown do not count, and a node with no conditions recorded produces nothing. ZopNight applies no usage figure or time window of its own; the kubelet’s thresholds decide. Note that the kubelet holds a condition for eviction-pressure-transition-period, 5 minutes by default, before lowering it, so the finding can outlast the cleanup by a few minutes.

Other node problems have their own findings

Memory and process-ID shortages are reported by Node under memory pressure and Node under PID pressure. A node that has stopped reporting Ready altogether is covered by Node not ready.

Evicted pods rather than dollars

No saving is attached to this high-severity reliability item. The damage is abrupt pod loss on the affected node, plus a node that accepts no new work while the taint is in place.

Relieving the disk

1. Read the condition message in kubectl describe node <node-name> to see which filesystem or inode count tripped the threshold.
2. Find pods writing heavily to logs or emptyDir, and give them ephemeral-storage requests and limits so the scheduler accounts for their disk use.
3. Trim unused images, and enlarge the boot disk in your node group or node pool template so new nodes start with more headroom.
4. If the node cannot recover, cordon it, drain it with kubectl drain --ignore-daemonsets, and replace it.