Skip to main content
reliability · kubernetes

DaemonSets not running on every eligible node

resource types
1
rule IDs covered
3
severity
high

What does ZopNight detect here?

A DaemonSet should have exactly 1 pod per eligible node. When scheduled is below desired, some nodes are running without it, and because DaemonSets are usually logging, monitoring or security agents, those nodes are silently unobserved rather than merely degraded.

Signal and threshold

How ZopNight evaluates DaemonSets not running on every eligible node.
Field Value
Rule IDsRC-1716 · RC-1816 · RC-1916
Categoryreliability
Severityhigh
Metricnone — pure configuration read
Sourceinternal/rules/k8s/daemonset_degraded.go

The gap is a coverage gap, not a capacity one

For a Deployment, missing replicas means less capacity. For a DaemonSet it means specific nodes lack the agent entirely.

That distinction matters because of what DaemonSets typically are: log shippers, metrics agents, security sensors, CNI components. A node missing its log shipper is not producing fewer logs. It is producing none, and the gap is invisible in the logging backend because absence looks identical to quiet.

The same applies to a security agent: the node is unmonitored, and nothing reports that it is.

Why a node gets skipped

Taints without a matching toleration. The most common cause by far. A tainted GPU or spot node pool is excluded unless the DaemonSet tolerates it, and new node pools frequently arrive with taints the existing DaemonSets were never updated for.

Insufficient allocatable resources. A fully-packed node has nothing left for the DaemonSet pod, which requests resources like any other. DaemonSet pods have no special reservation.

Node affinity or selectors that exclude nodes deliberately. That is legitimate, and worth confirming before treating this as a defect.

Unschedulable nodes, cordoned during maintenance.

Read desiredNumberScheduled, not your node count

The status field already accounts for affinity and node selectors, so it reflects the nodes the DaemonSet is actually meant to cover. Comparing against your total node count will produce false findings on any DaemonSet that deliberately targets a subset.

Diffing node names against DaemonSet pods

Terminal window
kubectl get daemonset -A -o json | jq -r '
.items[] | select(.status.numberReady < .status.desiredNumberScheduled)
| "\(.metadata.namespace)/\(.metadata.name)\t\(.status.numberReady)/\(.status.desiredNumberScheduled)"'

To find which nodes are missing it:

Terminal window
kubectl get nodes -o name | sed 's|node/||' | sort > /tmp/all
kubectl get pods -n <ns> -l <selector> -o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | sort > /tmp/has
comm -23 /tmp/all /tmp/has

Then check that node’s taints. That is the answer roughly nine times in ten.

See it fire on your bill.

Connect an account read-only. The first findings land in minutes.

417 rule families across 353 resource types on 22 platforms. Every threshold, metric, and IAM action is documented on these pages before you grant anything.

417 rule families documented
353 resource types covered
read-only default access level
Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console· Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console·