StatefulSets with unready replicas blocking the rollout
What does ZopNight detect here?
A StatefulSet short of its desired replicas is worse than a Deployment in the same state. Pods start in strict ordinal order, so 1 stuck pod halts every pod after it. Ordinal 3 failing means 4 and above never start at all, indefinitely.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1715 · RC-1815 · RC-1915 |
| Category | reliability |
| Severity | high |
| Metric | none — pure configuration read |
| Source | internal/rules/k8s/statefulset_degraded.go |
Where it applies
Ordered startup turns one failure into a stall
This is the property that separates a StatefulSet from a Deployment, and it is why the same symptom is more serious here.
With the default OrderedReady policy, pod N+1 is not created until pod N is Running and Ready.
A StatefulSet of five whose ordinal 2 cannot become ready never creates 3 or 4. It does not
degrade to three-fifths capacity and continue. It stops.
The controller waits indefinitely. There is no timeout, no fallback, and no event that says “blocked”.
Rollouts stall the same way, in reverse
Updates proceed from the highest ordinal downwards, one at a time. A new pod that fails readiness halts the rollout where it stands, leaving the StatefulSet split across two versions.
That is safer than a Deployment charging ahead, but it means a bad image can leave you with ordinals 3 and 4 on the new version and 0 through 2 on the old, potentially for hours, with whatever consistency implications that carries for your application.
Storage is the usual culprit
StatefulSets bind a PersistentVolumeClaim per ordinal, and those claims are where the blocking tends to originate: a PVC stuck Pending, a volume in the wrong availability zone for the node the pod must run on, or a StorageClass quota reached mid-scale-up.
Deleting the pod does not help. It comes back with the same claim, into the same problem.
Parallel policy changes the trade
podManagementPolicy: Parallel starts all pods at once, so one failure no longer blocks the
rest. It also discards the ordering guarantee, which for a database with a defined bootstrap
sequence is exactly the guarantee you wanted.
Change it deliberately, not to clear a finding.
Ready ordinals, then the PVC of the lowest one
kubectl get statefulset -A -o json | jq -r ' .items[] | select((.status.readyReplicas // 0) < .spec.replicas) | "\(.metadata.namespace)/\(.metadata.name)\t\(.status.readyReplicas // 0)/\(.spec.replicas)"'Then find the lowest unready ordinal, the one blocking everything above it, and check its PVC before anything else.