PersistentVolumeClaims in the Lost phase after their volume disappeared
What does ZopNight detect here?
ZopNight flags a Kubernetes PersistentVolumeClaim on EKS, GKE or AKS whose `status.phase` is `Lost`. Kubernetes uses that phase when a claim was bound to a PersistentVolume that no longer exists, and the API reference states that all data on it was lost, so anything mounting the claim has no storage behind it.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1779 · RC-1879 · RC-1979 |
| Category | orphan |
| Severity | medium |
| Metric | status.phase |
| Threshold | equals Lost |
| Source | ZopNight |
| Permissions used | list persistentvolumeclaims · list persistentvolumes |
Where it applies
How a claim ends up pointing at nothing
A claim has three phases in the
PersistentVolumeClaim API reference:
Pending, Bound and Lost. Lost is the unusual one. It means the claim was bound to a
PersistentVolume and that volume object no longer exists, and the reference adds that all data on
it was lost.
It points to the volume object having been removed outside the normal claim lifecycle, for
example by a partial restore of cluster objects. Removing the Kubernetes object does not always
remove the cloud disk: with a Retain reclaim policy, the
PersistentVolumes documentation
notes that the storage asset in the external infrastructure still exists after the PV is deleted.
So a Lost claim can sit beside an EBS volume, Persistent Disk or Azure managed disk that still
holds the data and is still billed, with nothing in the cluster connecting the two.
Finding lost claims and the volume names they expected
kubectl get pvc -A -o json | jq -r ' .items[] | select(.status.phase == "Lost") | "\(.metadata.namespace)/\(.metadata.name) expectedVolume=\(.spec.volumeName)"'Check whether that volume object exists at all with kubectl get pv <volume-name>. If it does
not, search your cloud account for a disk carrying the same name or the claim’s name in its
tags or description.
One phase value triggers it
ZopNight reads the claim’s phase as it was collected and fires when it equals Lost, ignoring
case. There is no age gate and no metric window, because unlike Pending, this phase is never a
normal step in a claim’s life. The finding clears as soon as the claim is deleted or rebound.
What the rule does not attempt
ZopNight does not look for the orphaned cloud disk, estimate its size or price it, so the finding
has no savings figure. It also does not identify the pods that reference the claim. Claims that
are Pending rather than Lost are reported by
Unbound PVC, and volumes left behind by a
deleted claim appear under
Released PersistentVolume.
Why it is filed as an orphan
The claim itself costs nothing; it is a dangling reference. The real exposure is the workload that depends on it and, often, a disk outside the cluster that nobody knows is still there.
Recovering or cleaning up
- Find the pods that mount the claim with
kubectl get pods -n <namespace> -o json | jq -r '.items[] | select(tostring | contains("<claim-name>")) | .metadata.name'. - Look for the backing disk in the cloud console or CLI. If it exists and the data matters, create a new PersistentVolume that points at the same disk, as the Kubernetes docs describe for reusing a retained storage asset.
- If the data is recoverable only from a snapshot or backup, restore it to a new disk and bind a fresh claim.
- If the data is not needed, delete the claim and let the workload provision new storage.