Orphan PVCs and released PVs

ORPHAN PATTERNS:
  1. PVC exists but no pod uses it          (most common)
  2. PVC bound to PV but pod deleted         (subtle waste)
  3. PV in "Released" state, no claim        (reclaim policy = Retain)
  4. PV in "Failed" state                    (broken; still billed)
  5. PVCs in deleted namespaces (cluster bug)

DEPLOYMENT DELETED:
  Pod deleted → PVC sometimes preserved (depending on
                 statefulSet.persistentVolumeClaimRetentionPolicy)
  PV depending on reclaim policy:
    - Delete: PV deleted automatically (good!)
    - Retain: PV stays (orphan risk)

MISCONFIGURATIONS:
  PVCs without proper deletion handling in IaC
  Multi-cluster orphans (same storage referenced from multiple clusters)
  Manual kubectl delete on pods without addressing PVCs

ABANDONED workloads:
  Engineer left; storage continues to bill
  Project ended; cleanup never happened

NAMESPACE DELETION INCOMPLETE:
  Sometimes K8s leaves PVs after namespace deletion
  Especially with finalizers / hooks

STORAGE COSTS (AWS EBS gp3 reference):
  gp3 baseline:       $0.08/GB-month
  gp3 provisioned IOPS (>3000): adds significant cost
  io2 (provisioned):    $0.125/GB-month + IOPS cost

EXAMPLES:
  Orphan PV 100 GB gp3:     $8/month
  Orphan PV 500 GB gp3:     $40/month
  Orphan PV 1 TB gp3:       $80/month
  Orphan PV 1 TB io2:        $125+/month
  Orphan PV 1 TB w/ provisioned IOPS: $200-500/month

ACROSS MANY ORPHANS:
  Mid-size cluster typical: 30-60 orphans
  Total orphan storage: 5-30 TB
  Monthly waste: $400-$3,000+

ANNUAL: meaningful even for cost-disciplined orgs

ORPHAN PVCs (Bound, but no pod mounts them: these bill):
  $ kubectl get pvc --all-namespaces
  Cross-reference with kubectl get pods to find unmounted ones

  Or: scripted check (claims pods mount vs Bound PVCs)
  $ kubectl get pods -A -o json | jq -r '
      .items[] | .metadata.namespace as $ns |
      .spec.volumes[]? | select(.persistentVolumeClaim) |
      $ns + "/" + .persistentVolumeClaim.claimName' | sort -u > mounted.txt
  $ kubectl get pvc -A -o json | jq -r '
      .items[] | select(.status.phase == "Bound") |
      .metadata.namespace + "/" + .metadata.name' | sort -u > bound.txt
  $ comm -13 mounted.txt bound.txt   # Bound PVCs no pod mounts

PV STATES:
  $ kubectl get pv

  Look for:
    STATUS=Available    - never bound; check if intended
    STATUS=Released      - was bound, claim deleted, still here
                          (Retain policy)
    STATUS=Failed        - broken; investigate + delete
    STATUS=Bound         - normal; PVC actively using

  Suspect: Released + Failed + old Available

AGE FILTER:
  $ kubectl get pv -o json | jq -r '
      .items[] | select(.status.phase == "Released") |
      .metadata.name + " " + .metadata.creationTimestamp'

NAMESPACE-LEVEL CHECK:
  $ kubectl get pvc -n <namespace>
  If namespace was decommissioned but PVCs remain: clean up

ZOPNIGHT K8S ORPHAN RULES (EKS ids; add 100 for GKE, 200 for AKS):

  RC-1740  Unbound PVC                               medium
    Flags PVCs stuck in Pending (never bound to a volume)
    No disk exists yet, so no storage bill: a
    misconfiguration signal, not a cost one

  RC-1741  Released PersistentVolume                 medium
    Flags PVs left in Released state
    Usually a "Retain" reclaim policy whose PVC was deleted

  RC-1742  Service With No Endpoints                 medium
    Not storage, but the same orphan category: a Service
    whose selector matches nothing still holds a cloud
    load balancer

FOR EACH FLAGGED ORPHAN:

  1. CHECK CREATION DATE
     New (<7 days): may be in-progress; defer
     Old (>30 days): likely true orphan

  2. CHECK TAGS / LABELS
     Find owner if labels include team/owner
     Find purpose from name conventions

  3. CHECK ANY RECENT ACTIVITY
     EBS/disk recent IO? (probably not for orphans)

  4. CHECK NAMESPACE STATUS
     If namespace deleted: definite orphan
     If namespace exists: check for related workloads

  5. DECISION:
     SNAPSHOT + DELETE (safe; preserves recovery option)
     DIRECT DELETE (if team confirms no recovery needed)
     LEAVE (if intentional reserve; tag it!)

CONFIRMATION:
  Reach out to identifiable owner with 7-day notice
  Default action after 7 days: snapshot + delete

OPTION A: SNAPSHOT + DELETE (safest; recommended default)
  1. Snapshot the PV (cheap):
     aws ec2 create-snapshot --volume-id vol-xxx ...
  2. Delete the PVC:
     kubectl delete pvc <name> -n <namespace>
  3. PV auto-deletes (if reclaim policy = Delete)
     OR delete PV manually (if Retain)

  Cost: small (snapshot is cheaper than active storage)
  Recovery: snapshot available for 30 days+ as needed

OPTION B: DIRECT DELETE
  1. Confirm with team (Slack / email)
  2. Delete PVC + PV

  Cost: $0 (no snapshot)
  Recovery: none
  Use only when: team explicitly confirmed data not needed

OPTION C: LEAVE AS-IS WITH OWNER
  1. Identify owner; tag with intentional-reserve label
  2. Set 90-day reminder to re-evaluate

  Cost: ongoing
  Use only when: genuine intentional reserve (rare)

PV.spec.persistentVolumeReclaimPolicy:

DELETE
  When PVC is deleted: PV is automatically deleted
  Storage released; no orphan
  Risk: data lost if PVC deleted accidentally

RETAIN
  When PVC is deleted: PV stays in "Released" state
  Data preserved but billed forever
  Risk: orphan accumulation

RECYCLE (deprecated; don't use)

DEFAULT for new PVCs (per StorageClass):
  Most cloud providers default to Delete (good)
  Custom StorageClasses may default to Retain (risk)

POLICY RECOMMENDATION:
  DELETE for ephemeral and dev workloads
  RETAIN for critical production data with manual review

  Set explicitly per StorageClass:
    storageClassName: standard
    reclaimPolicy: Delete

PATTERN A: MONTHLY CLEANUP
  Run script identifying orphans >30 days
  Notify owners (Slack DM with 7-day notice)
  Snapshot + delete after grace period
  Cost: 1 hour/month of effort

PATTERN B: AUTOMATION
  PVC deletion triggers PV deletion (reclaim policy Delete)
  No manual cleanup needed
  Prevents orphan accumulation at the source

PATTERN C: NAMESPACE-BASED CLEANUP
  Policy: when a namespace is deleted, all PVCs/PVs cleaned up
  Enforce via cluster policy (Kyverno, OPA Gatekeeper, etc.)

PATTERN D: TAGGING POLICY
  All PVCs MUST have owner + ttl label
  Auto-cleanup of PVCs past ttl
  Prevents orphan accumulation at provision time

PITFALL                              MITIGATION
──────────────────────────────────────────────────────────────────
Delete PV that still has data        Snapshot first (Option A)
                                      Recovery available 30 days

Reclaim policy "Retain" by default   Set "Delete" in StorageClass
                                      defaults

Cleanup script delete production     Always: confirm with team
PVC in error                          7-day notice for "abandoned"
                                      Whitelist labels for true orphans

Orphan accumulation between          Monthly audit cadence
audits (>$1K/mo in 6 months)         Automate where possible

Orphans across multi-cluster         Track per-cluster
("the other cluster owns it")        Periodic cross-cluster review

Snapshot orphans accumulate too      Snapshot retention policy
("snapshots forever")                Delete snapshots > 30 days
                                      after volume delete

ESTATE AUDIT (Q3 2026):

DETECTION (via ZopNight scan):
  Orphan PVCs:        47 across 3 clusters
  Orphan PVs (Released): 12
  Failed PVs:          3 (found with kubectl; no rule)

  Total orphan storage: 8.2 TB
  Monthly cost: $656 (mix of gp3 and io2)

INVESTIGATION (1 hour):
  Owner identifiable (via labels):  28 orphans
  Owner unidentifiable:               34 orphans

NOTIFY (7-day window):
  Identifiable owners:
    Slack DM with list of their orphans
    "Will delete in 7 days unless you confirm intent"

  Outcomes:
    18 owners: "delete; we don't need it"
    8 owners: "snapshot + delete" (recovery option)
    2 owners: "leave; we're using it" (false positive; tagged)

UNIDENTIFIABLE (no tags, no labels):
  Default: snapshot + delete
  Snapshot retention: 30 days
  Cost of snapshots: ~$50/mo total for 30 days

EXECUTION (week 2):

  Snapshot:
    34 unidentifiable PVs (default safe action)
    8 identifiable PVs (per owner request)
    Total snapshots: 42
    Snapshot cost (30 days): ~$65

  Delete:
    47 PVCs
    All affiliated PVs

  Update reclaim policies:
    All StorageClasses set to "Delete" (prevent future orphans)

VERIFICATION (week 3):
  Storage usage reduced by 8.2 TB
  Monthly cost reduced by $656
  No incidents (no engineer reporting "where's my data?")

ROI CALCULATION:
  Cleanup cost: 4 hours of platform engineering = $400
  Snapshot cost: $65 over 30 days
  Monthly savings: $656/mo recurring
  Annual savings: $7,872

  ROI: 17:1 in year 1; ~$8K/yr ongoing

CULTURAL IMPACT:
  Communicated the issue + cleanup to all engineering
  Teams now tag PVCs with owner + ttl labels
  New StorageClass defaults to Delete reclaim
  Quarterly audit scheduled

NEXT QUARTER:
  Audit found 8 new orphans (down from 47)
  Reclaim policy change working
  Took 30 minutes to clean up

□ STEP 1: Count orphan PVCs
  Unmounted (billed): run the comm check above; count lines
  Unbound (Pending, unbilled): kubectl get pvc -A | grep Pending | wc -l
    or use the ZopNight RC-1740 recommendation count
  Count: _____

□ STEP 2: Count orphan PVs
  $ kubectl get pv | grep -E 'Released|Failed' | wc -l
  Count: _____

□ STEP 3: Estimate cost
  Total orphan storage: _____ TB
  Storage type: gp3 / io2 / mixed
  Estimated $/mo: $_____

□ STEP 4: Plan cleanup
  Identifiable owners (via labels): _____ orphans
  Unidentifiable: _____ orphans

  Cleanup plan:
    Day 1: Notify identifiable owners (7-day notice)
    Day 8: Snapshot + delete unidentifiable (default safe)
    Day 8: Delete identifiable per owner instructions

□ STEP 5: Prevent future orphans
  □ Update StorageClass reclaim policy to "Delete"
  □ Document PVC tagging requirement (owner + ttl labels)
  □ Schedule monthly orphan audit