Scheduling StatefulSets: careful with replicas

STATEFULSET CHARACTERISTICS:
  Pod identity (sts-0, sts-1, sts-2 with ordinal index)
  Persistent storage (PVC per pod)
  Ordered scaling (one at a time)
  Often: distributed system with peer-to-peer communication

DEPLOYMENT                          STATEFULSET
─────────────────────────────────────────────────────────────────
Pods named randomly                 Pods named sts-0, sts-1, etc.
                                    (ordinal identity)

Pods can replace each other         Pods have specific identity
                                    PVCs are pod-bound

Rollout strategy: rolling           Rollout strategy: ordered
Scale: parallel                     Scale: ordered (one-at-a-time)

USE CASES:
  Stateless APIs                    Databases (when self-managed)
  Web tier                          Caches (some, like Redis Cluster)
  Workers                           Distributed systems (Kafka, etcd)

SCALE TO ZERO:
  kubectl scale statefulset/redis --replicas=0
  K8s response:
    Stop pod sts-2 (graceful shutdown; terminationGracePeriodSeconds)
    Wait for sts-2 to terminate
    Stop pod sts-1
    Wait for sts-1 to terminate
    Stop pod sts-0
    Wait for sts-0 to terminate

  Time: typically 30-60 seconds for 3-replica StatefulSet

SCALE FROM ZERO:
  kubectl scale statefulset/redis --replicas=3
  K8s response:
    Create pod sts-0
    Wait for sts-0 to be ready
    Create pod sts-1
    Wait for sts-1 to be ready
    Create pod sts-2
    Wait for sts-2 to be ready

  Time: typically 60-180 seconds for 3-replica StatefulSet

SAFE EXAMPLE: stateful caching layer in staging
  StatefulSet: redis-cluster, 3 pods, ephemeral cache

  Scale to zero: all pods stop. PVCs remain (empty cache anyway)
  Scale back: pods restart in order. Cache rebuilds from source.

  Cost saving: 3 pods × 12 hours/night = 36 pod-hours saved
  No data loss (cache was rebuildable)
  No service impact (staging only)

  DECISION: schedule with confidence

DANGEROUS EXAMPLE: production database StatefulSet
  StatefulSet: postgres-prod, 3 pods (primary + 2 replicas)

  Scale to zero: all pods stop

  RESULTS:
    Customers experience an outage
    Clean shutdown isn't guaranteed
      (terminationGracePeriodSeconds may not be enough for
       in-flight transactions)
    Risk of data corruption
    Replication lag could mean data loss
    Restore time: hours

  DECISION: NEVER schedule

ALL OF THESE MUST BE TRUE TO SCHEDULE SAFELY:

✓ NON-PRODUCTION ENVIRONMENT
  Production stateful workloads: never schedule

✓ DATA IS RECONSTRUCTIBLE
  Cache, replicated, or in-memory
  Can rebuild from source on restart

✓ NO ACTIVE CLIENT CONNECTIONS DURING OFF-HOURS
  No nightly batch jobs hitting this stateful workload
  No scheduled queries from analytics

✓ PVCs ARE NON-CRITICAL (or backed up regularly)
  Test data only; not source of truth

✓ RESTART LATENCY TOLERATES ~1-3 MINUTES
  Service can wait for stateful workload to come up

✓ POD STARTUP ORDER doesn't have specific business requirements
  Engineers OK with sts-0 starting before sts-1 (standard K8s)

ALL CHECKBOXES HIT → schedule with confidence
ANY UNCERTAINTY → default to "no scheduling" or "production = don't schedule"

NEVER SCHEDULE:
  Production databases (any kind)
  Production message queues
  Stateful systems where data could be lost on hard restart
  Distributed systems with consensus protocols (etcd, Raft, Paxos)
  Anything with cross-pod ordering dependencies that can break

FOR THESE, schedule the workloads that DEPEND on them
Not the stateful system itself

PERSISTENT VOLUME CLAIMS attached to StatefulSet:
  Remain bound when StatefulSet scales to zero
  Storage preserved; only compute stops

THIS MEANS:
  Data persists across scheduled stops ✓
  Next scale-up sees the same data ✓
  Storage costs continue while compute is paused ✓

COST IMPLICATION:
  For staging/dev with caches: savings primarily from compute
  Storage cost continues during pause
  Calculate: if storage is huge, savings may be small

EXAMPLE: 3-replica StatefulSet staging-elasticsearch
  Compute cost: $3/day (3 pods × $0.05/pod-hour × 24)
  Storage cost: $1.50/day (300 GB × $0.10/GB-month / 30)
  Total: $4.50/day = $135/month

SCHEDULING 12 hours/day on weekdays + full weekends:
  Compute saved: ~110 hours/week = $11/week
  Storage: unchanged ($1.50/day continues)

  Net savings: ~30% of compute = ~$50/month
  Vs compute-only savings: $50 / $90 (compute portion) = 55%

  Less impressive than for stateless Deployments
  But still meaningful

SCALE-TO-ONE for StatefulSet:
  3 replicas during business hours
  1 replica off-hours (skeleton)

BENEFITS:
  Maintains state continuity
  No cold-start latency
  Storage cost unchanged (PVCs preserved)
  Some compute saved

WHEN TO USE:
  Cache that benefits from warm state
  Distributed system that needs quorum (scale to N quorum, not 1)
  Single-pod operations possible (HA degraded but functional)

WORKLOAD: staging-elasticsearch (3 replicas)
  Ephemeral cache for staging data
  PVCs: 30 GB each (test indices)

CHECKLIST EVALUATION:
  ✓ Non-prod
  ✓ Data reconstructible (cache; can re-index from staging-postgres)
  ✓ No critical off-hours queries
  ✓ PVCs non-critical (only test data; no source of truth)
  ✓ Restart latency: ~3 minutes total (1 min per pod)
  ✗ Some test scripts run nightly that query staging-elasticsearch

DECISION:
  Schedule the StatefulSet
  Move nightly test scripts to after main start

CONFIGURATION:
  Group "staging-stateful" includes staging-elasticsearch
  Schedule: business-hours-staging (8 AM - 8 PM weekdays)

  Separate handling for test scripts:
    Daily 2 AM test runs moved to 8:30 AM (after main start)
    Avoids dependency on stopped elasticsearch

RESULT:
  Saves ~3 pod-hours/night × 5 nights × 4 weeks = 60 pod-hours/mo
  Compute savings: ~$3/mo
  Modest but real
  No test impact (rescheduled cleanly)

  Across multiple non-prod stateful workloads:
    Aggregated savings: $50-200/mo typical
    Aggregate is the value

□ STEP 1: Pick a StatefulSet
  Workload: __________
  Replicas: _____
  PVC size: _____

□ STEP 2: Run the checklist
  □ Non-production
  □ Data reconstructible
  □ No critical off-hours connections
  □ PVCs non-critical
  □ Restart latency tolerable
  □ No ordering issues

□ STEP 3: Decision
  □ Schedule (all checks pass)
  □ Don't schedule (any uncertainty)
  □ Consider scale-to-one alternative

□ STEP 4: Estimate savings
  Compute cost: $_____/mo
  Storage cost: $_____/mo (unchanged)
  Compute savings (scheduling 50%): $_____/mo

□ STEP 5: If scheduling: attach
  Schedule: __________
  Test cycle: __________