Scheduling StatefulSets: careful with replicas

Outcome

By the end of this lesson, you will be able to distinguish safe vs dangerous StatefulSet scheduling patterns, execute the safety checklist, and handle PVCs correctly during scheduling cycles.


Tier	Engineer
JTBD	”Schedule StatefulSets safely: capture savings on non-prod stateful workloads without risking production outages or data corruption.”
Personas	Platform Engineer · SRE · Database Engineer
Prerequisites	M2.6.L1 - L4 · M5.3.L4 (single-replica)
Time	9 minutes
Bloom verb	Distinguish (Analyze), Execute (Apply), Handle (Apply)

1. Concept

A StatefulSet is a Deployment where the copies are not interchangeable.

Each one has a fixed name and a fixed position, sts-0, sts-1, sts-2, and each keeps its own disk. That matters because databases and queues are built this way: copy 0 is not the same as copy 2, and each expects to find its own data where it left it.

Scaling one to zero stops every copy. Scaling it back up starts them again in order, from 0 upwards.

STATEFULSET CHARACTERISTICS:
  Pod identity (sts-0, sts-1, sts-2 with ordinal index)
  Persistent storage (PVC per pod)
  Ordered scaling (one at a time)
  Often: distributed system with peer-to-peer communication

Fixed names and attached disks are exactly why scheduling a StatefulSet needs more care than scheduling a Deployment.

What’s different from Deployment

DEPLOYMENT                          STATEFULSET
─────────────────────────────────────────────────────────────────
Pods named randomly                 Pods named sts-0, sts-1, etc.
                                    (ordinal identity)

Pods can replace each other         Pods have specific identity
                                    PVCs are pod-bound

Rollout strategy: rolling           Rollout strategy: ordered
Scale: parallel                     Scale: ordered (one-at-a-time)

USE CASES:
  Stateless APIs                    Databases (when self-managed)
  Web tier                          Caches (some, like Redis Cluster)
  Workers                           Distributed systems (Kafka, etcd)

Different characteristics → different scheduling considerations.

Scaling behavior: ordered

SCALE TO ZERO:
  kubectl scale statefulset/redis --replicas=0
  K8s response:
    Stop pod sts-2 (graceful shutdown; terminationGracePeriodSeconds)
    Wait for sts-2 to terminate
    Stop pod sts-1
    Wait for sts-1 to terminate
    Stop pod sts-0
    Wait for sts-0 to terminate

  Time: typically 30-60 seconds for 3-replica StatefulSet

SCALE FROM ZERO:
  kubectl scale statefulset/redis --replicas=3
  K8s response:
    Create pod sts-0
    Wait for sts-0 to be ready
    Create pod sts-1
    Wait for sts-1 to be ready
    Create pod sts-2
    Wait for sts-2 to be ready

  Time: typically 60-180 seconds for 3-replica StatefulSet

The ordered scaling can be slow. Plan for it.

Why this is sometimes safe

SAFE EXAMPLE: stateful caching layer in staging
  StatefulSet: redis-cluster, 3 pods, ephemeral cache

  Scale to zero: all pods stop. PVCs remain (empty cache anyway)
  Scale back: pods restart in order. Cache rebuilds from source.

  Cost saving: 3 pods × 12 hours/night = 36 pod-hours saved
  No data loss (cache was rebuildable)
  No service impact (staging only)

  DECISION: schedule with confidence

The ephemeral data + non-prod combination = safe to schedule.

Why this is sometimes dangerous

DANGEROUS EXAMPLE: production database StatefulSet
  StatefulSet: postgres-prod, 3 pods (primary + 2 replicas)

  Scale to zero: all pods stop

  RESULTS:
    Customers experience an outage
    Clean shutdown isn't guaranteed
      (terminationGracePeriodSeconds may not be enough for
       in-flight transactions)
    Risk of data corruption
    Replication lag could mean data loss
    Restore time: hours

  DECISION: NEVER schedule

The persistent data + production = catastrophic if scheduled wrong.

Safe StatefulSet scheduling: checklist

ALL OF THESE MUST BE TRUE TO SCHEDULE SAFELY:

✓ NON-PRODUCTION ENVIRONMENT
  Production stateful workloads: never schedule

✓ DATA IS RECONSTRUCTIBLE
  Cache, replicated, or in-memory
  Can rebuild from source on restart

✓ NO ACTIVE CLIENT CONNECTIONS DURING OFF-HOURS
  No nightly batch jobs hitting this stateful workload
  No scheduled queries from analytics

✓ PVCs ARE NON-CRITICAL (or backed up regularly)
  Test data only; not source of truth

✓ RESTART LATENCY TOLERATES ~1-3 MINUTES
  Service can wait for stateful workload to come up

✓ POD STARTUP ORDER doesn't have specific business requirements
  Engineers OK with sts-0 starting before sts-1 (standard K8s)

ALL CHECKBOXES HIT → schedule with confidence
ANY UNCERTAINTY → default to "no scheduling" or "production = don't schedule"

If any checkbox uncertain: don’t schedule.

Dangerous patterns: never schedule

NEVER SCHEDULE:
  Production databases (any kind)
  Production message queues
  Stateful systems where data could be lost on hard restart
  Distributed systems with consensus protocols (etcd, Raft, Paxos)
  Anything with cross-pod ordering dependencies that can break

FOR THESE, schedule the workloads that DEPEND on them
Not the stateful system itself

The blast-radius of scheduling production stateful workloads is too high.

PVC behavior: preserved through scheduling

PERSISTENT VOLUME CLAIMS attached to StatefulSet:
  Remain bound when StatefulSet scales to zero
  Storage preserved; only compute stops

THIS MEANS:
  Data persists across scheduled stops ✓
  Next scale-up sees the same data ✓
  Storage costs continue while compute is paused ✓

COST IMPLICATION:
  For staging/dev with caches: savings primarily from compute
  Storage cost continues during pause
  Calculate: if storage is huge, savings may be small

PVCs are independent of compute scheduling. The full cost picture includes both.

Calculating savings for StatefulSet

EXAMPLE: 3-replica StatefulSet staging-elasticsearch
  Compute cost: $3/day (3 pods × $0.05/pod-hour × 24)
  Storage cost: $1.50/day (300 GB × $0.10/GB-month / 30)
  Total: $4.50/day = $135/month

SCHEDULING 12 hours/day on weekdays + full weekends:
  Compute saved: ~110 hours/week = $11/week
  Storage: unchanged ($1.50/day continues)

  Net savings: ~30% of compute = ~$50/month
  Vs compute-only savings: $50 / $90 (compute portion) = 55%

  Less impressive than for stateless Deployments
  But still meaningful

The math for StatefulSets is less attractive than Deployments due to PVC cost continuity.

Alternative: scale-to-one for stateful

For some stateful workloads, scale-to-one (M5.3.L4) is better than scale-to-zero:

SCALE-TO-ONE for StatefulSet:
  3 replicas during business hours
  1 replica off-hours (skeleton)

BENEFITS:
  Maintains state continuity
  No cold-start latency
  Storage cost unchanged (PVCs preserved)
  Some compute saved

WHEN TO USE:
  Cache that benefits from warm state
  Distributed system that needs quorum (scale to N quorum, not 1)
  Single-pod operations possible (HA degraded but functional)

For workloads with warm-cache value: scale-to-one beats scale-to-zero.

2. Demo

A team’s StatefulSet scheduling decision:

WORKLOAD: staging-elasticsearch (3 replicas)
  Ephemeral cache for staging data
  PVCs: 30 GB each (test indices)

CHECKLIST EVALUATION:
  ✓ Non-prod
  ✓ Data reconstructible (cache; can re-index from staging-postgres)
  ✓ No critical off-hours queries
  ✓ PVCs non-critical (only test data; no source of truth)
  ✓ Restart latency: ~3 minutes total (1 min per pod)
  ✗ Some test scripts run nightly that query staging-elasticsearch

DECISION:
  Schedule the StatefulSet
  Move nightly test scripts to after main start

CONFIGURATION:
  Group "staging-stateful" includes staging-elasticsearch
  Schedule: business-hours-staging (8 AM - 8 PM weekdays)

  Separate handling for test scripts:
    Daily 2 AM test runs moved to 8:30 AM (after main start)
    Avoids dependency on stopped elasticsearch

RESULT:
  Saves ~3 pod-hours/night × 5 nights × 4 weeks = 60 pod-hours/mo
  Compute savings: ~$3/mo
  Modest but real
  No test impact (rescheduled cleanly)

  Across multiple non-prod stateful workloads:
    Aggregated savings: $50-200/mo typical
    Aggregate is the value

The careful pattern works for stateful workloads where the data isn’t critical.

3. Hands-on (5 min)

Evaluate a StatefulSet for scheduling:

□ STEP 1: Pick a StatefulSet
  Workload: __________
  Replicas: _____
  PVC size: _____

□ STEP 2: Run the checklist
  □ Non-production
  □ Data reconstructible
  □ No critical off-hours connections
  □ PVCs non-critical
  □ Restart latency tolerable
  □ No ordering issues

□ STEP 3: Decision
  □ Schedule (all checks pass)
  □ Don't schedule (any uncertainty)
  □ Consider scale-to-one alternative

□ STEP 4: Estimate savings
  Compute cost: $_____/mo
  Storage cost: $_____/mo (unchanged)
  Compute savings (scheduling 50%): $_____/mo

□ STEP 5: If scheduling: attach
  Schedule: __________
  Test cycle: __________

A 10-minute exercise per StatefulSet. Most production stateful: don’t schedule. Most non-prod: candidate.

4. Knowledge check

Q1

Scheduling a production database StatefulSet to zero is:

A. Standard practice
B. Required for FinOps
C. Dangerous
D. Up to the user

Show answer

Correct: C. Risk of outage and potential data corruption Production stateful workloads should not be scheduled. Even with terminationGracePeriodSeconds, in-flight transactions could fail. Service outage = customer impact. Production stateful = don’t schedule.

Q2

StatefulSet scaling order is:

A. Ordered: scale-down terminates in reverse (sts-2 → sts-1 → sts-0); scale-up creates in order (sts-0 → sts-1 → sts-2)
B. Reverse on scale-up and forward on scale-down, which is the opposite of how the Deployments actually behave
C. Parallel
D. Forward only

Show answer

Correct: A. Ordered scaling is a defining feature; takes longer than parallel Deployment scaling. Ordered scaling is a defining feature.

Q3

A safe StatefulSet to schedule has:

A. Production data
B. Non-prod environment + reconstructible data + tolerable restart latency + non-critical PVCs + no critical off-hours connections
C. Critical data
D. A PodDisruptionBudget configured and at least three replicas running at all times in the target cluster itself too

Show answer

Correct: B. Safe pattern requires ALL of these. Any uncertainty → don’t schedule. Safe pattern requires all four.

5. Apply

StatefulSet scheduling is on the StatefulSet’s detail page. Carefully evaluate the checklist before scheduling.

For your team: be conservative. Production stateful: never schedule. Non-prod: checklist first.

Glossary terms touched

StatefulSet · PVC preservation · Ordered scaling · StatefulSet checklist

Outcome

1. Concept

What’s different from Deployment

Scaling behavior: ordered

Why this is sometimes safe

Why this is sometimes dangerous

Safe StatefulSet scheduling: checklist

Dangerous patterns: never schedule

PVC behavior: preserved through scheduling

Calculating savings for StatefulSet

Alternative: scale-to-one for stateful

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.