Single-replica deployments: reliability vs cost

TRADE-OFF:
  COST:        1 pod × $X = $X (vs 3 pods × $X = $3X)
  RELIABILITY: pod dies = service unavailable until restart
              Typical restart time: 1-5 minutes

DECISION:
  Is the 1-5 minute downtime acceptable?
  Is the cost difference worth the resilience?

GOOD CASES for single-replica:

  INTERNAL TOOLS USED BY ENGINEERS
    Dev dashboards, admin panels, internal APIs
    Engineers can wait 2 minutes during a restart

  DEVELOPMENT / STAGING ENVIRONMENTS
    Pre-production; not user-facing
    Tolerates brief downtime

  BATCH WORKERS
    Failure → retry next batch (idempotent jobs)
    No interactive user waiting

  STATEFUL SERVICES WITH PERSISTENT VOLUMES
    PV preserves state across pod restarts
    Restart time is the only downtime

  COST-SENSITIVE NON-PROD
    Demo environments, training clusters
    Cost matters more than resilience here

  STATELESS SERVICES with retry-friendly clients
    Client retries handle pod restart gracefully
    Brief 503 acceptable

DON'T use single-replica for:

  CUSTOMER-FACING PRODUCTION
    Any downtime = customer impact
    2+ replicas minimum

  REAL-TIME CRITICAL SERVICES
    Financial transactions, alerting systems
    No tolerance for 1-minute outage

  HIGH-TRAFFIC LOAD BALANCERS / API GATEWAYS
    Single pod can't handle the traffic
    Capacity-bound, not just reliability-bound

  SERVICES WITH SLOW COLD-START
    Warm-up takes 5+ minutes
    Restart = extended outage

  COMPLIANCE / REGULATORY WORKLOADS
    Uptime SLAs (99.95%+)
    Single replica can't hit those numbers

  WORKLOADS UNDER ATTACK / HEAVY LOAD
    No headroom; one failure cascades

SCENARIO                              REPLICA COUNT
─────────────────────────────────────────────────────────────
Production + stateless                 2+ replicas (resilience)
Production + stateful                  1 with PV + monitoring
                                       OR 2+ with leader election
Non-prod + any                         1 typically fine
Demo / dev                             1 fine
Batch / async (idempotent)             1 fine (can retry)
Critical batch (timing matters)        2 (parallelism if needed)
Load balancer / ingress                3+ (HA + capacity)
Database (primary)                     1 with PV + read replicas
Database (replica)                     2-3 for read scaling

SCENARIO: stateful service (DB-like, cache, queue) on single replica

  WITH PERSISTENT VOLUME:
    Pod dies → K8s restarts → re-attaches to PV → resumes
    Downtime: 1-3 minutes (acceptable for many workloads)
    Data: intact across restart

  WITHOUT PERSISTENT VOLUME:
    Pod dies → K8s restarts → state lost
    Cold-start latency; broken sessions; cache miss storm
    Often: catastrophic for the service

PV makes single-replica viable for stateful workloads.

EXAMPLES:
  Single-replica PostgreSQL with PV (dev/staging)
  Single-replica Redis with PV (internal cache)
  Single-replica RabbitMQ with PV (low-traffic queue)

PRODUCTION-SCALE DBs still need replica/HA setups; this is for
dev/staging or low-traffic prod.

SINGLE-REPLICA REQUIRES:
  ✓ AGGRESSIVE LIVENESS CHECKS
    Pod restarts itself on health failure
    Don't rely on external monitoring alone

  ✓ AUTOMATIC RESTART POLICIES
    K8s restart policy: Always
    BackoffLimit for jobs/cronjobs

  ✓ MONITORING ALERTS TO HUMANS
    Pod restart count >N in 1 hour → alert
    Pod down >5 min → escalate

  ✓ DOCUMENTED RESTART PROCEDURE
    Runbook in team wiki
    On-call knows what to do

  ✓ STATEFUL: PV BACKUP STRATEGY
    Daily snapshot of PV
    Restore tested quarterly

WITHOUT THIS:
  Pod goes down at 2 AM Saturday
  Nobody notices until Monday morning
  Days of downtime; data loss possible

SINGLE-POD RELIABILITY (typical Kubernetes pod):
  Pod uptime: ~99.9%
  Service uptime: 99.9% (one pod = one point of failure)
  Allowed downtime: ~8.76 hours/year

TWO-POD RELIABILITY (upper bound; assumes independent failures):
  Probability both fail simultaneously: 0.001 × 0.001 = 0.000001
  Best case on paper: 99.9999%

REALITY CHECK: replicas share failure domains
  Same node or AZ (without anti-affinity / topology spread)
  Same bad deploy or config change
  Same dependency (database, DNS, control plane)
  These correlated failures, not single pods, set the real ceiling

THE MATH:
  A second replica removes single-pod failures
    (restarts, evictions, node loss with anti-affinity)
  It does not protect against bad deploys or zone outages

  Cost: 2× or 3× linearly
  Reliability: big real gain from 1 to 2; smaller from 2 to 3

  For prod: the math always favors 2+ replicas
  For non-prod: 1 replica is usually enough

PRODUCTION REPLICA OPTIONS:

OPTION A: ALWAYS 3 REPLICAS
  3 × pod cost = $3X
  Reliability: survives two pod losses (correlated
    failures still apply)
  Over-provisioned for most workloads

OPTION B: MIN 1 + HPA SCALES TO N
  Base: 1 × pod cost = $X
  HPA scales when load increases
  PROBLEM: 1 replica during off-peak = no HA
  Not recommended for prod

OPTION C: MIN 2, MAX N (recommended)
  Base: 2 × pod cost = $2X
  HPA scales up to N for traffic
  Reliability: survives a single pod loss even at min
  This is the sweet spot for most production workloads

OPTION D: MIN 3, MAX N (premium)
  Base: 3 × pod cost = $3X
  HPA scales up to N
  Reliability: survives two pod losses even at min
  Reserve for critical / high-traffic services

PATTERN: RIGHT-SIZE REPLICA COUNT PER ENVIRONMENT

  Production:    minReplicas=2 (or 3 for critical), HPA scales
  Staging:       minReplicas=1, HPA scales to test
  Dev:           replicas=1, no HPA

  Per-workload review quarterly
  Many workloads default to 3 by habit; can reduce to 2

PATTERN: TIERED RELIABILITY

  Tier 1 (revenue-critical): 3+ replicas, multi-AZ
  Tier 2 (customer-facing): 2 replicas
  Tier 3 (internal): 1 replica (or 2 if SLA matters)
  Tier 4 (dev/demo): 1 replica

  Document tier per service; default for new services Tier 2

MISTAKE                              FIX
──────────────────────────────────────────────────────────────────
Production service at 1 replica      Bump to 2 (resilience math
                                      strongly favors)

Dev/staging at 3 replicas             Reduce to 1 (over-provisioned)

Stateful service single-replica       Add PV for persistence
without PV

Multi-replica without anti-affinity  Add podAntiAffinity rules
(all pods on one node)                Or topology spread constraints

Single-replica without monitoring     Add liveness + alerts
                                      Document restart procedure

"Just in case" 3 replicas everywhere  Quarterly review per workload
                                      Many can reduce to 2

TEAM: payment-team, 18 services in production

REPLICA STRATEGY DECISIONS:

  notification-service (customer-facing)
    Decision: 3 replicas (Tier 1: customer-impacting)
    Cost: $90/mo
    Survives: 2 pod losses

  internal-dashboard (engineer-facing)
    Decision: 1 replica (Tier 3: internal; restart OK)
    Cost: $30/mo
    Survives: 0 pod losses (down until it restarts)
    Monitoring: aggressive liveness + alert if down >5 min

  api-gateway (customer-facing)
    Decision: 3 replicas + anti-affinity (Tier 1)
    Cost: $90/mo
    Survives: 2 pod losses, or a node loss (anti-affinity)

  payment-processor (revenue-critical)
    Decision: 3 replicas + 2 AZs + leader election (Tier 1+)
    Cost: $135/mo (3 reps + AZ replication)
    Survives: 2 pod losses, or a full AZ loss

  ml-inference (low-latency customer)
    Decision: 2 replicas with HPA to 10 (Tier 2)
    Cost: $60/mo base + autoscale up
    Survives: 1 pod loss (HPA adds more under load)

  dev/staging environments (all)
    Decision: 1 replica everywhere
    Cost: ~$30/mo per workload
    Survives: 0 pod losses (acceptable for non-prod)

TOTAL MONTHLY COST: $1,800/mo across 18 services

COMPARISON to "all 3 replicas always":
  Naive: $1,800 × 1.5 = $2,700/mo (~$10,800/yr extra)
  Discipline: $1,800/mo (current)
  SAVINGS: $10,800/yr from right-sized replicas

DOCUMENT in team wiki:
  Tier definition per service
  Replica count + reasoning
  Quarterly review schedule

□ STEP 1: List your top 10 deployments
  $ kubectl get deployment -A -o wide

  Deployment 1: __________   Replicas: ___   Env: ___
  Deployment 2: __________   Replicas: ___   Env: ___
  ...

□ STEP 2: Classify each by tier
  Tier 1 (revenue-critical):    __________
  Tier 2 (customer-facing):      __________
  Tier 3 (internal):              __________
  Tier 4 (dev/demo):             __________

□ STEP 3: Identify mismatches
  Production at 1 replica:        __________ (BUMP TO 2+)
  Dev/staging at 3 replicas:      __________ (REDUCE TO 1)
  Stateful without PV:            __________ (ADD PV)

□ STEP 4: Plan changes
  Bumps (production):  ___ workloads × ~$60/mo = $_____/mo cost
  Reductions (non-prod): ___ workloads × ~$60/mo = $_____/mo savings
  Net cost impact: $_____/mo

□ STEP 5: Apply changes
  Update HelmReleases / Kustomize / yaml
  Canary 1 workload
  Roll out remaining