Reliability theatre anti-pattern

CLASSIC EXAMPLES:

  Multi-AZ for READ-replicas without multi-AZ primary
    → No write fail-over benefit

  Backups that are NEVER restored
    → Unknown reliability; may fail when needed

  HA setup with a SINGLE POINT of failure elsewhere
    → SPOF makes the HA moot

  Cross-region replication of NON-CRITICAL data
    → Pay for protection nobody uses

  100 health checks that nobody reads
    → Alert fatigue; real signals lost

  Active-passive DR with synchronous replication on slow link
    → Replication lag = data loss anyway

SCENARIO:
  Read replicas in multi-AZ configuration
  Application writes to PRIMARY (single-AZ)
  Reads from replicas in multiple AZs

WHEN AZ FAILS:
  Replica in failed AZ goes down
  App reads from other AZ's replicas (still works)
  BUT: primary is in failed AZ → ALL WRITES FAIL
  No write fail-over benefit despite "multi-AZ" label

ACTUAL RELIABILITY:
  Read availability: good
  Write availability: NONE during AZ failure

COST: paying for multi-AZ on read replicas
       Net: cost without protection

THE FIX:
  Multi-AZ PRIMARY (write fail-over works automatically)
  OR: accept single-AZ as the operational reality
  Don't pretend with the half-measure.

SCENARIO:
  Daily snapshots configured
  Retention policy set (e.g., 30 days)
  Compliance requirement met (on paper)

REALITY:
  Snapshot rarely restored
  Verification never done
  When disaster hits:
    Snapshot might be corrupted
    Restore process not documented
    Time to recover: unknown
    Could fail entirely

INDUSTRY DATA:
  backup systems that are never restore-tested frequently fail under real conditions
  Untested rate is presumably worse

THE FIX:
  Annual restore test
  Document the procedure
  Measure actual restore time (this becomes your real RTO)
  Test on multiple workload types

The backup is only as good as your last successful restore.

SCENARIO:
  3 web servers behind load balancer (looks HA!)
  Database is a single instance, single-AZ

WHEN DB FAILS:
  Web servers can't read/write
  3-server HA doesn't help
  Service down

COST: 3× web server + 1× single DB
PROTECTION: actual HA = 0 (DB is SPOF)

THE FIX:
  Either:
    1 web server + multi-AZ DB (real protection of the SPOF)
    3 web servers + multi-AZ DB (real HA across both tiers)

  DON'T pretend with multi-server on the wrong layer.

VARIANTS OF SAME PATTERN:
  Multi-AZ K8s nodes but single ETCD
  Load balanced API but single-region database
  Multi-region web but single-region critical service

SCENARIO:
  Daily backup of internal-tool database to remote region
  Considered "DR-ready"

REALITY:
  Backup data: 24+ hours stale
  Failover to backup region = significant data loss
  But the workload doesn't NEED low RPO
  So why pay for cross-region?

COST: 2× backup storage + cross-region replication
PROTECTION: protection against full region loss (rare)
            But: for internal tool that can wait days for restore

THE FIX:
  Either:
    Multi-region active-active (REAL DR for critical workloads)
    Single-region with cross-AZ HA (most cost-effective for non-critical)

  Don't pretend with the half-measure (cross-region backup
  with 24h RPO doesn't help workloads that "need DR").

  Make the call:
    Critical: pay for real DR
    Non-critical: don't pay; accept the failure mode

REPEATING THEME:
  Right-sizing per workload (M5.5.L1) handles this.
  Theatre = blanket policies applied without analysis.

SCENARIO:
  100 health checks per service
  Custom metrics per endpoint
  Alerts on every metric

REALITY:
  Alert fatigue → on-call mutes the channel
  Real issues missed in the noise
  Engineers stop investigating alerts
  False sense of "we'll catch everything"

COST:
  $200-2,000/month per service in CloudWatch + custom monitoring
  Plus: 5-10x more pages
  Plus: on-call burnout

PROTECTION: degraded (alert fatigue = real issues missed)

THE FIX:
  5-10 critical metrics per service (not 100)
  Alert thresholds tuned to actual incidents
  Aggregate health (golden signals: latency, traffic, errors,
                                     saturation)
  Page only on actual customer impact

  Quality over quantity.

THE NORMAL CASE:
  3-5 alerts per service per month
  Each requires investigation
  100 alerts per service per month = burnout-driven mute

PATTERN 6: SYNCHRONOUS REPLICATION ON HIGH-LATENCY LINK
  Cross-region synchronous replication
  Network latency > replication SLO
  Writes block waiting for confirmation
  Actually slower than async + good RTO

  THE FIX: async replication with reasonable RPO

PATTERN 7: REDUNDANCY IN COMPONENTS THAT DON'T FAIL
  Triple-redundant power on resources that don't have single power
  Multi-vendor failover for cloud services that have own redundancy
  Over-engineering at component level

  THE FIX: trust the platform's redundancy; layer YOUR
            redundancy at the application boundary

PATTERN 8: BACKUP-OF-BACKUP-OF-BACKUP
  3 copies of every backup in different storage
  Cost compounds linearly
  Each additional copy: marginal protection (2 copies usually enough)

  THE FIX: 2 copies in different regions/accounts maximum
            (3-2-1 rule: 3 copies, 2 media, 1 off-site)

PATTERN 9: HEALTH CHECKS THAT DON'T MATTER FOR SLAs
  Pinging endpoints not on critical path
  Monitoring services that don't affect customer experience

  THE FIX: monitor what customers experience; not internal plumbing

ASK FOR EACH RELIABILITY INVESTMENT:

  1. WHEN WAS THE LAST DR TEST?
     >1 year ago = red flag (theatre risk)

  2. WHAT'S THE ACTUAL RESTORE TIME?
     "Unknown" = red flag
     "We've never measured" = red flag

  3. WHERE ARE THE SPOFs?
     "Don't know" = red flag
     The reliability investment may be on the wrong dimension

  4. WHAT DOES THIS INVESTMENT PROTECT AGAINST?
     Vague answer = red flag
     Should be specific failure mode

  5. HOW DOES THIS COMPARE TO A SIMPLER DESIGN?
     "We've always done it this way" = red flag
     Should have explicit comparison

  6. WHEN WAS THIS LAST REVIEWED?
     Years ago = red flag (assumptions may be stale)

  7. HOW MANY TIMES HAS THIS BEEN INVOKED IN PRODUCTION?
     Never = either lucky or theatre (test to know)
     Often = real value

RELIABILITY THEATRE COSTS:

DIRECT COSTS:
  Extra resources (multi-AZ overhead, replication, etc.)
  Storage (untested backups)
  Network (cross-region transfer)
  Monitoring (excessive metrics + alerts)

INDIRECT COSTS:
  Operational complexity (more to manage)
  Maintenance burden (more to keep working)
  Alert fatigue (real issues missed)
  Wrong expectations (DR plan that doesn't work)
  False sense of security (no urgency to fix real gaps)

TYPICAL ORG-LEVEL:
  10-30% of reliability spending is theatre
  Mid-size org: $50K-$200K/year in theatre cost
  Plus indirect costs (incident-response slowdown, etc.)

QUARTERLY RELIABILITY AUDIT:

STEP 1: INVENTORY
  Map every reliability investment
  Multi-AZ resources, replication, DR setup, monitoring

STEP 2: INTERROGATE
  For each, ask the audit questions above
  Identify candidates for theatre

STEP 3: TEST
  For backup: do annual restore test
  For DR: run drill
  For HA: chaos engineering (kill components)

STEP 4: CLASSIFY
  Real protection: keep + document
  Partial / unclear: upgrade to real
  Theatre: remove or downgrade

STEP 5: CALCULATE
  Cost of theatre identified: $$$
  Cost of real upgrade: $$
  Cost of removal: -$$

STEP 6: EXECUTE
  Remove pure theatre
  Upgrade partial protection
  Document real protection

TEAM AUDIT (Q3 2026):

INVESTMENT 1: Multi-AZ for read replicas (RDS)
  Cost: +30% on DB cost ($1,200/mo extra)
  Protection claim: HA via replicas
  Actual: read availability only; primary still single-AZ
  Test result: simulated AZ failure → reads work, writes fail
  STATUS: THEATRE

  ACTION:
    Upgrade to multi-AZ primary
    Additional cost: +10% (~$400/mo more = +$400/mo total)
    Real fail-over works
    Net: $400/mo more for real HA
    Tested in next chaos drill

INVESTMENT 2: Cross-region backup of customer dashboard data
  Cost: +5% on storage ($300/mo)
  Protection claim: regional DR for dashboard
  Actual: 24-hour data loss possible
  Workload requirement: dashboard can be down 24h; data tolerance unclear

  Investigation:
    Asked product: "OK to lose 24h of dashboard data in disaster?"
    Answer: "Yes, that's acceptable"

  STATUS: REAL (just barely)
  ACTION: keep; document expected RPO for next review

INVESTMENT 3: 100 health checks per service across 30 services
  Cost: $6,000/mo CloudWatch + monitoring tools
  Protection claim: catch every issue
  Actual: alert fatigue; on-call mutes most alerts; only top 5 actioned
  Test result: in last 6 incidents, only 3 of the 100 alerts fired BEFORE
              the customer reported
  STATUS: THEATRE (mostly)

  ACTION:
    Audit alerts per service
    Reduce to 5-8 critical golden-signal metrics
    Tune thresholds to actual incidents
    Estimated savings: $4,200/mo from removed monitoring
    Expected: alert quality improves; on-call no longer mutes

INVESTMENT 4: Backup of backups in 3 regions
  Cost: 3× backup storage ($1,500/mo total)
  Protection claim: ultra-DR
  Actual: 3rd copy never used in 5 years of operation
  STATUS: THEATRE (3-2-1 rule satisfied with 2 copies)

  ACTION:
    Remove 3rd region backups
    Keep: 1 region for primary, 1 region for DR
    Savings: $500/mo

INVESTMENT 5: Active-active DR for internal admin tool
  Cost: 2× compute ($800/mo)
  Protection claim: zero downtime
  Actual: tool isn't customer-facing; downtime cost ~$0
  STATUS: THEATRE

  ACTION:
    Downgrade to single-region with backup
    Savings: $400/mo
    Acceptable: tool can be down for hours; engineers can wait

AUDIT TOTALS:
  Theatre identified: 4 of 5 investments (partial or full)
  Total monthly waste: $11,100/mo
  Upgrade cost: $400/mo (Investment 1)
  Net savings: $10,700/mo = $128,400/year

  Plus: actual reliability IMPROVED (real HA on Investment 1)
        Plus: alert quality improved (fewer, better signals)

□ INVESTMENT 1: __________
  Cost: $_____/mo
  Protection claim: __________
  Last tested: __________
  Audit questions:
    □ DR test in last year?
    □ Restore time known?
    □ SPOFs mapped?
    □ Compared to simpler design?
  Status: □ Real  □ Theatre  □ Partial
  Action: __________

□ INVESTMENT 2: __________
  (same template)

□ INVESTMENT 3: __________
  (same template)

CALCULATED THEATRE:
  Investments classified as theatre: _____
  Estimated waste: $_____/mo
  Upgrade cost (for partial): $_____/mo
  Net annual savings: $_____