Databases as monitor-only

THE ASYMMETRY:

  COMPUTE (ASG, ECS, etc.):
    Auto-scaled by Event Readiness
    Safe to mutate
    Reversible

  DATABASES (RDS, Cosmos, etc.):
    Monitor-only by design
    Recommendations surface
    Customer DBA executes manually
    Higher-risk mutations

RISKS of auto-scaling databases for an event:

  Cloud-side resize causes brief connection drops
    Customers experiencing the drop during peak event = bad
    Revenue lost during the connection drop

  Performance regression on the smaller-during-rebalance state
    During scale-up, database may briefly serve from smaller instance
    Latency spikes during transition

  Replication lag during the scale operation
    Replicas may fall behind during the scale
    Read-after-write consistency at risk

  Data inconsistency risk in edge cases
    Some rare scenarios involve data state during scaling
    Recovery may require manual intervention

  CONNECTION DROPS during cloud-side scale
    Even "online" scaling has brief disconnects
    Connection pool churn affects performance

THE ASYMMETRIC COST:
  Bad event-time database scaling = revenue lost + customer impact
  Manual DBA scaling = controlled; tested; planned
  ZopNight's choice: never auto-scale databases

EVENT READINESS DB MONITORING:
─────────────────────────────────────────────────────────────────
RESOURCE: db-production-orders (RDS db.r5.xlarge)
EXPECTED LOAD INCREASE: 3x (per multiplier on this event)

CONNECTION POOL MATH:
  App instances × connections per app: 12 × 10 = 120
  Current max connections:       2,730 (db.r5.xlarge default)

  Expected event connections:    120 × 3 = 360
  Share of the limit:            13% (good: within limit)

RECOMMENDATION: NO ACTION needed. Headroom sufficient.

BUT IF the projection passes 70% of the limit, a warning surfaces
(above 100% it is critical):

  "Connection pool may exhaust during event. Three options:
   1. Increase max_connections (requires restart, careful timing)
   2. Scale instance class (db.r5.xlarge → db.r5.2xlarge)
      which doubles max connections by default
   3. Add read replicas to offload reads
      Or: implement connection pooler (PgBouncer / ProxySQL)"

DB TIER ANALYSIS:

  App instances × connections per app: 16 × 10 = 160
  Current max_connections:        2,730
  Predicted event connections:    320 (assuming 2x load)

  Recommendation:
    Suitable: db.r5.xlarge supports this load

  OR:

  App instances × connections per app: 80 × 10 = 800
  Predicted event connections:    2,400 (assuming 3x load; 88%)

  Recommendation:
    Scale db.r5.xlarge → db.r5.2xlarge before event
    (sized at 1.3x projected = 3,120; max_connections
     goes from 2,730 to 5,461)
    No connection pool stress

3 WEEKS BEFORE EVENT:
  Run load test to validate database can handle 2-3x current load
  Identify any slow queries that may bottleneck
  Plan query optimization or index improvements

1 WEEK BEFORE EVENT:
  If scaling decided: deploy a temporary database scale
    (e.g., from db.r5.xlarge → db.r5.2xlarge)
    Outside business hours / planned maintenance window
  Add read replicas if read-heavy
  Pre-warm any caches

1 DAY BEFORE EVENT:
  Verify replication lag is low
  Confirm backup is recent
  Document rollback procedure
  Verify connection limits sufficient

DURING EVENT:
  Monitor connections, query latency, replication lag
  Watch for unusual error patterns
  Plan to scale down 1-2 weeks AFTER event
    (avoid premature rollback)

1 WEEK AFTER EVENT:
  Review actual database performance during event
  Decision: scale back or keep at higher tier?
  Postmortem if any issues occurred

WHY CONNECTION POOL MATH MATTERS:

  Most database performance issues during traffic spikes are NOT
    CPU saturation: they are connection pool exhaustion
  An app trying to acquire a connection waits or fails
  Connection pool issues cascade
    Slow connection → request times out → retry → more connections

HOW IT IS CALCULATED:

  App instances:                  how many app instances connect
  Connections per app:            pool size per instance
  Multiplier:                     from event multiplier
  Expected event connections:     app instances × connections per app
                                  × multiplier (÷10 with a pooler)
  Max connections per instance:    per-instance-class default

  ZopNight takes the tightest of four ratios:
    CPU vs 85%, IOPS vs the instance limit,
    connections vs max, memory vs 90%
  ABOVE 70%:  warning
  ABOVE 100%: critical
  Upgrade suggestion is sized at 1.3x the projection

ZOPNIGHT WILL NOT:
  Restart a database (would cause connection drops)
  Scale instance class (would cause connection drops during rebalance)
  Modify max_connections (requires restart)
  Promote a replica (potential data loss)
  Change replication topology
  Modify schema
  Run database optimization

ALL OF THE ABOVE are customer DBA responsibilities
ZopNight is advisory for databases

FOR EVENTS THAT REPEATEDLY OVERFLOW connection pool:

  Connection pooler (PgBouncer / ProxySQL):
    Multiplexes app connections to fewer DB connections
    Allows app to maintain "unlimited" connection appearance
    Database sees fewer actual connections
    Costs: ~$20-100/mo per pooler instance

  Read replicas:
    Offload read queries to replicas
    Each replica = max_connections more capacity
    Eventual consistency on read path

  Application-level retry + circuit breaker:
    App handles connection failures gracefully
    Fewer cascade failures during pool exhaustion

LONGER-TERM ARCHITECTURAL improvements:
  Connection pooling is industry-standard
  Almost always worth implementing once at scale

EVENT: Black Friday 2026
SCOPE: production-checkout-api group (3 ASGs + 1 database)

DATABASE: db-prod-orders (RDS db.r5.xlarge)
  Current max_connections: 2,730
  App instances × connections per app: 80 × 10 = 800
  Predicted event connections: 800 × 3 = 2,400 (88%: warning)

ZOPNIGHT RECOMMENDS (monitor-only):
  "Connection limit at risk. Three options:

   1. SCALE UP: db.r5.xlarge → db.r5.2xlarge
      Cost: ~$430/month extra
      max_connections becomes 5,461
      Plan: 1 week before event, during maintenance window

   2. ADD READ REPLICA: offload read queries to replica
      Plan: 2 weeks before event, replica needs to catch up

   3. CONNECTION POOLER (PgBouncer or ProxySQL):
      Plan: longer rollout; application changes required
      Most durable solution"

DBA TEAM DECISION:
  Short-term: Option 1 (scale up)
  Long-term: Plan Option 3 for next year
  Reasoning: Option 1 quick; Option 3 better architecture but takes time

EVENT TIMELINE:
  T-7 days   Scale db.r5.xlarge → db.r5.2xlarge during planned
              maintenance window
              Verify max_connections is now 5,461
              Verify replication lag returns to normal post-scale

  T-1 day    Final verification:
              Backup recent
              Replication lag low
              All systems green

  T+0        Event starts
              Database serves 3x traffic comfortably
              P99 connection count: 2,250 (within 5,461 limit)
              Query latency: normal

  T+30 days  Decide whether to scale back or keep at 2xlarge
              Decision based on ongoing post-event traffic
              Often: keep until ongoing capacity needs assessed

RESULTS:
  Event: smooth, no DB issues
  Cost: $430 extra for the month
  Lessons learned: documented for next event

□ STEP 1: List databases in event scope
  Database 1: __________
  Current max_connections: _____
  App instances × connections per app: _____

□ STEP 2: Calculate event multiplier
  Event multiplier: ___x
  Predicted event connections: _____ × ___ = _____

□ STEP 3: Assess
  Predicted > 70% of max? □ Yes (warning) □ No (OK)
  Predicted > max? □ Yes (critical) □ No

□ STEP 4: Plan action if needed
  Option 1: Scale up: cost $_____/mo
  Option 2: Read replica: cost $_____/mo
  Option 3: Connection pooler: effort _____ weeks

□ STEP 5: Schedule
  Action date (1 week before event): __________
  Owner: __________
  Maintenance window: __________