Requests and limits: the math behind the bill

RESOURCE REQUEST    Minimum guaranteed to the pod.
                    Reserved on a node when pod is scheduled.
                    Pod won't get less.

RESOURCE LIMIT      Maximum allowed.
                    Pod throttled (CPU) or OOM-killed (memory)
                    if exceeded.
                    Pod can't get more.

NODE has 8 vCPU, 32 GB RAM available (after k8s overhead).

POD A requests 4 vCPU, 16 GB.
POD B requests 4 vCPU, 16 GB.
NODE FULL. K8s won't schedule any more pods.

ACTUAL USAGE:
  Pod A using 1 vCPU, 4 GB (25% of request)
  Pod B using 0.5 vCPU, 2 GB (12% of request)
  Node total actual usage: 1.5 vCPU, 6 GB (~18% of capacity)

EFFECTIVE COST:
  $X/hour for the node, 2 pods on it.
  Each pod: $X/2 per hour

  But: actual computation cost would justify $X/4 if requests
       were 2 vCPU instead of 4.

THE WASTE: 50% of node cost is "reserved but unused."
           Multiplied by hundreds of pods, this is enormous.

SYMPTOMS:
  Node has spare CPU/memory but new pods stay in Pending
  Cluster auto-scales up despite low actual utilization
  Cost per request: very high
  Many nodes; each lightly loaded

DIAGNOSIS:
  kubectl top nodes shows <30% CPU utilization
  kubectl describe nodes shows requests "Allocated" near 100%
  Gap = waste

FIX:
  Lower request to p95 of actual usage
  More pods fit per node
  Fewer nodes needed
  Cost drops 30-50% typical

SYMPTOMS:
  Pods OOM-killed unexpectedly
  CPU throttling under load
  p99 latency spikes
  Replica restarts during traffic peaks

DIAGNOSIS:
  Pod stats: usage frequently at the limit
  Container logs: "OOMKilled" events
  Throttle metrics from container runtime

FIX:
  Raise request closer to p95 actual
  Raise limit closer to p99 + buffer
  Verify stability over 2 weeks

REQUEST  ≈ p95 of actual usage (30 days)
LIMIT    ≈ p99 of actual usage + 30-50% buffer

REASONING:
  REQUEST at p95: pods comfortably operate within their reserved
                  budget. 5% of the time they need more.
  LIMIT at p99 + buffer: allows occasional spikes without OOM.
                         The buffer is for true peaks (P&L season,
                         traffic surge, etc.)

EXAMPLE:
  Workload p50: 0.3 CPU, 200 MB
  Workload p95: 1.2 CPU, 800 MB
  Workload p99: 2.1 CPU, 1.5 GB

  REQUEST: 1.2 CPU, 800 MB (p95)
  LIMIT:   3.0 CPU, 2.2 GB (p99 × 1.4 buffer)

  Result: pods comfortable, occasional spikes handled,
          cluster packs tightly.

MISTAKE 1: REQUEST = LIMIT
  Both at 4 CPU. No room for bursts.
  Any spike → OOM-killed.

  FIX: limit > request by 30-50%

MISTAKE 2: LIMIT VERY HIGH, REQUEST VERY LOW
  Request 0.5 CPU, limit 8 CPU.
  Scheduler doesn't reserve enough; bursts can't get capacity
  (other pods consume the headroom).

  FIX: request reflects realistic operation; limit modest buffer above

MISTAKE 3: BOTH VERY HIGH
  Request 4 CPU when pod uses 0.5.
  Wastes node capacity; cluster grows unnecessarily.

  FIX: measure actual; set request to p95

MISTAKE 4: COPYING FROM ANOTHER WORKLOAD
  "Worked for the API team; use same values"
  Different workloads have different profiles.

  FIX: measure each workload; tune per-workload

MISTAKE 5: SETTING ONCE, NEVER REVIEWING
  Initial values from year 1.
  Workload profile changed (new features, traffic shifts).
  Requests/limits drift from reality.

  FIX: quarterly review of top-10 workloads

TOOLS:

kubectl top pod (basic):
  Shows current CPU/memory per pod
  Not historical

metrics-server (cluster):
  Aggregates pod metrics
  Used by HPA
  Limited history

Prometheus + Grafana (production-grade):
  Long-term history (30+ days)
  Configurable queries
  p50/p95/p99 calculations easy

ZopNight (recommended):
  Cluster-aware recommendations
  Reads actual metrics
  Recommends requests/limits per pod
  Bulk view across cluster

GUARANTEED (highest priority):
  Request == Limit for both CPU and memory

BURSTABLE:
  Request < Limit OR request not set for some resources
  Pod can burst into available capacity

BESTEFFORT (lowest priority):
  No request, no limit
  First to be evicted under pressure

RECOMMENDATION for most workloads:
  BURSTABLE class (request < limit)

  GUARANTEED for latency-sensitive (rare)
  BESTEFFORT for batch/non-critical (rare)

QUARTERLY RIGHT-SIZING (per workload):

WEEK 1: Measure
  Pull 30 days of CPU/memory usage
  Calculate p50, p95, p99 per workload
  Compare to current request/limit

WEEK 2: Recommend
  For each workload, new values:
    Request: p95
    Limit: p99 × 1.4
  Estimate cost impact

WEEK 3: Apply (canary)
  Pick 1 workload (low-risk, well-monitored)
  Apply new requests/limits
  Observe 1 week for stability

WEEK 4: Rollout
  If canary stable: apply to top-10 workloads
  Repeat measurement after 2 weeks
  Iterate next quarter

EXAMPLE: 200-node cluster, average right-sizing opportunity

CURRENT:
  Avg request CPU utilization: 28%
  Cluster CPU cost: $40K/mo

RIGHT-SIZED:
  Avg request CPU utilization: 75%
  Cluster needs: 200 × (28/75) = 75 nodes (down from 200)

  But: keep some headroom + don't take it that far
  Actual right-size: ~100 nodes

  New cluster CPU cost: $20K/mo

SAVINGS: $20K/mo = $240K/yr
EFFORT: 1-2 quarters of right-sizing
PAYBACK: weeks

WORKLOAD: order-processor (50 replicas)

CURRENT SETTINGS (set 18 months ago):
  request: cpu=2, memory=4Gi
  limit:   cpu=4, memory=8Gi

MEASURED ACTUAL USAGE (30 days from Prometheus):
  cpu p50:  0.3 cores
  cpu p95:  1.2 cores
  cpu p99:  2.1 cores
  cpu max:  2.4 cores

  memory p50:  600 MB
  memory p95:  1.4 GB
  memory p99:  2.3 GB
  memory max:  2.8 GB

GAP ANALYSIS:
  Request CPU: 2 cores; actual p95: 1.2 cores
    → 40% over-requested
    → 50 replicas × 0.8 wasted cores = 40 cores reserved-but-unused

  Request memory: 4 GB; actual p95: 1.4 GB
    → 64% over-requested
    → 50 replicas × 2.6 wasted GB = 130 GB reserved-but-unused

OPTIMIZED SETTINGS:
  request: cpu=1.2 (p95), memory=1.5Gi (p95)
  limit:   cpu=3 (p99 × 1.4), memory=3.5Gi (p99 × 1.5)

CLUSTER IMPACT:
  Per node (m5.2xlarge = 8 CPU, 32 GB):
    Before: fits 4 replicas (8 / 2 = 4 by CPU; 32/4 = 8 by mem)
            Actually: fits 4 (CPU-bound)
    After:  fits 6 replicas (8 / 1.2 = 6.7 by CPU; 32/1.5 = 21 by mem)
            Actually: fits 6 (CPU-bound)

  Nodes needed for 50 replicas:
    Before: 50 / 4 = 13 nodes
    After:  50 / 6 = 9 nodes

  Cost per node: $200/mo
  Before: 13 × $200 = $2,600/mo
  After:  9 × $200  = $1,800/mo

  SAVINGS: $800/mo = $9,600/yr

  Just for this one workload.

DEPLOYMENT:
  Update HelmRelease / Kustomize / yaml
  Canary 5 replicas first (1 week)
  Monitor: latency, error rate, OOM events
  Roll out remaining 45 replicas

VERIFICATION (2 weeks):
  No OOM events
  CPU throttling: <0.1% of pod-seconds (acceptable)
  p99 latency: unchanged
  Cost recovered as expected

□ STEP 1: Pick a high-cost workload
  Workload: __________
  Replicas: _____
  Current request: cpu=___ memory=___
  Current limit:   cpu=___ memory=___

□ STEP 2: Get actual usage
  $ kubectl top pod -l app=<workload>

  Sample CPU usage: _____ cores
  Sample mem usage: _____ MB

□ STEP 3: Pull 30-day p95/p99 (from Prometheus or ZopNight)
  CPU p95: _____  p99: _____
  Mem p95: _____  p99: _____

□ STEP 4: Calculate gap
  CPU request vs p95: _____ % over/under
  Mem request vs p95: _____ % over/under

□ STEP 5: Propose new values
  New request: cpu=___ memory=___
  New limit:   cpu=___ memory=___
  Estimated cluster cost reduction: $_____/mo