HPA: the signal it gives you

HPA STATUS shows:
  Current replicas       (how many running now)
  Desired replicas       (what HPA wants)
  Min / max settings     (bounds you set)
  Current resource usage vs target
  ScalingLimited condition (if any: important!)

HPA HISTORY (over 30 days) shows:
  How often at min      (over-provisioned at min if always there)
  How often at max      (under-provisioned at max if always there)
  Oscillation pattern   (bursty workload signal)
  ScalingLimited events (capacity constraint signal)

ScalingLimited = the replica count the HPA wanted was clamped by a bound
  reason TooManyReplicas: it wanted more than max
  reason TooFewReplicas:  it wanted fewer than min

EXAMPLE (at max):
  HPA target: 70% CPU
  Workload at: 95% CPU (clearly above target)
  HPA wants to scale OUT to handle the load
  But: pod count at max (e.g., 10)
  HPA stuck at 10 replicas; ScalingLimited fires

This is a strong signal:
  Workload is UNDER-provisioned at max
  Either increase max replicas
  Or right-size pods up (bigger pods = fewer needed)

At min (TooFewReplicas) the signal reverses:
  Load would fit in fewer pods than the min allows
  Workload is OVER-provisioned at the min
  Reduce min replicas

PATTERN 1: STUCK AT MIN
  HPA at minimum replicas for >50% of the time
  CPU below target most hours
  ScalingLimited with reason TooFewReplicas

  DIAGNOSIS: Min too high
  WORKLOAD: Over-provisioned at the low end
  ACTION: Reduce min replicas
  COST IMPACT: 20-40% savings on low-traffic hours

PATTERN 2: STUCK AT MAX
  HPA at maximum replicas for >20% of the time
  ScalingLimited (reason TooManyReplicas) frequent
  CPU at or above target

  DIAGNOSIS: Max too low OR pods too small
  WORKLOAD: Under-provisioned at the high end
  ACTION: Increase max OR right-size pods up
  COST IMPACT: maybe higher cost; lower latency / better UX

PATTERN 3: OSCILLATING (rapid scale up/down)
  HPA replica count changing >5×/hour
  Constant scale up → scale down → scale up cycle

  DIAGNOSIS: Bursty workload; HPA reacting to peaks
  WORKLOAD: Spiky but autoscaler can keep up
  ACTION: Increase scaling stabilization window
          OR set target higher (less sensitive)
          OR add KEDA for event-driven scaling (predictive
          scaling needs an add-on such as the PredictKube scaler)
  COST IMPACT: thrash creates churn; smoothing reduces wasted scaling

PATTERN 4: BALANCED (healthy)
  HPA at desired ≈ current always
  Replica count varies smoothly with load
  Rarely at min or max

  DIAGNOSIS: Workload matches target well
  ACTION: No action needed; maintain current settings
  COST IMPACT: Optimal

CURRENT STATE (ZopNight Kubernetes recommendations, EKS IDs shown;
GKE uses 18xx, AKS 19xx):
  RC-1717 HPA At Max Capacity
    → raise max OR right-size pods up; risk of latency during spikes
  RC-1770 / RC-1771 CPU Over-Provisioned
    → the HPA reports ScalingLimited / TooFewReplicas (it wants to go
      below min); lower min or the CPU request
  RC-1743..RC-1746 HPA posture: max = 1, min = max, min = 0,
    no metrics
    → fix the HPA spec itself

HISTORY (your own metrics: Prometheus, kube-state-metrics):
  Hours per day at min           → candidate to reduce min replicas
  ScalingLimited events per week → candidate to raise max
  Replica changes per day        → oscillation; lengthen the
                                    stabilization window, or add KEDA
                                    for event-driven scaling
                                    (predictive scaling needs an
                                    add-on such as the PredictKube
                                    scaler)

WHEN HPA AND A SCALING SCHEDULE ARE BOTH ACTIVE:
  A scheduled job (CronJob patching the HPA, or KEDA's cron
    scaler) sets the replica RANGE (min, max)
  HPA scales WITHIN that range based on metrics

EXAMPLE:
  BUSINESS HOURS: min=4, max=12
  OFF-HOURS    : min=1, max=4
  HPA: targets 70% CPU

  During business hours: HPA scales 4-12 based on CPU
  During off-hours: HPA scales 1-4 based on CPU

COMBINED EFFECT:
  The scheduled range provides the floor (cost control)
  HPA provides the dynamic adjustment (responsiveness)

  Together: low cost during off-hours, full capacity during peaks

HPA SCALES POD COUNT, NOT POD SIZE
  If pods need more memory: HPA can't help
  Need to adjust resource requests/limits manually
  Or: VPA (Vertical Pod Autoscaler): separate tool

HPA SCALES BASED ON METRICS YOU PROVIDE
  CPU: standard, always available
  Memory: built-in resource metric (autoscaling/v2, metrics-server)
  Custom metrics (queue depth, request rate): need configuration
    Tools: Prometheus Adapter, KEDA

HPA HAS LATENCY
  Detection: ~30 seconds
  Scale-up: 30-60 seconds typically
  For sub-second spike response: HPA isn't fast enough
  Use over-provisioning + queue-based load shedding for ultra-low-latency

MISTAKE                                  FIX
──────────────────────────────────────────────────────────────────
Setting min replicas too high            Measure off-peak utilization
"Just in case" thinking                  Reduce min until off-peak
                                          is healthy (CPU 30-60%)

Setting max replicas too low             Watch for ScalingLimited
Limits scale-out during real spikes      events; increase max
User impact during peaks                 OR right-size pods up

Wrong target value (too high)            Target at 80% → scaling near
e.g., 80%                                saturation; latency suffers
                                          Set 65-70% for most workloads

Wrong target value (too low)             Target at 30% → constantly
e.g., 30%                                over-provisioned
                                          Set higher; only go low for
                                          extremely latency-sensitive

Custom metric mis-configured             Verify metric in Prometheus
("queue depth" reports wrong)            BEFORE relying on HPA action

HPA + autoscaler not coordinated         Cluster autoscaler scales nodes
HPA scales pods but no nodes              Both must be configured
                                          See your cluster setup

QUARTERLY HPA TUNING (per workload):

WEEK 1: MEASURE
  Pull 30 days of HPA history
  Categorize: stuck-at-min / stuck-at-max / oscillating / balanced
  Quantify: % time at extremes, scaling event count

WEEK 2: DIAGNOSE
  For each non-balanced workload:
    Identify the issue (too high min, too low max, etc.)
    Estimate cost impact of fixing

WEEK 3: APPLY
  Pick top 3 highest-impact tunings
  Update HPA spec (kubectl apply)
  Monitor for 1 week

WEEK 4: VERIFY
  Confirm pattern moved toward balanced
  Confirm no degradation (latency, error rate, OOM)
  Document changes

WORKLOAD: order-processor (e-commerce, payment-team)
HPA SETTINGS: min=4, max=12, target=70% CPU

OBSERVATIONS OVER 30 DAYS:

  Replica count distribution:
    At 4 replicas (min):  62% of time (overnight + low-traffic)
    At 5-7 replicas:       28% of time (normal business hours)
    At 8-10 replicas:       9% of time (peak times)
    At 11-12 replicas:      <1% of time
    ScalingLimited events:  0

  CPU utilization:
    Daily avg: 38%
    Daily p95: 62%
    Daily p99: 78%

DIAGNOSIS:
  PATTERN 1: STUCK AT MIN (62% of time)
    Min too high: 4 replicas pinned during off-peak
    Each replica at ~25% CPU during off-peak
    Waste: 4 replicas × 75% unused capacity = 3 replica-equivalents wasted

  Max also too high: never reached 12
    Could reduce max to 10 without losing capacity

RECOMMENDATIONS:
  Reduce min from 4 to 2
    Off-peak: 2 replicas handle the light load
    HPA scales up to 4-7 during business hours as needed
  Reduce max from 12 to 10
    Never used 12; 10 is comfortable peak
  Keep target at 70%
    p95 CPU 62% within target; healthy

COST IMPACT:
  Each replica: ~$0.04/hr
  Off-peak window: ~15 hrs/day (62% of 24)
  Savings per day: 2 replicas saved × 15 hrs × $0.04 = $1.20
  Monthly savings: $1.20 × 30 = $36/mo per workload

  Doesn't sound huge BUT scales across:
    18 workloads with similar pattern: ~$648/mo = $7,776/yr

DEPLOYMENT:
  Update HPA spec:
    minReplicas: 2
    maxReplicas: 10
    target: 70% (unchanged)

  Monitor: 2 weeks
    Verify off-peak CPU stays healthy (<80%)
    Verify business hours scaling reaches 4-7 as expected
    Watch for any latency degradation

VERIFICATION (after 2 weeks):
  Off-peak CPU at 2 replicas: 65% (healthy)
  Business hours scaling: 4-6 replicas typical (matches old)
  No latency change
  Saved: $36/mo as projected

DECISION: roll out pattern to other 17 similar workloads
          (similar low-utilization-at-min pattern)

□ STEP 1: Pick a workload
  Workload: __________
  Current min: ___ max: ___ target: ___%

□ STEP 2: Pull HPA history (30 days)
  Prometheus: kube-state-metrics series for the HPA's current
    replicas and conditions (30-day range)
  $ kubectl describe hpa <workload>   (current state, recent events)
  ZopNight: the cluster's HPA list shows current min/max/replicas

  Time at min: _____ % of period
  Time at max: _____ % of period
  ScalingLimited events: _____

□ STEP 3: Classify pattern
  □ Stuck at min (>50% time)
  □ Stuck at max (>20% time)
  □ Oscillating (>5 events/hr)
  □ Balanced

□ STEP 4: Plan tuning
  Action: __________
  Estimated savings: $_____/mo

□ STEP 5: Apply + observe
  Apply: __________
  Monitor for: ___ days
  Verification metric: __________