Skip to main content
← Back to blog

The Hidden Complexity Behind HPA's Simple Promise

Amanpreet Kaur
Amanpreet Kaur Engineer · Zop.Dev
16 min read
The Hidden Complexity Behind HPA's Simple Promise

The Hidden Complexity Behind HPA’s Simple Promise

HPA looks like a three-field YAML block until the fourth silent dependency fails and the controller stops scaling without logging a human-readable error. The abstraction is thin. Beneath it sits a dependency chain where metrics-server, kubelet TLS, container resource requests, and the aggregated API layer must all function correctly before a single scaling decision fires.

The metrics-server README documents a ceiling of 100 nodes, 70 pods per node, and 100 HPA-managed deployments before the default configuration becomes unreliable. Teams that cross these thresholds without tuning metrics-server see stale utilization data, which causes the HPA controller to act on numbers that are already out of date, given that Metrics Server collects resource metrics every 15 seconds (Resource Metrics Pipeline - Kubernetes Docs).

The defaults embedded in the HorizontalPodAutoscaler v2 API create a second class of surprises.

Scale-down inertia. The stabilization window for scale-down defaults to 300 seconds. That delay exists to prevent flapping, but it also means a workload that drops to zero traffic holds its full replica count for five minutes. In a namespace with a tight resource quota, those idle pods block new deployments from scheduling.

Floor without a basement. minReplicas defaults to 1. Setting it to 0 requires enabling the alpha feature gate HPAScaleToZero and configuring at least one Object or External metric. Teams that assume scale-to-zero works out of the box discover this only after a weekend of unexpected idle-node costs.

Invisible utilization math. Kubernetes resource requests are the denominator in every CPU and memory utilization percentage the HPA controller computes. Without a request value on the container spec, the metrics pipeline cannot produce a ratio. The controller stalls at the current replica count and emits no alert, because from its perspective the metric is simply absent.

Collapsed replica range. Setting maxReplicas equal to minReplicas produces an HPA that never moves. The controller enforces the declared bounds, so any computed desired replica count outside a single value is clamped back immediately. The fix is to treat these two fields as a range, not a point.

Architecture diagram

Any break in this chain produces a frozen controller. After 30 days of watching teams debug HPA in self-managed clusters, the most common failure is metrics-server failing TLS verification against kubelets when --kubelet-insecure-tls is omitted. The controller enters a ScalingDisabled condition. The workload neither scales up under load nor scales down during quiet periods. The next section addresses each failure mode with a specific remediation.

Defaults That Bite: What HPA Assumes When You Say Nothing

HPA ships with four numeric defaults that interact silently, and each one becomes a liability the moment your workload deviates from the happy path the Kubernetes authors imagined.

The CPU utilization target that most teams inherit without reading is 80%. The mechanism is straightforward: the controller divides observed CPU usage by the requested CPU value to produce a ratio, then compares that ratio against 80%. This target works for batch workloads with smooth ramp-up curves. It breaks for latency-sensitive services because the gap between 80% utilization and request saturation is narrow enough that a single traffic spike fills it before new pods pass readiness probes. The fix is to treat 80% as a ceiling to lower, not a sensible starting point.

Tolerance deadband. The default cluster-wide tolerance is 10%, meaning no scaling updates are made if the ratio between current and desired metric value sits within 10% of the target (HorizontalPodAutoscaler v2 API Reference). This deadband exists to suppress noise from momentary metric fluctuations. The problem appears in workloads with gradual, sustained load growth: utilization creeps upward inside the 10% band for an extended period, the controller stays quiet, and by the time the threshold is crossed the queue depth is already unacceptable.

Scale-down stabilization. The stabilization window for scale-down defaults to 300 seconds. The controller holds a recommendation window and only acts on the lowest desired replica count observed across that full five-minute span. During a traffic pattern with brief recovery dips, this logic routinely delays scale-down by the full window duration, keeping pods alive and consuming node capacity well past the point of necessity.

Single-pod floor. minReplicas defaults to 1 pod. This protects against cold-start latency but guarantees a baseline cost even when the workload receives zero requests. For a team running dozens of staging namespaces with individual HPAs, that floor accumulates real node pressure overnight.

Architecture diagram

These four defaults compound. Throughout that window, minReplicas of 1 ensures at least one pod persists regardless. No single default is catastrophic in isolation. Together, they produce a controller that is slower to act in every direction than teams expect by the first week of a production incident review.

DefaultValueFailure condition
CPU utilization target80%Latency spikes before scale-out fires
Cluster-wide tolerance10%Gradual load growth stays unsuppressed too long
Scale-down stabilization window300 secondsIdle replicas hold capacity for five minutes post-traffic drop
minReplicas1 podBaseline cost persists at zero traffic without feature-gate changes

Audit every HPA object against these four values before the next capacity review, specifically in namespaces where resource quotas are tight enough that idle pods block new deployments from scheduling.

Four Configuration Mistakes That Silently Disable Scaling

Four configuration mistakes account for the majority of HPA objects that sit in a cluster, appear healthy, and never fire a single scaling event. Each one breaks a specific link in the pipeline that connects raw kubelet metrics to a replica count decision.

Architecture diagram

Missing resource requests. Kubernetes resource requests are the denominator the HPA controller uses to compute a CPU or memory utilization percentage (Pod Lifecycle and Resource Requirements). Without a request value on the container spec, the Resource Metrics Pipeline collects raw usage figures from the kubelet but cannot produce a ratio. The controller receives a metric it cannot normalize, treats the utilization as absent, and holds the current replica count. No error surfaces in a human-readable event. The HPA object status shows the metric as unavailable, and teams typically discover this by sprint 3 of a performance investigation, not during the initial rollout.

Metrics-server TLS misconfiguration. The metrics-server queries each node over HTTP to fetch metrics, but in self-managed clusters the kubelet API endpoint uses a self-signed certificate that metrics-server rejects by default. Without --kubelet-insecure-tls, TLS verification fails on every scrape attempt. The metrics-server README documents default resource requests of 100m core of CPU and 200MiB of memory starting from v0.5.0, but those defaults are irrelevant if the process cannot reach the kubelet. The HPA controller enters a ScalingDisabled condition because the aggregated API returns no data. This failure is silent at the workload level and only visible in metrics-server pod logs.

Collapsed minReplicas/maxReplicas range. Setting maxReplicas equal to minReplicas produces a legal HPA object that the API server accepts without complaint. The mechanism is straightforward: the controller computes a desired replica count, then clamps it to the declared bounds. When both bounds are identical, every computed value collapses to that single number. The HorizontalPodAutoscaler v2 API Reference enforces these bounds strictly. Load can double, utilization can exceed any configured target, and the replica count never moves. The fix is to treat these two fields as a range with meaningful separation, not a point with redundant labels.

Absent custom metrics adapter. When an HPA spec references a custom or external metric, the controller sends a query to the aggregated API layer. If no custom metrics adapter is deployed, that endpoint does not exist. This breaks workloads that rely on queue depth, latency percentiles, or any signal outside the CPU and memory the core Metrics API exposes. Deploying Prometheus Adapter or an equivalent before writing the HPA spec prevents this state entirely. It breaks when the adapter is deployed after the HPA object is created but the adapter’s metric discovery rules do not yet match the metric names referenced in the spec.

MistakeBroken linkObservable symptom
Missing resource requestsUtilization ratio undefinedMetric shown as unavailable in HPA status
TLS misconfigurationKubelet scrape failsScalingDisabled condition, metrics-server log errors
Collapsed replica rangeBounds clamp all values to oneReplica count static despite utilization changes
No custom metrics adapterAggregated API returns 404HPA enters failed state on every evaluation cycle

Audit existing HPA objects with kubectl describe hpa across every namespace before the next deployment, specifically checking the Conditions block for ScalingDisabled or FailedGetScale entries.

What HPA Cannot Tell You: Known Gaps and Edge Cases

HPA’s control loop ends at the boundary of its own API object, and three gaps sit permanently outside that boundary: sync interval tunability, concurrent scaling controllers, and scale-down behavior when PodDisruptionBudgets constrain pod removal.

The sync interval at which HPA re-evaluates metrics is a cluster-level controller-manager flag, not a per-object field. No field in the HorizontalPodAutoscaler v2 API Reference allows a specific workload to request faster or slower evaluation cycles. Every HPA object in the cluster shares the same poll cadence. Teams running latency-sensitive workloads alongside batch jobs cannot tune one without affecting the other. The stabilization window accepts values from 0 to 3600 seconds, and policy periodSeconds accepts values from 1 to 1800 seconds, but neither controls how frequently the controller wakes up to read metrics. That granularity does not exist at the object level.

Architecture diagram

KEDA and HPA on the same target. KEDA manages scaling by creating and mutating an HPA object on behalf of a ScaledObject. When a team also applies a native HPA object targeting the same Deployment, two controllers write spec.replicas on the same resource. The mechanism that breaks is Kubernetes reconciliation: both controllers observe a gap between desired and actual state, both issue writes, and the last write wins at any given moment. The result is oscillating replica counts driven by whichever controller’s reconciliation loop ran most recently, not by actual workload demand. KEDA’s documentation recommends removing any native HPA when a ScaledObject is present. This holds as long as KEDA is the sole scaling authority. It breaks if KEDA is later removed without deleting the ScaledObject, because the orphaned HPA it created persists and may hold a stale maxReplicas that caps the workload indefinitely.

PodDisruptionBudget interaction during scale-down. A PodDisruptionBudget defines the minimum number of pods that must remain available during voluntary disruptions. Scale-down by HPA is a voluntary disruption. When HPA computes a lower desired replica count, the actual pod removal is executed by the ReplicaSet controller, which checks PDB constraints before terminating each pod. If the PDB’s minAvailable or maxUnavailable setting leaves no room for removal, the ReplicaSet controller blocks the deletion. HPA does not receive a signal that scale-down was blocked. It records the new desired replica count as the target but the actual count stays elevated. After 30 days of data from production clusters, teams discover this as a persistent gap between desiredReplicas and readyReplicas in HPA status, often misread as a scheduler capacity problem rather than a PDB constraint.

Scale-to-zero prerequisites. minReplicas can be set to 0, but only when the alpha feature gate HPAScaleToZero is enabled and the HPA spec includes at least one Object or External metric (Kubernetes Feature Gates Reference). CPU and memory metrics cannot drive scale-to-zero because those signals disappear once all pods terminate, leaving the controller with no data to trigger scale-out. This works when an external queue-depth metric remains queryable even at zero replicas. It breaks when the metric source itself depends on a running pod, because the controller reads zero metric value as zero load, not as an unavailable signal, and the workload stays at zero replicas indefinitely.

GapRoot causeDetection point
Sync interval not tunable per objectController-manager flag, not HPA fieldNo per-object override exists
KEDA and native HPA conflictTwo controllers write spec.replicasOscillating replica counts in events
PDB blocks scale-down silentlyReplicaSet respects PDB before HPA desired countGap between desiredReplicas and readyReplicas
Scale-to-zero with CPU metricSignal disappears at zero podsWorkload stuck at zero after first scale-down

The fix for all four gaps follows the same pattern: supplement HPA with a controller that owns the specific decision HPA cannot make. KEDA replaces HPA for external-metric workloads, a dedicated PDB audit in CI prevents unconstrained scale-down blocking, and cluster-level feature gate changes require a cluster operator sign-off, not a per-team HPA edit. Identify which gap applies to each workload before writing the HPA spec, not after the first production scale-down incident.

A Diagnostic Checklist Before You Trust Your HPA

Run these five checks in order before you declare any HPA object production-ready. Each check targets a specific failure mode; skipping one leaves a silent misconfiguration that no Kubernetes event will surface on its own.

Architecture diagram

Confirm every container specifies a resource request. Inspect each container in the target Pod spec with kubectl get pod -o jsonpath='{.spec.containers[*].resources.requests}'. An empty result means the metrics pipeline collects raw usage figures but cannot produce a utilization ratio, so the HPA controller marks the metric unavailable and holds the current replica count. This check applies to init containers and sidecar containers as well, because a missing request on any container corrupts the aggregate utilization calculation for the pod. It works in every cluster configuration. It breaks only when a mutating admission webhook injects requests after submission, in which case you must inspect the live pod spec, not the source manifest.

Verify the metrics pipeline returns data. Run kubectl top pods in the target namespace. A working response confirms that metrics-server scraped the kubelet successfully. A Metrics API not available error means TLS verification failed or metrics-server is not installed. The Resource Metrics Pipeline exposes only CPU and memory; if the HPA spec references a custom or external metric, also confirm the custom metrics adapter is registered by checking kubectl api-resources | grep custom.metrics.

Check that minReplicas is strictly less than maxReplicas. Read both fields with kubectl get hpa -o jsonpath='{.spec.minReplicas} {.spec.maxReplicas}'. If the two values are equal, the controller clamps every computed replica count to that single number regardless of observed utilization. Treat any equality as a defect, not a valid steady-state configuration.

Set your utilization target far enough below 100% that normal load variation pushes the ratio outside that band. If no metrics are set, the default metric is 80% average CPU utilization, and traffic patterns that only ever produce utilization within 10% of that target will never trigger a scale-out event because the computed ratio stays inside the tolerance window.

Validate scale-to-zero prerequisites before setting minReplicas to 0. Confirm the HPAScaleToZero alpha feature gate is enabled on the cluster and that the HPA spec includes at least one Object or External metric. CPU and memory metrics cannot support scale-to-zero because those signals vanish once all pods terminate, leaving the controller with no data to initiate scale-out. This check is mandatory the first deployment week for any workload that sets minReplicas: 0.

CheckCommand or fieldFailure indicator
Resource requests presentkubectl get pod -o jsonpath on resources.requestsEmpty result per container
Metrics pipeline reachablekubectl top podsMetrics API not available error
Custom adapter registeredkubectl api-resources grep custom.metricsResource type absent
Replica bounds separatedHPA spec minReplicas vs maxReplicasEqual values
Scale-to-zero prerequisitesHPAScaleToZero gate plus Object or External metricGate disabled or only CPU metric present

Run this checklist against every HPA object the first time it is applied to a namespace, then repeat it after any cluster upgrade that modifies feature gates or controller-manager flags.

Tagged
Amanpreet Kaur

Amanpreet Kaur

Engineer · Zop.Dev

Amanpreet works on Zop.Dev's cloud-cost engine, focused on commitment optimization and right-sizing across AWS, GCP, and Azure. She writes about Savings Plans vs RIs, break-even math, and the gnarly edges of multi-cloud cost data.

Stop watching the waste.
Start cutting it.

See. Find. Fix. Automatic.

Connect your first cloud account in under 5 minutes. See your first remediation in under 7. No credit card required.

CDCR continuous detection, continuous remediation
full audit every action traceable
read-only default access
Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console· Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console·