Skip to main content
← Back to blog

The Hidden Cost of Default Eviction Settings

Riya Mittal
Riya Mittal Engineer · Zop.Dev
18 min read
The Hidden Cost of Default Eviction Settings

The Hidden Cost of Default Eviction Settings

The kubelet’s eviction logic operates on a pressure-detection cycle. The kubelet updates node conditions based on the configured nodeStatusUpdateFrequency, which defaults to 10 seconds (Node-pressure Eviction). That 10-second polling interval is the minimum lag between a node crossing a pressure threshold and the scheduler learning about it. Pods keep landing on a saturated node during that window, compounding the problem before any eviction fires.

The Reclaim Floor problem compounds this. If the storage a kubelet could reclaim is less than 2GiB, the kubelet skips reclamation entirely. A node where the amount of storage that the kubelet could reclaim is less than 2GiB receives zero relief, because the kubelet does not reclaim anything in that case. The node stays under pressure, the eviction loop stalls, and new pods scheduled onto that node inherit the same constrained environment.

The gap between threshold breach and actual pod removal creates a window where the scheduler keeps assigning work to a node that is already failing. The fix is explicit soft eviction thresholds with grace periods, so the kubelet begins shedding load before hard limits trigger abrupt OOMKill events.

Architecture diagram

Three failure modes produce the incident tickets teams file after the fact.

BestEffort pod loss. When no resource requests are set, the kubelet classifies pods as BestEffort QoS and evicts them first under memory pressure. The eviction is correct behavior per the spec, but the team experiences it as random pod death because no alert fired on a threshold they never configured.

OOMKill loops. Without soft eviction thresholds, the kubelet waits for hard limits and then kills processes abruptly. The pod restarts, consumes memory again, and hits the same wall. By sprint 3 of a new service rollout this pattern typically produces a CrashLoopBackOff that teams misdiagnose as an application bug.

Scale-down disruption. Cluster Autoscaler drains nodes through the eviction API. Without a PodDisruptionBudget, the autoscaler evicts every pod on a node before respecting availability constraints, because there are no constraints to respect. The Disruptions documentation is explicit: the eviction API honors minAvailable and maxUnavailable only when a PodDisruptionBudget exists.

The split image filesystem feature adding containerfs eviction signals reached beta in Kubernetes v1.31 and is enabled by default. Teams running v1.31 or later now have a new eviction signal they never explicitly opted into. If your kubelet configuration was written before v1.31, audit it against the current signal set before the next node upgrade.

The next concrete step is to pull the kubelet configuration from one production node and check three fields: evictionSoft, evictionSoftGracePeriod, and evictionMinimumReclaim. Absent entries in those fields mean you are running on defaults designed for correctness, not for service availability.

How the Kubelet Decides What to Kill and When

The kubelet’s kill decision is not a single judgment call. It is a pipeline: signal collection, QoS class ranking, OOM score assignment, and a nodeStatusUpdateFrequency lag that sits between pressure detection and scheduler awareness.

The pipeline starts with eviction signals. Kubernetes defines eviction signals as computed values derived from node metrics, such as memory.available, nodefs.available, and imagefs.available (Node-pressure Eviction). When a signal crosses a configured hard threshold, the kubelet applies a 0-second grace period, meaning the target container receives immediate shutdown with no SIGTERM window. No application lifecycle hook fires. No graceful drain occurs. The process is gone.

The 10-second nodeStatusUpdateFrequency default creates a structural blindspot. The kubelet updates node conditions on that cycle, so the scheduler learns about memory pressure at most 10 seconds after the kubelet detected it. During those 10 seconds, the scheduler continues placing pods on a saturated node. Those incoming pods then compete for the same constrained memory, raising the pressure further, which accelerates the next eviction. The lag does not cause a single incident. It amplifies the one already in progress.

Architecture diagram
MetricValue
Hard eviction grace period0s
nodeStatusUpdateFrequency default10s
Guaranteed pod oom_score_adj-997

Once the kubelet identifies a pressure threshold breach, it ranks candidate pods by QoS class before selecting a target. The three QoS classes in descending eviction priority are BestEffort, Burstable, and Guaranteed. BestEffort pods carry no resource requests or limits, so the kubelet treats them as the most expendable and evicts them first. Burstable pods have requests set but consume above those requests, making them the next candidate. Guaranteed pods, where requests equal limits on every container, are protected last.

The OOM score mechanism enforces this ordering at the kernel level. The kubelet sets an oom_score_adj value of -997 for containers in Guaranteed-class pods. The Linux OOM killer interprets lower scores as higher protection. A score of -997 places a Guaranteed container near the bottom of the kernel’s kill list, below almost every other process on the node. This protection is real, but it only applies when the kernel’s OOM killer fires. The kubelet’s own eviction logic can still terminate a Guaranteed pod if no lower-QoS candidates remain.

QoS class assignment. A pod’s QoS class is computed at admission, not at eviction time. The kubelet does not re-evaluate class under pressure. Teams that set requests but forget to set matching limits on a sidecar container drop their pod from Guaranteed to Burstable, exposing it to earlier eviction without any visible warning at deploy time (Configure Quality of Service for Pods).

OOM score protection boundaries. The -997 oom_score_adj assigned to Guaranteed containers protects against kernel-level OOM kills. It does not protect against kubelet-driven eviction when the node exhausts all BestEffort and Burstable candidates. A cluster running exclusively Guaranteed pods under memory pressure will still see kubelet evictions, because the signal threshold, not the OOM score, is what triggers the eviction pipeline.

Same-class ordering. Within a single QoS class, the kubelet uses resource consumption relative to requests to rank candidates. The pod consuming the most memory above its request is targeted first. The exact tiebreak logic when two pods sit at identical relative consumption is unspecified in official documentation. Build eviction safety on QoS class boundaries and explicit priority classes, not on assumptions about same-class ordering.

The actionable next step is to run kubectl get pods --field-selector=status.qosClass=BestEffort -A on each production cluster. Every pod returned by that query has a 0-second kill guarantee under any hard eviction threshold breach, and the scheduler placed it there because no resource requests told it otherwise.

The Configuration Gaps That Turn Pressure Into Outages

Four configuration omissions compound into a self-reinforcing failure loop: missing resource requests, absent soft thresholds, zero evictionMinimumReclaim, and untuned hugepage handling each degrade the kubelet’s ability to recover gracefully, and together they produce eviction cascades that look like application bugs.

Kubernetes resource requests are the declared minimum CPU and memory a container requires, used by both the scheduler for placement and the kubelet for QoS class assignment (Resource Management for Pods and Containers). Without them, the kubelet has no floor to measure consumption against. A container that never declared a request cannot be measured as “above its request,” which means the entire relative-consumption ranking breaks down. The kubelet falls back to treating the pod as BestEffort, the lowest QoS class, and evicts it first under any memory pressure event.

Missing resource requests. When a container has no request set, the kubelet classifies the pod as BestEffort and targets it immediately under memory pressure. The container’s OOMKilled event confirms what happened: the container attempted to use more memory than its configured limit, or in the BestEffort case, any memory at all once the node is saturated. The team sees the event in pod logs after the fact, not before. The fix is to set explicit requests and limits on every container, including init containers and sidecars, before the first production deployment.

Absent soft eviction thresholds. Without evictionSoft and evictionSoftGracePeriod configured, the kubelet skips the warning stage entirely. When a hard threshold is crossed, the kubelet applies a grace period of 0 seconds, meaning the container receives immediate termination with no SIGTERM window (Node-pressure Eviction). The application cannot flush state, close connections, or drain in-flight requests. After 30 days of data from a service with no soft thresholds configured, the OOMKill rate will reflect abrupt kills rather than graceful restarts. Soft thresholds work when memory pressure is transient. They break when sustained utilization keeps the node above the soft threshold longer than the grace period, at which point the kubelet promotes the condition to hard and kills immediately anyway.

Zero evictionMinimumReclaim. The default evictionMinimumReclaim is 0 for all resources. A node under disk pressure that evicts one pod and reclaims 500MiB drops below the eviction threshold by the smallest margin, stops evicting, and then crosses the threshold again as new pods are scheduled. This produces a rapid oscillation between pressured and stable states. The reclaim floor mechanism exists specifically to break this cycle: setting a nonzero evictionMinimumReclaim forces the kubelet to continue evicting until it has recovered enough capacity to stay below the threshold.

Architecture diagram
Configuration fieldDefaultFailure mode when unset
requests and limitsNoneBestEffort QoS, first eviction target
evictionSoftNoneNo warning stage, 0s grace kills only
evictionMinimumReclaim0 for all resourcesThreshold oscillation, continuous eviction loop
PodDisruptionBudgetNoneAutoscaler drains node with no availability constraint

Hugepage handling. Hugepages allocated to a container are not reclaimable by the kubelet’s standard eviction pipeline. A node running database or high-throughput network workloads that pre-allocate hugepages can reach memory pressure even when standard page memory appears available, because the kubelet’s memory.available signal does not account for hugepage reservations in all configurations. This failure mode surfaces specifically on nodes with large hugepage allocations and no corresponding hugepage limits set in the container spec.

PodDisruptionBudget absence. Cluster Autoscaler removes underutilized nodes by calling the eviction API against each pod. Without a PodDisruptionBudget, there are no minAvailable or maxUnavailable constraints for the API to respect. All pods drain simultaneously. The service loses every replica on that node at once. A PodDisruptionBudget with minAvailable: 1 costs nothing at scheduling time and prevents total replica loss during any API-driven eviction, whether from autoscaling or a manual node drain.

The concrete next step is to audit one production Deployment per tier: check that every container has explicit requests set, that evictionSoft is present in the kubelet config, that evictionMinimumReclaim is nonzero for memory and storage, and that a PodDisruptionBudget exists. Missing any one of these on a node under sustained load is enough to start the loop.

PodDisruptionBudgets and the Eviction API: Where Voluntary and Involuntary Eviction Diverge

The Eviction API and the kubelet’s node-pressure eviction pipeline share a name but operate under entirely different rules, and conflating them produces misconfigured systems that fail precisely when you expect protection.

Voluntary eviction via the API. A voluntary eviction happens when a controller, such as Cluster Autoscaler or a node drain command, submits an eviction object through the Kubernetes Eviction API rather than deleting the pod directly. The API checks any matching PodDisruptionBudget before proceeding (Disruptions - Kubernetes Docs). If the eviction would violate the budget’s constraints, the API returns a 429 and the caller must wait and retry. The gate is real: tools that respect PodDisruptionBudgets call the Eviction API instead of directly deleting pods or deployments.

Involuntary eviction by kubelet. Node-pressure eviction is initiated by the kubelet itself when a resource signal crosses a threshold. The kubelet does not consult the Eviction API. It does not check PodDisruptionBudgets. It ranks pods by QoS class, selects a target, and terminates the container. A PodDisruptionBudget with minAvailable: 100% or maxUnavailable: 0 stops the Cluster Autoscaler cold but provides zero protection the moment the node runs out of memory (Node-pressure Eviction).

This divergence is the source of a specific production gap. A team sets a PodDisruptionBudget, watches it block drain operations correctly during a node rotation, and concludes their service is eviction-safe. It is not. Memory pressure on the same node bypasses every budget constraint because the kubelet’s eviction path never touches the policy API.

The concreteness of the PDB enforcement model is worth stating plainly. If a Deployment targets 5 replicas and the budget sets maxUnavailable to 1, the Eviction API permits removal of exactly one pod at a time (Disruptions - Kubernetes Docs). A caller attempting to evict a second pod receives a rejection until the first pod is replaced and healthy. The arithmetic is enforced per-request, not per-batch. This prevents Cluster Autoscaler from draining multiple nodes simultaneously and wiping out all 5 replicas in one operation.

Architecture diagram

Locking down voluntary eviction completely. Setting maxUnavailable: 0 or minAvailable: 100% in a PodDisruptionBudget blocks all voluntary evictions unconditionally (PodDisruptionBudget v1 API Reference). This configuration works when you need to prevent any API-driven disruption during a critical batch window. It breaks when you also need to drain nodes for maintenance, because the drain will stall indefinitely waiting for a budget that will never open. Cluster Autoscaler can enter a blocked state for hours, holding underutilized nodes at full cost, because no caller can evict any pod.

The coverage gap. PodDisruptionBudgets cover exactly the voluntary eviction surface: drains, autoscaler removals, and any tool that calls the Eviction API. They cover nothing on the involuntary side. A service with a tight budget and no soft eviction thresholds configured on the kubelet is protected against scheduled disruptions and exposed to unscheduled ones. By sprint 3 of a cluster hardening project, teams that audit only PDB coverage have typically addressed 50% of their eviction exposure without knowing it.

The gap has one concrete fix: treat PDB coverage and kubelet eviction threshold configuration as two separate audit items on the same checklist, not as alternatives. PDBs belong in every Deployment manifest. Soft thresholds and QoS class enforcement belong in the kubelet configuration. Neither substitutes for the other.

What You Should Configure Before the Next Pressure Event

Five configuration items, addressed in this order, give the kubelet enough runway to absorb a pressure event without cascading into uncontrolled kills.

Start with resource requests. A Kubernetes resource request is a declared guarantee that the scheduler uses for placement and the kubelet uses to assign a Quality of Service class to each pod (Resource Management for Pods and Containers). Without a declared request, the kubelet assigns BestEffort status and the pod becomes the first termination target under any memory pressure condition. This is not probabilistic: the QoS ranking is deterministic. BestEffort pods go first, every time. Audit every container spec, including init containers and sidecars, and confirm a nonzero requests.memory value exists before the next deployment window.

Soft threshold tuning. When only a hard eviction threshold is configured, the kubelet applies a 0 second grace period at the moment the threshold is crossed, terminating the container immediately with no SIGTERM window (Configure Out of Resource Handling). The fix is to set evictionSoft and evictionSoftGracePeriod so the kubelet enters a warning stage before the hard threshold is reached. This works when memory pressure is transient and the warning window is longer than the spike duration. It breaks when sustained utilization holds the node above the soft threshold past the grace period, because the kubelet promotes to hard and kills with 0 seconds regardless.

Hard threshold sizing. For a node with 10GiB of total memory, the threshold expressed as memory.available<10% and the threshold expressed as memory.available<1Gi are equivalent (Node-pressure Eviction). Use the absolute form on nodes with fixed memory sizes; use the percentage form when the same kubelet config is applied across instance types with differing capacities. Mixing both expressions on the same node produces the lower of the two as the effective trigger, so verify which form your kubelet config uses today.

evictionMinimumReclaim floor. The default evictionMinimumReclaim is 0 for all resources. Zero reclaim means the kubelet stops evicting the moment the node drops below the threshold by any margin, even a few kilobytes. The scheduler then places new pods, consumption rises again, and the threshold is re-crossed within seconds. Setting a nonzero floor forces the kubelet to continue evicting until it has recovered enough headroom to hold below the threshold through a normal scheduling cycle. The right floor value depends on your average pod memory footprint. A floor equal to one pod’s declared request is a reasonable starting point.

PodDisruptionBudget scope. A PodDisruptionBudget with minAvailable: 1 prevents Cluster Autoscaler from draining the last replica of a service during a scale-down event, because the autoscaler calls the Eviction API and the API enforces the budget constraint before proceeding. This configuration costs nothing at scheduling time. It breaks when the budget is set to minAvailable: 100% and a node drain is required for maintenance, because the drain stalls indefinitely waiting for a budget window that never opens.

Architecture diagram

One additional step that teams miss is vendor-specific threshold verification. Managed providers including GKE, EKS, and AKS each set default eviction thresholds in their kubelet configurations, and those defaults are not consolidated in upstream Kubernetes documentation. A threshold you set in a DaemonSet-managed kubelet config may be overridden by the provider’s node bootstrap process. After 30 days of running a tuned configuration, compare the kubelet flags on a live node against what your config management system declares. The kubelet CLI flags are the ground truth (kubelet CLI Reference). If they diverge, your tuning is not in effect.

The concrete next action: pull the kubelet config from one node per instance type in production today and verify these five items against the values above. Any field showing its default state is a live exposure.

Tagged
Riya Mittal

Riya Mittal

Engineer · Zop.Dev

Riya is an AI engineer at ZopDev, working on production LLM pipelines behind the company's content and account-intelligence platforms. She works on the engineering that makes these systems reliable and repeatable, from multi-provider orchestration and structured output validation to evals, idempotent pipelines, and automated recovery. She writes about what it takes to make AI systems reliable enough to run in production.

Stop watching the waste.
Start cutting it.

See. Find. Fix. Automatic.

Connect your first cloud account in under 5 minutes. See your first remediation in under 7. No credit card required.

CDCR continuous detection, continuous remediation
full audit every action traceable
read-only default access
Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console· Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console·