# The Hidden Cost of Ignoring QoS Classes in Production
> The kubelet's eviction logic runs continuously, and the classification rules that determine which pod dies first are set at pod creation, not at the moment of pressure. By the time a node crosses its…
Source: https://zop.dev/resources/blogs/kubernetes-qos-classes-explained-which-pods-get-killed-first
Published: 2026-11-10 · Author: muskan-bandta · Tags: kubernetes
---
## The Hidden Cost of Ignoring QoS Classes in Production
The kubelet's eviction logic runs continuously, and the classification rules that determine which pod dies first are set at pod creation, not at the moment of pressure. By the time a node crosses its memory threshold, the outcome is already decided.
Kubernetes QoS class is a pod-level designation, assigned automatically from the resource configuration of every container in that pod, that controls the kubelet's eviction priority order and the Linux OOM killer's target selection under node memory pressure ([Kubernetes Documentation: Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)).
**The classification trap.** A pod receives Guaranteed class only when requests and limits are specified for every container and those values are equal ([Kubernetes Documentation: Node-pressure Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/)). One container missing a limit, one container with a request below its limit, and the entire pod drops to Burstable. Teams that configure limits without explicit requests assume they get Guaranteed. They do not, because Kubernetes does not silently equalize requests to limits in all contexts. By sprint 3 of a typical rollout, those pods are misclassified without any warning in kubectl describe.
**The eviction order you cannot see.** The kubelet evicts BestEffort pods and Burstable pods whose memory usage exceeds their requests before it touches Guaranteed pods or Burstable pods consuming less than their requests. A pod with no requests set receives BestEffort class and sits at the front of the eviction queue regardless of business criticality. The mechanism is explicit: the kubelet sorts candidates by QoS class, then by distance above the request threshold.
**The OOM kill ranking.** At the Linux kernel level, the kubelet sets an oom_score_adj of -997 for every container in a Guaranteed pod. A score of -997 makes the OOM killer almost certain to bypass that container. BestEffort containers receive no such protection. When memory pressure accelerates faster than the kubelet's 10-second nodeStatusUpdateFrequency cycle, the OOM killer acts before eviction logic runs, and unprotected containers are the first targets.
**The priority blind spot.** QoS class and PriorityClass are separate axes. A critical application pod with Guaranteed QoS but no PriorityClass set loses to a lower-priority system pod that has an explicit priority value, because the kubelet checks QoS first but priority overrides class-based ranking in certain eviction paths ([Kubernetes Documentation: Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)). Teams that rely on QoS alone ship an incomplete safety model.
The fix is to audit every workload's effective QoS class before the first capacity event, not after. Run `kubectl get pods -o custom-columns` with the `.status.qosClass` field across every namespace. Any BestEffort result on a revenue-critical service is a misconfiguration, and the nodeStatusUpdateFrequency of 10 seconds means you have a narrow window to act once a node starts degrading.
## How Kubernetes Assigns QoS Classes and Why the Rules Surprise Teams
Kubernetes assigns QoS classes through a strict, deterministic ruleset, and the edge cases in that ruleset cause misclassification far more reliably than teams expect.
The classification logic works as a process of elimination. A pod reaches Guaranteed only when every container specifies both CPU and memory requests and limits, and every request value equals its corresponding limit ([Kubernetes Documentation: Configure Quality of Service for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)). That is a conjunction of conditions across every container in the pod. A pod reaches BestEffort only when no container sets any request or limit at all. Everything else falls into Burstable: specifically, any pod that does not meet the Guaranteed criteria but has at least one container with a memory or CPU request or limit.
The limits-without-requests trap is the most common classification mistake in production configurations.
**The limits-only misconfiguration.** Setting only resource limits without explicit requests places a pod into Burstable class, not Guaranteed. Some admission controllers and LimitRange policies do inject requests equal to limits automatically. When that admission-time injection does not apply, Kubernetes does not equalize them silently, so the pod lands in Burstable. The team sees limits defined, assumes the pod is protected, and finds out otherwise after 30 days of stable operation ends in a pressure event.
**The no-request default trap.** A pod with zero resource requests and zero limits receives BestEffort class. The kubelet evicts BestEffort pods before Burstable pods consuming less than their requests, and before Guaranteed pods entirely. A single container in a multi-container pod left without requests pulls the whole pod into BestEffort or Burstable. There is no partial class protection at the container level.
**The Burstable split.** Burstable is not a single eviction risk level. The kubelet evicts Burstable pods whose memory usage exceeds their requests before it evicts Burstable pods whose usage stays below requests. Two pods in the same QoS class face different eviction timing based on runtime consumption, not just configuration. A pod with requests set to a low value is Burstable on paper but behaves like BestEffort under load once usage climbs past the request floor.
**The PriorityClass gap.** QoS class and PriorityClass are independent control surfaces. A pod with Guaranteed QoS but no PriorityClass set does not automatically win against a lower-QoS pod that carries an explicit priority value. The kubelet checks QoS ordering, but priority overrides that ordering in certain eviction paths. Relying on QoS alone without PriorityClass produces an incomplete eviction model.
The table below maps the four common configuration states to their resulting QoS class and eviction position.
| Configuration State | Resulting QoS Class | Eviction Position |
|---|---|---|
| Requests and limits set, equal, all containers | Guaranteed | Last |
| Requests and limits set, unequal, any container | Burstable | Middle (usage-dependent) |
| Limits only, no explicit requests, no LimitRange injection | Burstable | Middle (usage-dependent) |
| No requests, no limits, any container | BestEffort | First |
The classification decision is frozen at pod admission. No runtime behavior changes the QoS class after the pod starts. That means a misconfigured pod carries its eviction risk from the first deployment week through every subsequent capacity event. Inspect `.status.qosClass` on every workload before the next node pressure event, not during it.
## Eviction Ordering, PriorityClass, and the Gaps Teams Fall Into
QoS class determines the default eviction order, but PriorityClass rewrites that order at runtime, and teams that treat the two as equivalent ship a system that fails predictably [under pressure](https://zop.dev/resources/blogs/static-runbooks-vs-llm-driven-playbooks-what-breaks-at-3-am).
The kubelet's eviction sequence starts with QoS rank, but that rank is not final. When a pod carries an explicit PriorityClass, the scheduler and kubelet consult the priority integer, not the QoS bucket, to resolve conflicts in certain paths. Kubernetes ships two built-in PriorityClasses out of the box: `system-cluster-critical` and `system-node-critical`. These exist specifically because critical infrastructure pods must survive node pressure regardless of their memory footprint or QoS assignment.
The `preemptionPolicy` field defaults to `PreemptLowerPriority`, which authorizes pods of that PriorityClass to terminate lower-priority pods to claim node resources. That default is active unless you explicitly set the field otherwise. A team that provisions a high-priority application pod without reviewing `preemptionPolicy` inherits preemption behavior they may not have intended.
**The eviction-hard threshold gap.** Conservative `eviction-hard` thresholds cause pod churn before genuine resource exhaustion occurs. The kubelet begins evicting BestEffort pods the moment a node crosses the configured memory threshold, applying a 0-second grace period for immediate shutdown. No drain window, no SIGTERM delay. If that threshold is set too high as a safety buffer, routine memory fluctuations trigger evictions against workloads that would have recovered without intervention. The mechanism is binary: threshold crossed equals eviction starts, with no consideration of trajectory or rate of change.
**The priority override blind spot.** A Guaranteed pod without an assigned PriorityClass does not automatically outlast a Burstable pod that carries a high priority integer. The kubelet checks QoS first, but priority overrides class-based ranking in preemption paths. Critical application pods left without PriorityClass assignment lose to lower-QoS system components that carry explicit priority values. This failure mode surfaces specifically when teams secure QoS classification carefully but skip the second control surface entirely.
**The critical pod admission guarantee.** Critical pods are admitted even under active resource pressure because the runtime treats them as required for system stability ([kubernetes/kubernetes eviction_manager.go](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/eviction_manager.go)). This admission guarantee applies to pods using `system-cluster-critical` or `system-node-critical`. Application pods that need similar protection must claim it explicitly through a custom PriorityClass. Assuming the system will protect a business-critical pod without an assigned priority class is an operational gap, not a configuration nuance.
The interaction between QoS class and PriorityClass produces four distinct eviction risk states, shown below.
| QoS Class | PriorityClass Set | Eviction Risk |
|---|---|---|
| Guaranteed | Yes (high value) | Lowest: class and priority both protect |
| Guaranteed | No | Partial: QoS protects, priority gap exposed |
| Burstable | Yes (high value) | Mixed: priority protects, QoS rank is vulnerable |
| BestEffort | No | Highest: evicted first, 0s grace period applied |
The fix is to audit PriorityClass coverage alongside QoS class. Start by listing every pod where `.status.qosClass` is Guaranteed and `.spec.priorityClassName` is empty. Those pods carry half the protection they need.
## Memory QoS and the memoryThrottlingFactor Most Teams Leave Unset
Memory QoS is a cgroup v2 feature that introduces a proactive throttling boundary between normal container operation and the OOM killer, and most clusters never activate it because `memoryThrottlingFactor` defaults to nil.
The default nil value means the kubelet does not write `memory.high` into the container's cgroup. Without `memory.high`, the kernel has no throttling boundary to enforce. Memory usage climbs unchecked until it hits the hard `memory.max` ceiling, at which point the OOM killer fires immediately. [CPU limits](https://zop.dev/resources/blogs/cpu-throttling-in-java-go-and-node-containers) are enforced by [CPU throttling](https://zop.dev/resources/blogs/cpu-throttling-is-lying-to-you-why-your-container-limits-bleed-latency), while memory limits are enforced by the kernel with OOM kills ([Kubernetes Documentation: Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)). Those two enforcement mechanisms are not equivalent: throttled CPU degrades performance gradually, but an OOM kill terminates the process instantly with no recovery window.
`memory.high` is the cgroup v2 soft limit, defined as the threshold at which the kernel begins actively throttling memory allocation and reclaiming pages before usage reaches the hard boundary. Setting it gives the runtime a pressure signal it can act on. Leaving it unset collapses the two-stage response into one: OOM kill with no preceding warning.
The `memoryThrottlingFactor` field translates into a concrete `memory.high` value using the container's limit and request. With `memoryThrottlingFactor` set to 0.9, a container with a 256 MiB request and a 1 GiB limit gets `memory.high` set to roughly 947 MiB. The mechanism is: kubelet multiplies the limit by the factor, producing a soft ceiling the kernel enforces before OOM territory begins. With `memoryThrottlingFactor` set to 0.9, a container with a 256 MiB request and a 1 GiB limit gets `memory.high` set to roughly 947 MiB, and the [gap between](https://zop.dev/resources/blogs/cloud-rightsizing-without-breaking-production-gp2-to-gp3-ec2-cpu-baselines-and-g) that throttling threshold and the 1 GiB hard limit is the kernel's reclaim window, the space in which memory pressure is visible and actionable rather than immediately fatal.
**The nil default gap.** When `memoryThrottlingFactor` is nil, the kubelet skips writing `memory.high` entirely. The container runs with only a hard memory ceiling. Any spike in allocation goes straight to OOM kill territory with no intermediate throttling signal. The first sign of a memory problem is a terminated process, not a slowdown that an alert could have caught by sprint 3.
**The factor-to-limit arithmetic.** The throttling factor applies against the container limit, not the request. A low limit with a 0.9 factor produces a tight soft ceiling. A high limit with the same factor leaves a wide reclaim window. Teams that set large limits as a defensive measure inadvertently raise the `memory.high` threshold, reducing the buffer between normal operation and OOM territory. The fix is to size limits based on actual working set data, then apply the factor deliberately.
**The QoS class dependency.** Memory QoS throttling activates specifically for pods in the Burstable and Guaranteed QoS classes, because those pods have memory requests or limits set. BestEffort pods, which have no memory request or limit, are outside the `memory.high` calculation entirely. Setting `memoryThrottlingFactor` on a cluster where BestEffort workloads run alongside Burstable workloads produces uneven protection across the node.
| Scenario | memory.high Set | First Failure Signal |
|---|---|---|
| memoryThrottlingFactor nil | No | OOM kill, no warning |
| memoryThrottlingFactor 0.9, 1 GiB limit | 947 MiB | Kernel throttle, process slows |
| BestEffort pod, any factor | No | Eviction, not throttling |
Set `memoryThrottlingFactor` to 0.9 in the kubelet configuration on every node running Burstable or Guaranteed workloads. Then verify `memory.high` is populated by reading the container's cgroup directory on a live node. An unset value after the kubelet restarts means the cgroup v2 hierarchy is not active, and the node is running cgroup v1 where `memory.high` does not exist.
## Actionable Recommendations for Stable QoS Configuration in Production
Four production steps close the [gap between](https://zop.dev/resources/blogs/ci-cd-readiness-checklist) accidental QoS assignment and deliberate, stable configuration: audit current class assignments, fix misclassification at the spec level, pair QoS with correctly tuned eviction thresholds, and make an explicit decision about Memory QoS now that it is beta in Kubernetes v1.27 and enabled by default in v1.37.
Start the audit before touching any configuration. Run a query across all namespaces that lists every pod's `.status.qosClass` alongside its container resource specs. The output will surface three [failure patterns](https://zop.dev/resources/blogs/kubernetes-mttr-43-minutes-to-9-structured-runbooks). First, pods with no requests or limits land in BestEffort because Kubernetes requires both values to be non-zero for any higher class assignment. Second, pods with only limits set land in Burstable rather than Guaranteed because Kubernetes defaults requests to equal limits only under specific admission conditions, and that default does not apply universally. Third, pods with mismatched requests and limits land in Burstable regardless of how close the values are, because Guaranteed requires exact equality across every container in the pod. The audit produces a prioritized remediation list in the first sprint.
**QoS misclassification at the spec level.** A pod is Burstable when at least one container sets a memory or CPU request or limit but the pod does not meet the Guaranteed criteria. The fix is to set requests equal to limits for every container in pods that must be Guaranteed. This works when the workload has a stable, measured memory footprint. It breaks when the team sets artificially high limits to compensate for unknown memory behavior, because Kubernetes then schedules the pod based on those inflated requests, reducing node density.
**Eviction threshold calibration.** The kubelet evicts BestEffort pods immediately when a node crosses the `eviction-hard` memory threshold, with no consideration of whether usage is rising or stabilizing. Setting that threshold too conservatively produces pod churn during routine memory fluctuations. The correct approach is to set thresholds based on measured node memory variance over at least 30 days of production data, then add a buffer sized to your largest single-container memory footprint. This works when workload memory usage is statistically consistent. It breaks when a new workload is introduced mid-cycle, because the historical baseline no longer reflects current node behavior.
**OOM score and eviction ordering.** For Burstable pods, the kubelet calculates `oom_score_adj` as `min(max(2, 1000 - (1000 x memoryRequestBytes) / machineMemoryCapacityBytes), 999)` ([Kubernetes Documentation: API-initiated Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/eviction-policy/)). A pod requesting a small fraction of node memory receives an `oom_score_adj` close to 999, making it a primary OOM kill target. Increasing the memory request reduces the score and lowers eviction priority. Teams that set symbolic 10 MiB requests on production services to satisfy admission webhooks inadvertently push those services to the top of the OOM kill queue.
**Memory QoS enablement decision.** Memory QoS reached beta in Kubernetes v1.37 and is enabled by default. On clusters running cgroup v2, the decision is not whether to evaluate it but whether the default `memoryThrottlingFactor` configuration matches your workload profiles. BestEffort pods receive no `memory.high` protection regardless of cluster-level settings, because the calculation requires a memory request or limit. Clusters mixing BestEffort and Burstable pods get uneven protection across the node, and the BestEffort workloads remain first in the eviction sequence.
| Audit Check | Failure Indicator | Remediation |
|---|---|---|
| `.status.qosClass` on production pods | BestEffort where Guaranteed expected | Set requests and limits, equal values |
| Limits set, requests absent | Burstable assignment, unpredictable eviction | Explicitly set requests equal to limits |
| `oom_score_adj` on high-request pods | Score near 999 despite large request | Increase memory request to reflect actual working set |
| `memory.high` in container cgroup | Empty value on cgroup v2 node | Set `memoryThrottlingFactor` in kubelet config, restart kubelet |
The concrete next action: read `.status.qosClass` for every pod in your production namespace today, cross-reference against the pods that carry no PriorityClass, and treat any Guaranteed pod with an empty `priorityClassName` as a misconfigured pod. It carries one of two required protections, not both.
