The Hidden Cost of Kubernetes DNS Defaults

The Hidden Cost of Kubernetes DNS Defaults

The mechanism starts with ndots:5, the resolver threshold baked into every Pod’s /etc/resolv.conf by default (DNS for Services and Pods). When a hostname contains fewer than 5 dots, the Linux resolver treats it as a relative name and iterates through every search domain before attempting the bare absolute name. A call to api.stripe.com has 2 dots, so the resolver first tries api.stripe.com.default.svc.cluster.local, then api.stripe.com.svc.cluster.local, then api.stripe.com.cluster.local, and so on through the full search list before it finally queries the name as written. Each attempt is a live UDP round trip. Each one that fails returns NXDOMAIN and burns a cache slot with a 30-second TTL before the entry expires and the query fires again.

Call this the Query Amplification Tax: the ratio of actual resolver queries to intended lookups. At default settings, one application-level call to an external hostname generates up to 10 DNS queries. At moderate request rates, CoreDNS CPU and conntrack table pressure accumulate from this overhead alone, not from application traffic.

The four failure modes that compound this problem are distinct and each needs a separate fix.

Default search domain expansion. A Pod’s search list includes the Pod’s own namespace, the svc subdomain, and cluster.local by default. Every unqualified or under-dotted hostname walks this list sequentially, generating NXDOMAIN responses that CoreDNS forwards upstream when its negative cache expires.

NXDOMAIN cache exhaustion. The default negative-response TTL is 30 seconds. High-churn workloads, think batch jobs or autoscaled services with rapid connection cycling, exhaust negative cache entries faster than TTLs refill them. The result is a steady stream of upstream queries for names that will never resolve, consuming both CoreDNS capacity and upstream resolver quota.

Prefetch timing pressure. CoreDNS begins prefetching a cached record when its TTL drops below 10% of the original value, or at most 1 second before expiration (CoreDNS cache plugin). Under the Query Amplification Tax, positive cache entries for real external names compete for cache space against the flood of NXDOMAIN entries, so prefetch triggers fire against a smaller effective cache, increasing upstream query rates further.

Cluster-wide ndots reduction risk. Lowering ndots to 1 or 2 without auditing every internal service call breaks short-name resolution. A service addressed as my-service relies on search domain expansion to resolve inside the cluster. Remove that expansion and the name fails. The fix is a per-workload audit before any cluster-wide change, not a flag flip on the kubelet.

By sprint 3 of a typical cluster hardening effort, teams find that DNS tuning alone, before any infrastructure change, reduces upstream query volume measurably. Start the audit with kubectl exec into a representative Pod and inspect the actual search list in /etc/resolv.conf against your external call inventory.

How Default DNS Search Domains Multiply Every Lookup

The search domain list built into every Pod is not a convenience feature. It is a query multiplier, and its hard limits exist precisely because unconstrained expansion would make DNS unusable at scale.

A Pod’s DNS configuration is a resolved product of cluster defaults (DNS for Services and Pods). The resolver writes three baseline entries into /etc/resolv.conf automatically: the Pod’s own namespace, the svc subdomain, and cluster.local. Those three entries exist so that short service names like payments resolve inside the cluster without requiring a fully qualified address. The design assumption is that most traffic is intra-cluster. External calls are the exception the defaults were not built for.

The 32-domain ceiling and the 2048-character total length limit are the guardrails Kubernetes enforces once a custom dnsConfig is applied. Neither limit activates on the default three-entry list. They become relevant when operators append corporate resolver domains, environment-specific suffixes, or legacy internal zones to the Pod spec. Each appended domain adds one more sequential candidate the resolver must exhaust before treating a name as absolute. At 32 entries, a hostname with fewer than 5 dots generates 32 failed attempts plus one final absolute query, before any real answer arrives.

The Search Domain Expansion Model formalizes this: every query for a relative name produces N plus 1 resolver round trips, where N is the number of search domains tried before the absolute attempt succeeds or fails. The default three-entry list produces up to 10 queries per external lookup because ndots:5 forces the resolver through all search candidates first. Each candidate query returns NXDOMAIN. Each NXDOMAIN consumes a negative cache slot with a 30-second TTL. The upstream resolver sees all of them.

The three structural properties of this design each carry a specific failure condition.

Sequential exhaustion. The resolver walks the search list in order, left to right, and only attempts the absolute name after every entry fails. This means lookup latency for external FQDNs scales linearly with search list length, not with network quality. A well-peered upstream resolver still waits while local NXDOMAIN queries complete.

Length ceiling interaction. The 2048-character limit on total search domain string length bites before the 32-domain count does in clusters with long zone names. A single domain like production.us-east-1.internal.corp.example.com consumes a significant number of characters toward the total length budget. Enough long domains can exhaust the total character budget of 2048 before the 32-count ceiling is reached, so the effective limit is whichever constraint is hit first (DNS for Services and Pods - Kubernetes Documentation).

Namespace proliferation side effect. A Pod’s search list includes its own namespace by default. In clusters with many namespaces and cross-namespace service calls, the wrong namespace entry generates an NXDOMAIN before the correct one is tried. After 30 days of data collection in a multi-tenant cluster, the namespace mismatch pattern typically surfaces as a consistent spike in NXDOMAIN rates correlated with cross-team service dependencies.

Audit the actual search list in a running Pod before adjusting any resolver setting. The list in /etc/resolv.conf reflects the merged result of cluster defaults and any dnsConfig overrides, and the two sources are not always consistent with what the team believes is deployed.

Four Failure Patterns Teams Hit When Tuning DNS

Four failure patterns account for the majority of DNS incidents in production Kubernetes clusters, and each one has a distinct mechanical cause that determines the correct fix.

ndots misconfiguration. The ndots:5 default forces the Linux resolver to test every search domain suffix before treating an external hostname as absolute (DNS for Services and Pods). A call to api.stripe.com has 2 dots, so the resolver produces up to 10 queries before the name resolves correctly. The instinct is to lower ndots to 1 or 2 cluster-wide. That fix works for external traffic. It breaks internal traffic because short service names like my-service depend on search domain expansion to resolve inside the cluster. The correct approach is a per-workload audit before changing the value anywhere, confirming that no internal call relies on unqualified name expansion.

NXDOMAIN cache exhaustion. The default negative-response TTL in CoreDNS is 30 seconds (CoreDNS cache plugin). Batch jobs and autoscaled workloads with rapid connection cycling generate NXDOMAIN responses faster than TTLs refill cache slots. The upstream resolver absorbs every cache miss directly. Negative caching can be re-enabled with NodeLocal DNSCache, which keeps negative entries local to the node and reduces queries reaching the central CoreDNS service. This fails when the node agent itself is misconfigured or the cluster.local domain is not correctly targeted, because queries bypass the local cache entirely and return to the same exhaustion pattern.

NodeLocal DNSCache contention. NodeLocal DNSCache improves cluster DNS performance by running a caching agent on each node, eliminating conntrack table contention on UDP packets destined for the central CoreDNS service. The contention exists because Linux conntrack tracks each UDP DNS query as a connection, and high query rates saturate the conntrack table, causing packet drops that surface as intermittent resolution failures. The node-local agent intercepts queries for cluster.local before they hit the conntrack-tracked path. When serve_stale is configured, the cache serves an expired entry for up to 1 hour before declaring the name unresolvable, which provides continuity during brief upstream outages. This breaks when the DaemonSet rollout is incomplete: nodes without the agent fall back to the central service and reintroduce the contention the deployment was meant to eliminate.

Windows Pod resolution gaps. Windows-based Pods do not share the Linux resolver stack that the preceding three patterns target. Search domain expansion, ndots behavior, and the NodeLocal DNSCache DaemonSet all assume Linux networking primitives. Windows Pods resolve DNS through a separate path with different defaults, so tuning applied at the cluster level or via dnsConfig on Linux workloads produces no effect on Windows Pods. Teams discover this in the first deployment week after enabling NodeLocal DNSCache cluster-wide: Windows workload latency is unchanged, and the fix requires separate per-Pod dnsConfig entries scoped explicitly to Windows node pools.

Start by identifying which pattern is active. Pull NXDOMAIN rates from CoreDNS metrics, check conntrack usage on nodes with the highest DNS query volume, and confirm NodeLocal DaemonSet coverage before adjusting any resolver setting.

The Interaction Traps No Single Source Documents

No single piece of official documentation covers what happens when ndots settings, the autopath plugin, and a service mesh DNS proxy operate simultaneously on the same query path. The interaction zones between these three mechanisms are where production clusters accumulate silent, compounding failure.

The core problem is that each component assumes exclusive authority over query transformation. The Linux resolver applies ndots rules to decide whether to expand a name against the search list. CoreDNS autopath intercepts that expanded query at the server and rewrites it to the absolute form, returning a CNAME chain so the client skips redundant suffix attempts (CoreDNS autopath plugin). A service mesh DNS proxy, such as Istio’s, intercepts queries before they reach CoreDNS entirely, applying its own resolution logic. When all three are active, a single query passes through three independent transformation layers, and none of them has visibility into what the others are doing.

autopath and negative caching conflict. CoreDNS enables negative caching in its default deployment, while kube-dns does not (CoreDNS vs kube-dns). autopath short-circuits the client’s search domain expansion by rewriting the first query to the absolute name. That is the intended behavior. The conflict emerges when the absolute name does not exist: autopath returns NXDOMAIN, negative caching stores it for 30 seconds, and the client receives a cached failure on retry. Without autopath, the client would have tried the next search domain suffix. With autopath active, the client stops at the first NXDOMAIN and never reaches a suffix that would have resolved correctly.

ndots reduction with autopath active. Lowering ndots to 1 or 2 without auditing internal service calls breaks cluster-internal resolution because short names no longer trigger search domain expansion. When autopath is also enabled, the failure mode shifts: autopath expects to intercept a suffix-expanded query and rewrite it. If ndots suppresses expansion entirely, autopath receives an absolute name, finds no match, and returns NXDOMAIN immediately. By sprint 3 of a cluster hardening project, teams applying this combination typically see internal service calls fail silently, with no search list fallback and no retry.

Service mesh interception position. A service mesh DNS proxy intercepts queries before they reach CoreDNS. It resolves names for mesh-enrolled services locally and forwards everything else. The problem is ordering: ndots rules apply inside the Pod before the query leaves the network namespace, autopath applies at the CoreDNS server, and the mesh proxy sits between them. If the mesh proxy forwards an already-expanded name to CoreDNS, autopath attempts a second rewrite on a name that is already absolute. The result is a malformed CNAME chain or a query that never reaches the upstream resolver. The Kubernetes DNS configuration documentation does not address this three-layer interaction.

The named risk here is the Triple Interception Problem: three components each performing query transformation without shared state. The fix is not to tune all three simultaneously. Disable autopath before reducing ndots. Confirm mesh proxy forwarding behavior against a known-good resolution path. Then reintroduce each component one at a time, with CoreDNS query logging active, verifying that NXDOMAIN rates and upstream query counts stay stable after each change.

Actionable Recommendations for Safer DNS Tuning

DNS tuning without a prior audit is remediation theater. The four steps below are sequenced by blast radius: the ones that break the fewest workloads when misconfigured come first.

Audit search domain exposure before touching ndots. A pod’s default DNS search list includes its own namespace and the cluster’s default search domains (DNS for Services and Pods). Kubernetes enforces a hard ceiling of 32 search domains and 2048 total characters across all of them. Staying inside those limits is not the goal. The goal is knowing which workloads issue short, unqualified names that depend on search domain expansion. Collect that inventory first. This works when you have a service registry or can inspect application configuration. It breaks when service names are assembled at runtime from environment variables, because static analysis misses those call sites entirely.

Add trailing dots to external FQDNs before reducing ndots. Each missing trailing dot on an external hostname costs up to 10 extra DNS queries per lookup, because ndots:5 causes the resolver to try every search domain suffix before treating the name as absolute. The fix is adding the trailing dot at the application or configuration level, making the name explicitly absolute. After 30 days of data showing stable NXDOMAIN rates and no increase in upstream query volume, reducing ndots to 1 or 2 is safe for workloads confirmed to use fully qualified external names. Lowering ndots without the trailing-dot audit first produces the same broken internal resolution described in the previous section.

Raise the negative-response cache TTL above the 30-second default. CoreDNS caches NXDOMAIN responses for only 30 seconds by default (CoreDNS cache plugin). High-churn workloads exhaust those slots, and every miss goes directly to the upstream resolver. Increasing the negative TTL in the CoreDNS ConfigMap reduces upstream pressure by keeping negative entries in the local cache longer. This fails for workloads where a name that returned NXDOMAIN genuinely becomes valid quickly, because a longer TTL delays successful resolution for the duration of the cached negative entry.

Deploy NodeLocal DNSCache only after confirming DaemonSet coverage is complete. NodeLocal DNSCache removes conntrack table contention on UDP by routing cluster.local queries to a node-local agent instead of the central CoreDNS service (NodeLocal DNSCache). Partial rollout is the specific failure condition: nodes missing the agent fall back to the central service, reintroducing conntrack saturation on exactly the nodes under the highest query load.

Start with the CoreDNS query log. Filter for the top 20 names by NXDOMAIN count, check whether each has a trailing dot, and you have the exact list of FQDNs to fix before any cluster-wide setting changes.