ResourceQuotas with at least one resource fully used up

What happens once a namespace hits its quota

A ResourceQuota caps what a namespace may consume: CPU and memory requests and limits, storage, or counts of objects such as pods, Services and LoadBalancers. The resource quotas documentation says that when creating or updating a resource would violate a quota, the control plane rejects the request with HTTP 403 Forbidden and a message naming the constraint.

The failure is easy to misread. A Deployment that asks for more than the quota allows is still created; the same page explains that the Deployment may simply fail to get all of its pods, and you have to inspect its status to see why. An exhausted quota therefore shows up as replicas that never appear, a rollout that hangs or an HPA that cannot add pods, not as a clear error in the CI log. Existing workloads keep running, because quota changes and contention do not affect objects already created.

Reading used against hard

kubectl describe resourcequota -A

Each quota prints a Used and Hard column per resource. For a quick filter on count-style entries, which are plain integers:

kubectl get resourcequota -A -o json | jq -r '
  .items[] | .metadata as $m | .status as $s
  | ($s.hard // {}) | to_entries[]
  | select(($s.used[.key] // "") | test("^[0-9]+$"))
  | select(.value | test("^[0-9]+$"))
  | select((.value | tonumber) > 0 and ($s.used[.key] | tonumber) >= (.value | tonumber))
  | "\($m.namespace)/\($m.name) \(.key) \($s.used[.key])/\(.value)"'

CPU and memory entries use suffixes such as m and Gi, so read those from the describe output.

How ZopNight scores the quota

For every resource that appears in both hard and used, ZopNight converts the two values from Kubernetes quantity notation and computes used divided by hard. The quota fires when the worst ratio is 1.0 or more. One exhausted resource is enough, even if everything else in the quota is nearly empty. There is no lookback window; the ratio is read fresh each evaluation.

Entries it ignores

Resources listed in hard with no matching used value, values it cannot parse, and entries whose hard limit is zero are skipped. That last point matters: a quota such as services.loadbalancers: "0", used to forbid an object type entirely, is not reported as exhausted. Quotas between 85% and 100% are reported by ResourceQuota near limit instead.

Blocked deploys rather than dollars

This high-severity governance finding has no savings estimate. The cost is delivery: releases and autoscaling in the namespace stop until someone frees capacity or raises the limit.

Freeing or raising the quota

1. Run kubectl describe resourcequota -n <namespace> to find the resource at its limit.
2. List what consumes it, for example kubectl get pods -n <namespace> for a pods count, or the largest requests for requests.cpu.
3. Remove what is finished: failed Jobs, idle Deployments, unused claims.
4. If the demand is real, raise the value under spec.hard with kubectl edit resourcequota, keeping cluster capacity in mind, since quotas are independent of how much the nodes can supply.