Reduce Kubernetes Costs: Measure Requests, Then Cut Nodes

On EKS, GKE and AKS the bill is mostly nodes, and the number of nodes is driven by resource requests, not by actual usage: kube-scheduler places Pods by their requests, and the cluster autoscaler adds nodes when Pods sit Pending because no node has room for their requests. A cluster whose Pods request four times what they use pays for four times the nodes. This procedure measures that gap, closes it namespace by namespace, and then removes the idle capacity.

Step 1: See how full the nodes are on paper

kubectl describe nodes | grep -A 6 "Allocated resources"

Each node reports CPU and memory requests as a percentage of what it can allocate to Pods. Nodes sitting at 80% requested are full as far as the scheduler is concerned, even if the processes on them are idle.

Step 2: Compare against real usage

kubectl top reads the Metrics API, which the metrics-server add-on provides. Install it if the command errors.

kubectl top nodes
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory --containers

Put the two views side by side per workload: the request from its spec, the usage from kubectl top. kubectl top is a point-in-time sample, so collect it across a normal working day (or use your monitoring stack’s history) before deciding anything.

Step 3: Right-size the biggest over-requesters

Start with the Pods whose request is far above observed peak usage and lower the request with headroom left for bursts:

kubectl set resources deployment checkout -n shop \
  --requests=cpu=250m,memory=384Mi --limits=memory=512Mi

This rolls out new Pods with the new values. Keep memory requests above real peak usage: the kubelet evicts Pods that exceed their requests first when a node runs short of memory. The companion guide fixing Pods with no requests covers workloads at the other extreme.

Step 4: Put a ceiling on each namespace

A ResourceQuota caps the total a namespace can request, which stops one team’s defaults from inflating the node count:

kubectl create quota team-a-compute -n team-a \
  --hard=requests.cpu=8,requests.memory=16Gi,limits.memory=24Gi
kubectl describe quota team-a-compute -n team-a

Once a quota covers CPU and memory, Pods that set no requests for them can be rejected, so pair it with a LimitRange that injects defaults (see step 2 of the requests guide above).

Step 5: Let the autoscaler remove what you freed

Lower requests only save money when nodes go away. Check that a cluster autoscaler is enabled on each pool and that its minimum is not holding spare nodes. On GKE you can bias the autoscaler toward packing:

gcloud container clusters update dev-cluster --location europe-west1 \
  --autoscaling-profile optimize-utilization

The default profile is balanced. AKS and EKS expose their own autoscaler settings; the principle is the same: the minimum node count is capacity you pay for around the clock.

Step 6: Turn off what nobody uses at night

For development and test clusters the remaining lever is time. Stop whole AKS clusters, empty EKS node groups or GKE node pools, or scale single namespaces to zero. The procedures are in stopping clusters on each provider and scheduling one namespace. The EKS and GKE control-plane fee of $0.10 per cluster per hour continues either way, so merging several small, mostly idle clusters into one also removes fees.

Verify the effect

Re-run step 1 a day after each change. Requested percentages per node should fall, then the node count should follow as the autoscaler consolidates. If requested percentages fall but the node count does not, look at the autoscaler minimums, at Pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false", and at PodDisruptionBudgets that a node’s removal would exceed; GKE documents all three as reasons the autoscaler keeps a node.

With ZopNight

ZopNight’s Kubernetes workload rules flag containers missing requests or limits, degraded and single-replica workloads, idle workloads such as stopped Deployments and suspended CronJobs, unbound PVCs and released PVs, and CPU over-provisioning. Node pools and namespaces can then go on start and stop schedules. Showback then attributes resource cost to teams and tags. See Kubernetes workload rules for the rule families.