Drain and Resize Kubernetes Node Pools Without Losing Pods

Shrinking a node pool is easy. Shrinking it without deleting local data, stranding a stateful app or fighting the autoscaler takes a few checks first. This page is the checklist and the drain sequence. The provider-specific resize commands live in stopping a cluster on EKS, GKE and AKS and scaling GKE pools to zero.

Pre-flight: four things to check on the pool

1. Which pool is the system pool. Some pools can never reach zero. On AKS, system node pools need at least two nodes and only user pools can go to zero. On GKE Standard, the autoscaler always keeps at least one node in the cluster for system Pods. List pools and their mode or labels before choosing targets:

az aks show -g rg-dev -n dev-aks \
  --query "agentPoolProfiles[].{Name:name,Mode:mode,Count:count}" -o table

kubectl get nodes -L cloud.google.com/gke-nodepool,kubernetes.azure.com/agentpool

2. What stores data on the node. emptyDir volumes are deleted permanently when a Pod leaves its node, so any workload that keeps scratch state there loses it on every drain. Find them:

kubectl get pods -A -o json | jq -r '.items[]
  | select(any(.spec.volumes[]?; has("emptyDir")))
  | "\(.metadata.namespace)/\(.metadata.name) node=\(.spec.nodeName)"'

PersistentVolumeClaims are not deleted by a drain or a resize. The disks behind them stay in place for the Pod to use when it is rescheduled, and on AWS and Google Cloud they keep billing for their provisioned size while the pool is empty.

3. Which Pods have no controller. A drain evicts bare Pods and nothing recreates them. kubectl drain refuses to proceed past them unless you pass --force, which is your cue to move them under a Deployment first.

4. What the PodDisruptionBudgets allow.

kubectl get pdb -A

While a PDB shows ALLOWED DISRUPTIONS of 0, the API server answers eviction requests for its Pods with 429 Too Many Requests and the drain keeps retrying. That is the point of a PDB, but it also means a scheduled scale-down can stall. For a pool you are emptying entirely, every replica has to go anyway, so decide per workload whether to relax the PDB during off-hours or keep that workload on a pool that stays up.

The drain sequence

Cordon first so nothing new lands, then drain each node, then resize the pool.

POOL=apps-pool
for n in $(kubectl get nodes -l cloud.google.com/gke-nodepool=$POOL -o name); do
  kubectl cordon "$n"
done

for n in $(kubectl get nodes -l cloud.google.com/gke-nodepool=$POOL -o name); do
  kubectl drain "$n" --ignore-daemonsets --delete-emptydir-data --timeout=10m
done

--ignore-daemonsets is required when DaemonSet Pods are present; the DaemonSet controller would recreate them anyway. --delete-emptydir-data acknowledges the data loss from check 2. A drain that uses the Eviction API respects PDBs; --disable-eviction bypasses them and should not be in a scheduled job. For AKS, swap the label selector for kubernetes.azure.com/agentpool=POOL.

Draining yourself matters most on EKS: when a managed node group’s desired count drops, EKS does not honour PDBs and terminates a node 15 minutes after it starts evicting, whether or not its Pods have left.

Stop the autoscaler from fighting you

Evicted Pods become Pending, and a cluster autoscaler will add nodes to place them. Take the pool out of the autoscaler’s hands for the night:

# AKS: required before a user pool can be scaled to 0
az aks nodepool update -g rg-dev --cluster-name dev-aks -n apps --disable-cluster-autoscaler
az aks nodepool scale  -g rg-dev --cluster-name dev-aks -n apps --node-count 0

# GKE
gcloud container clusters update dev-cluster --no-enable-autoscaling \
  --node-pool apps-pool --location europe-west1

On AKS you can instead set --min-count 0 on an autoscaled user pool, which allows the autoscaler to reach zero but does not force it. On EKS, AWS advises not to edit desiredSize directly while the Cluster Autoscaler is running.

Restore and verify

Scale the pool back to its recorded size and re-enable the autoscaler with its original bounds. The replacement nodes join schedulable, so there is nothing to uncordon. Confirm:

kubectl get nodes -L cloud.google.com/gke-nodepool,kubernetes.azure.com/agentpool
kubectl get pods -A --field-selector=status.phase=Pending
kubectl get pdb -A

Nodes should be Ready, the Pending list empty, and each PDB back to its normal allowed disruptions once replicas are healthy.

Where ZopNight fits

For node pools ZopNight runs the provider operation for you (EKS node pool scaling, GKE and AKS start and stop) from a schedule, and resources in a group start in tiers (storage, then databases, then compute, then apps) and stop in reverse, with a per-resource delay on top. If the goal is to quiet one team’s workloads rather than a pool, its namespace-level stop scales that namespace’s Deployments and StatefulSets to zero and removes the HPA and PDB during the stop, restoring them on start; see the namespace schedule page.