Why K8s is hard to cost-optimize

WHY K8S IS HARD:
  1. Cost lives at the wrong level (node bill ≠ workload responsibility)
  2. Schedule-stop the cluster loses everything (including prod)
  3. Pod-hours ≠ instance-hours (decoupled time accounting)

COST BILLING                          WORKLOAD CONTEXT
─────────────────────────────────────────────────────────────────
EC2 m5.large × 8 in EKS nodegroup     prod-payment-api: 4 pods
                                       ml-training-job: 1 pod
                                                       (resource-hungry)
                                       analytics-dashboard: 2 pods
                                       sidecar-logger: 7 pods
                                                       (1 per pod)

THE BILL SAYS:
  "8× m5.large = ~$561/month"

THE WORKLOAD TRUTH:
  ml-training-job's single pod requests a whole node (2 vCPU)
  A pod cannot span nodes, so that node is effectively its own
  The bill doesn't tell you this

WITHOUT POD-LEVEL ATTRIBUTION:
  You can't tell which workload is driving the cost
  Showback per team breaks down
  Cost optimization decisions are blind

NAIVE THINKING:
  "Just stop the cluster off-hours."

REALITY:
  Stopping the cluster's nodes (scaling its node groups
  to zero) takes down ALL workloads
  Not just non-prod
  Dev workloads stop
  Production workloads on the same cluster also stop
  Catastrophic for shared clusters

ZOPNIGHT'S SOLUTION:
  Schedule the non-prod namespace
  Not the cluster
  Cluster keeps running
  Every workload in that namespace scales to zero
  Production namespaces unaffected

SCENARIO:
  Node m5.large running 10 pods
  Pod A: scales to zero (pod runtime = 0)
  Node still running (other 9 pods active)
  Cluster bill: same (node-hours unchanged)

COST ACCOUNTING question:
  Did we save anything?

ANSWER depends on accounting model:
  Naive: no (node still running)
  Sophisticated: yes (could scale down node if more pods drop)

WITHOUT INSTANCE-LEVEL vs POD-LEVEL ACCOUNTING:
  FinOps tools either:
  Show "cluster costs $X" without showing what drives → useless
  Show pod-level cost via complex allocation → expensive to compute
  Cannot do both well

ZOPNIGHT'S MODEL:
  Cluster is the COST entity (bill applies to cluster)
  Namespace is the SCHEDULING entity (you stop a namespace's workloads)
  The two are decoupled
  Workload cost = requested CPU + memory, by namespace and node

PROBLEM                                  SOLUTION
─────────────────────────────────────────────────────────────────
Cost lives at wrong level                Requested-cost attribution per workload,
                                          broken down by namespace and node

Schedule-stop the cluster                 Schedule the namespace
loses everything                          (all its workloads together)
                                          Without affecting cluster

Pod-hours vs instance-hours              Schedule at the namespace level;
                                          the cluster stays the billed
                                          entity; requested resources
                                          link the two

EFFECT ON THE CLUSTER:
  Pods scale to zero
  Cluster-autoscaler may scale down nodes (if other workloads
                                            don't need them)
  Or: nodes stay running (if other workloads still need capacity)

EFFECT ON COST:
  The cloud bill falls only if nodes actually scale down
  Fewer running pods free capacity, but a node that
  stays up still bills in full

WHERE TO LOOK:
  Workload cost: requested cost per namespace and workload
  Cluster cost:  node cost, the number the cloud bills
  ZopNight has no "pod-hours saved" or "node-hours saved"
  metric; compare the two views to see the outcome

HOW WORKLOAD COST GETS COMPUTED:

  Cluster cost  = node cost (what the cloud bills)
  Workload cost = requested cost: what its pods REQUEST
                  (CPU + memory), priced against the nodes
  Example: order-api, 4 replicas x (500m CPU, 1 GB)
           its requested cost is its slice of the node bill
  Capacity no workload requests stays as unallocated node cost

BREAKDOWN options:
  By namespace: the workloads in each namespace
  By node: which nodes carry which workloads
  Per workload: kind, namespace, replicas, requested cost

NAMESPACES DRIVE THE BREAKDOWN
  One environment per namespace: environment cost is easy to read
  Mixed namespaces: you still see workloads, but not clean totals

PATTERN                            COST IMPACT      ZOPNIGHT ACTION
───────────────────────────────────────────────────────────────────
Cluster over-provisioned          High               L1 + L4 (right-size)
                                                     M3.3.L1

Non-prod workloads 24/7            High               L3 (scale to zero)
                                                     M3.2 patterns

CronJobs scheduled but never used  Moderate           L4 (suspend cronjobs)

Stateful workloads always-on        High               L5 (statefulset
                                                     scheduling)

Cross-cluster duplication           High               L6 (cross-cluster)
                                                     M3.4 multi-account

Idle workloads accumulating          High               M3.3.L3 (7 shapes)

ESTATE: 1 EKS cluster, ~$8,400/month
WORKLOADS:
  Production deployments:    3 (always-on, ~60% of node capacity)
  Production cronjobs:        12 (cron-driven, ~5% of node capacity)
  Staging deployments:        8 (working hours, ~25% of capacity)
  Dev deployments:             5 (working hours, ~10% of capacity)

NAIVE APPROACH:
  "Stop the cluster nights/weekends"
  Problem: also stops production
  Cannot do

ZOPNIGHT APPROACH:
  Schedule at the namespace level (the K8s scheduling unit):
    Attach the Business Hours schedule (8 AM - 8 PM weekdays)
    to the staging and dev namespaces
    (or add both namespaces to one group and attach it there)
    Action: workloads scale to zero outside hours

  Effect:
    Cluster keeps running
    Production namespaces are never attached, so they keep running
    Staging + dev workloads scale to zero off-hours
    Cluster-autoscaler scales nodes down when pods drop

COST IMPACT:
  Non-prod share: 35% of $8,400 = $2,940/mo
  Off 108 of 168 hours a week (64%): saves ~$1,890/mo
  Before: $8,400/mo (24/7 cluster + workloads)
  After:  ~$6,510/mo (production preserved; non-prod scheduled)
  Savings: ~$1,890/mo = 22.5% reduction
  No impact on production
  Engineers don't notice (their dev environment came back at 8 AM)

□ STEP 1: Inventory
  Number of clusters: _____
  Number of namespaces: _____
  Number of deployments: _____
  Number of CronJobs: _____
  Number of StatefulSets: _____

□ STEP 2: Namespace hygiene check
  Is each namespace a single environment (no prod mixed in)? □ Yes □ No
  Is each namespace owned by one team? □ Yes □ No

□ STEP 3: Identify non-prod
  Namespaces that are dev/staging/test only: _____
  Currently always-on (24/7): _____

□ STEP 4: Estimate savings
  Non-prod fraction of cluster cost: _____%
  Expected scheduling savings: ~64% of that = $_____/mo
  (108 off-hours out of 168 hours a week)

□ STEP 5: Plan rollout
  First cluster to schedule: __________
  Pilot deployments: __________
  Target completion: __________