Scheduling a Deployment to zero replicas

SCALE-TO-ZERO IS:
  Setting replicas: 0 on the Deployment spec
  All pods drain gracefully (30-60 seconds)
  Deployment object preserved (config, labels, etc.)
  Service objects routing to no endpoints

NOT THE SAME AS:
  Deleting the deployment (would lose config)
  Stopping the cluster (way too coarse)

HOW ZOPNIGHT GETS THERE:
  You schedule the NAMESPACE; namespace Stop scales every
  Deployment in it to zero. You cannot target one Deployment alone.

BEFORE (running):
  Deployment: staging-api
  Replicas:   4 (configured desired count)
  Pods:       4 running
  Service:    routing to 4 endpoints

AFTER (scheduled stop):
  Deployment: staging-api (still exists)
  Replicas:   0 (scaled down)
  Pods:       0
  HPA, PDB (if any): deleted; their specs saved
  Service:    routing to no endpoints
  ServiceMonitor (if any): no targets

ON RESTART:
  Replicas restored to 4 (saved count)
  HPA, PDB recreated from the saved specs
  Pods come up; warm-up ~30-60 sec
  Service routing resumes

SCALE-TO-ZERO WORKFLOW:

1. Pre-check: refuse reserved namespaces (default, kube-*, etc.)
2. Delete any KEDA ScaledObjects in the namespace (saved first)
3. For each Deployment: SAVE the replica count (e.g., 4),
   delete and save any matching HPA and PDB, then set replicas=0
   (Executed via the cluster's K8s API; ZopNight has credentials)
4. Validate: confirm replicas dropped to 0
5. Pods drain over 30-60 seconds (graceful shutdown)
6. Action complete
7. Audit log entry created

SCALE-FROM-ZERO WORKFLOW:

1. Read the state saved at Stop for each workload
2. Action: set staging-api replicas=4, then recreate its HPA/PDB
   (The "4" is the saved-on-stop replica count)
3. Validate: confirm replicas climbed to 4
4. Pods come up (image pull if needed; warm-up if applicable)
5. Service routing resumes
6. Action complete; audit log entry

IF THE CLUSTER HAS CLUSTER-AUTOSCALER:
  Scale-to-zero on a deployment frees node resources
  Cluster-autoscaler may scale down the node pool after a cool-down
  Cost savings come from:
    (a) Zero pod-hours for the deployment
    (b) Reduced node-hours (if cluster downsizes)

IF NO CLUSTER-AUTOSCALER:
  Scale-to-zero frees pod-level resources
  Doesn't change node count
  Cost savings: pod's allocated resources available for other pods
  For non-shared clusters: saving via "more capacity"

WHAT THIS MEANS FOR SAVINGS:
  With autoscaler: much of the deployment's node share comes back
  With no autoscaler: lower real savings until nodes are removed

CHECK BEFORE APPLYING SCHEDULE:

✓ Deployment is non-production (always check)
✓ Service has no critical inbound traffic during off-hours
✓ No dependencies on this deployment from always-on services
✓ HPA (if any): fine; ZopNight deletes it on Stop, recreates on Start
✓ PVCs (if any) are NOT affected (they remain mounted; data preserved)
✓ Restart latency is acceptable (typically 1-3 min warm-up)
✓ No background processes that need to run continuously

WITHOUT HPA:
  Scale-to-zero: kubectl scale --replicas=0
  Scale-back:    kubectl scale --replicas=N (saved)

WITH HPA:
  Scale-to-zero:
    Step 1: Find the HPA targeting the deployment; save its spec
    Step 2: Delete the HPA (so it cannot scale the pods back up)
    Step 3: Scale deployment to 0
    State: no HPA; replicas at 0

  Scale-back:
    Step 1: Scale deployment to saved replicas
    Step 2: Recreate the HPA from the saved spec
    State: HPA active; replicas restored

  HPA's min/max settings come back exactly as saved
  Delete-and-recreate is the integration point (HPAs have no pause)

SCENARIO:
  Service-A: staging-database-service
  Service-B: staging-api (calls service-A)

  Schedule service-A off at 8 PM
  service-B at 8 PM tries to query A → endpoints empty → connection failure
  service-B reports errors

PROBLEM:
  Dependent service breaks when dependency is off

TWO SOLUTIONS:

  SOLUTION 1: SCHEDULE TOGETHER:
    Both staging-database-service and staging-api in the same
    namespace, so one namespace schedule covers both
    Both stop at 8 PM; both restart at 8 AM
    No error window
    Cleanest approach

  SOLUTION 2: SEQUENCED EXECUTION (different namespaces):
    Put both namespaces in one group with an ordered sequence
    Stop staging-api's namespace first
    Then staging-database-service's namespace
    Avoids the error window
    More complex coordination
    See T1.M1.4.L4 for sequenced execution

WHY ZOPNIGHT SAVES THE REPLICA COUNT:

  Without save:
    Scale-to-zero
    On restart: how many replicas?
    Default? Hardcoded? Configurable?
    Loses the deployment's actual desired state

  With save-on-stop:
    Stop: save replicas=4 (or whatever current value is)
    Start: restore replicas=4
    Preserves the desired state

  Edge cases handled:
    HPA increased to 8 yesterday → save=8 on next stop
    HPA min changed to 2 → save reflects the new minimum
    Manual scaling to 6 → save=6 if stop captures that state

EXAMPLE: staging-api deployment
  Replicas: 4 during working hours
  Schedule: 8 AM - 8 PM Mon-Fri
  Off-hours: 12 hours weekday + all weekend

PER WEEK:
  Active hours: 12 × 5 = 60 hours
  Off hours: 168 - 60 = 108 hours

PER POD-HOUR cost: $0.05 (cluster pod-share)

SAVINGS:
  Off-hours pods saved: 4 × 108 = 432 pod-hours
  At $0.05/pod-hour: $21.60/week
  Monthly: ~$94
  Annual: ~$1,128 ($94 × 12)

ACROSS THE CLUSTER:
  10 similar deployments: ~$11,280/year
  20 similar deployments: ~$22,560/year

The compounding is where the savings come from

SETUP:
T+0      Navigate to the staging namespace in eks-cluster-prod
         (its only Deployment is staging-api)
T+5s     Note pre-conditions:
            Replicas: 4
            HPA: none
            PVCs: none (stateless)
            Dependencies: none critical

T+10s    Attach the namespace to "business-hours-staging" schedule
T+10s    Schedule cron:
            Stop: "0 20 * * 1-5"
            Start: "0 8 * * 1-5"

EVENING (8 PM):
T+0      Cron fires
T+1s     ZopNight: save replicas=4
T+1s     ZopNight: patch deployment/staging-api replicas=0
T+5s     K8s acknowledges the scale
T+30s    Pods drain (graceful shutdown)
T+30s    Deployment has 0 replicas; pods all gone
T+30s    ZopNight notification fires
T+30s    Audit log entry created

NEXT MORNING (8 AM):
T+0      Cron fires
T+1s     ZopNight: patch deployment/staging-api replicas=4
T+5s     K8s acknowledges
T+30s    4 pods come up
T+60s    Service routing resumes
T+60s    Application available for the day

REALIZED SAVINGS:
  Pod-hours saved nightly: 12 hours × 4 pods = 48 pod-hours
  At average pod cost of ~$0.05/pod-hour: $2.40/night
  Weeknights: $2.40 × 5 nights = $12
  Plus full weekend: 48 h × 4 pods = 192 pod-hours = $9.60
  Weekly: $21.60 (same as the Concept example)
  Monthly: ~$94

  Across 12 similar deployments in cluster:
    Monthly: ~$1,128
    Annual: ~$13,536

□ STEP 1: Pick a non-prod namespace and one deployment in it
  Namespace: __________
  Workload: __________
  Replicas: _____
  HPA: □ Yes □ No
  Dependencies: __________

□ STEP 2: Verify pre-conditions
  Non-prod: □ Yes
  Off-hours traffic: □ None □ Some
  Dependent services: □ Yes (handle) □ No

□ STEP 3: Attach the namespace to a schedule
  Schedule: __________
  Stop time: __________
  Start time: __________

□ STEP 4: Wait for stop cron (or trigger)
  Observed at time: __________
  Replicas dropped to 0: □ Yes □ No
  Pods drained gracefully: □ Yes

□ STEP 5: Verify restart
  Replicas restored: __________
  Service available: __________

BEFORE  A ZopNight account with one cloud connected. A Kubernetes cluster connected: EKS, GKE or AKS, with a non-prod namespace you can safely scale down.
ASK     "List the workloads in this namespace and scale the non-prod deployments to zero tonight."
CHECK   the workload list first. Scaling something to zero without reading what it is remains the fastest way to page yourself.