A team writes the cron job that shuts non-prod down at 8 PM. The cron runs three commands in parallel: scale the EKS Deployments to zero, pause the Aurora cluster, stop the ElastiCache Redis nodes. It works for two months. On a Tuesday in week nine, the morning wake-up surfaces a Postgres replica with a corrupted WAL because the writer was paused mid-transaction while the application was still committing. The team spends a day restoring from snapshot. Two weeks later the same thing happens to a different service.
This is the dependency-ordering trap. Cron schedulers fire jobs in parallel by default. The cloud doesn’t tell you in what order to shut things down. The application doesn’t refuse a write to a database it is in the middle of pausing. Everything worked fine for 60 days because nothing was actively writing at 8 PM most nights, until the night someone shipped a long-running batch job at 7:55 PM.
This post is the explicit dependency model for non-prod shutdown: which components depend on which, what the right order is, what breaks at each wrong order, and why the solution is not “longer sleep timers” but explicit dependency-aware sequencing. The pattern composes with automated cloud scheduling for non-prod environments, why cron jobs fail, and the SRE rule about 24/7 infra.
The dependency direction
In production, dependencies flow from the user toward the database. A user request hits the load balancer, which calls the application, which calls the database. The application depends on the database; the database does not depend on the application.
For graceful shutdown, the order reverses: stop the thing at the top of the chain first, drain it, then stop what it depends on. A typical web application’s dependency chain runs User → Load balancer (ALB/NLB) → Application (Deployment) → Cache (ElastiCache Redis) → Database (RDS Aurora).
Shutdown order is top-down. The load balancer drains first (stop accepting new connections, finish in-flight ones). The application drains next (let pre-stop hooks fire, let goroutines finish, let the connection pool flush). Then the cache (it is not authoritative, so a hot drop is annoying but not catastrophic). Then the database (final, with no in-flight transactions to corrupt).
Wake-up is bottom-up, the reverse. Database first, cache second, application third, load balancer last. The application cannot start without the database. The cache warms up after the application connects.
| Layer | Shutdown order | Wake-up order | What breaks if order is wrong |
|---|---|---|---|
| Load balancer | 1 (drain) | 4 (last) | New connections during drain |
| Application | 2 | 3 | Active transactions on shutdown DB |
| Cache | 3 | 2 | Cold cache hit storm on wake |
| Database | 4 | 1 | WAL corruption, lost commits |
The four ordering bugs that bite
Bug 1: Database paused while app is committing. The classic. The application is mid-commit on a transaction. The schedule pauses the Aurora cluster (or sends the RDS instance to “stopped”). The pending commit is dropped. On wake, WAL replay either succeeds (best case, latency spike on the replay) or fails (worst case, corruption flag set, requires manual intervention).
The fix: drain the application first. Wait until pod count is zero AND active connections to the database are zero. THEN pause the database. The 4-stage shutdown sequence:
| Stage | Action | Wait condition |
|---|---|---|
| 1 | Scale the application Deployment to zero replicas (kubectl scale deployment/api --replicas=0) | Command returns |
| 2 | Wait for pods to terminate (kubectl wait --for=delete pod -l app=api --timeout=120s) | All matching pods deleted |
| 3 | Confirm zero active DB connections from the application (query pg_stat_activity for active sessions with application_name matching the service) | Count is zero, or alert and stop |
| 4 | Pause the database cluster (aws rds stop-db-cluster --db-cluster-identifier dev-aurora) | Cluster status stopped |
Bug 2: Cache stopped while app is reading. Less catastrophic than Bug 1, but ugly. The app is mid-request, hits Redis for a session lookup, gets a connection error, returns a 500 to the user. If the user was the developer running a test, the test fails for the wrong reason. If the user is a webhook from an external partner, you might have a real outage on retry.
The fix is the same shape: drain the application before stopping the cache. The cache should be the third thing to stop, not the first.
Bug 3: Load balancer killed while connections are in-flight. ALB / NLB graceful drain is built-in (deregister the target with a deregistration delay), but only if you use it. Hard-deleting the load balancer takes the connections with it. Most teams do not delete the load balancer (it has a stable DNS name they want to keep), but if your cleanup script does, this is the trap.
The fix: don’t delete load balancers as part of a schedule. Leave them up; load-balancer cost is negligible compared to compute and database. Drain the targets behind them.
Bug 4: Wake-up out of order. Application starts before the database is healthy. The first 30 seconds of pod startup is CrashLoopBackOff because the DB connection fails. The pod retries, eventually succeeds, but the kubelet has now backed off the restart interval and the pod is in a 5-minute crash loop while the database has been up for 2 minutes.
The fix: wake the database first, wait for it to be healthy (RDS instance status available), THEN scale up the application. Most native schedulers don’t have this sequence. They fire all wake actions in parallel.
Why cron schedulers don’t express dependencies
A cron expression is 0 20 * * *. It says “fire at 8 PM.” It says nothing about what depends on what. If you write three crons (one for the app, one for the cache, one for the DB), they all fire at 8 PM. Whichever Lambda finishes first is the one that “won” the race; the order is non-deterministic.
You can fix this with sleep timers (“wait 60 seconds before pausing the DB”) but the right interval depends on what is in flight. Sometimes 60 seconds is enough; sometimes a long-running batch job needs 10 minutes. A fixed sleep is wrong on both directions.
The honest fix is to express the dependency. AWS Step Functions can do this: a state machine where state B does not start until state A signals “drained.” So can a Kubernetes operator with init / pre-stop hooks. So can a workflow tool like Argo Workflows. So can a purpose-built scheduler (ZopNight ships dependency-aware sequencing as a first-class primitive: define the graph, the system orders the actions).
| Tool | Express dependencies? | Wake-up order? | Operational burden |
|---|---|---|---|
| Cron + Lambda | No | No | DIY, brittle |
| AWS Instance Scheduler | No (parallel only) | No | Low |
| Step Functions | Yes (state machine) | Yes | DIY but stable |
| Argo Workflows | Yes (DAG) | Yes | Cluster-resident, K8s-only |
| Purpose-built scheduler (ZopNight) | Yes (graph + UI) | Yes | Managed |
The 5-step dependency-aware sequence
For a typical dev environment with EKS application + Aurora database + Redis cache + ALB:
Shutdown sequence:
-
Drain ALB targets. Deregister the EKS targets from the ALB target group. Wait for
deregistration_delay(default 300s; lower it to 30-60s for non-prod). Connections in flight finish, new ones don’t arrive. -
Scale application Deployments to 0. Pre-stop hooks fire. Goroutines / threads finish. Connection pools flush. Wait for
kubectl wait --for=delete podwith a 120s timeout. -
Verify zero DB connections. Query
pg_stat_activity(Postgres) or equivalent. Should be zero application-named connections. If not zero after 30s, alert: something is leaked, do not proceed. -
Stop / pause cache. Send
SHUTDOWNto Redis (or stop the ElastiCache nodes). Cache is non-authoritative; this is fast and safe. -
Pause / stop database.
aws rds stop-db-clusterfor Aurora;aws rds stop-db-instancefor RDS standalone. Aurora Serverless v2 paused state has the lowest cost if your dev volume warrants it.
Wake-up sequence (reverse):
-
Resume database.
start-db-cluster. Wait for statusavailable. Aurora typically 60-180s; RDS standalone 5-10 min on cold start. -
Resume cache. Start ElastiCache nodes. Wait for status
available. -
Scale application Deployments back to N. Pods come up, connect to DB, populate cache from DB. First requests will be slow (cold cache); steady state in 30-60s.
-
Re-register ALB targets. Health checks pass. Traffic resumes.
-
Verify health end-to-end. Synthetic probe: hit the public URL, expect a 200, log the latency. Anything above 2× the steady-state baseline is a smoke signal.
| Step | Action | Wait condition | Timeout |
|---|---|---|---|
| 1 (down) | Drain ALB targets | Deregistration complete | 60s |
| 2 (down) | Scale app to 0 | Pods deleted | 120s |
| 3 (down) | Check DB connections | Zero app connections | 30s |
| 4 (down) | Stop cache | Redis shutdown OK | 30s |
| 5 (down) | Stop DB | Cluster stopped | 600s |
| 1 (up) | Start DB | Status available | 600s |
| 2 (up) | Start cache | Nodes available | 120s |
| 3 (up) | Scale app to N | Pods ready | 180s |
| 4 (up) | Re-register ALB | Health checks pass | 60s |
| 5 (up) | Synthetic probe | 200 from URL | 30s |
The total shutdown takes 5-15 minutes. The total wake takes 6-12 minutes. The difference between this and parallel cron is that nothing breaks at the boundary, and the engineer who ships the long-running batch at 7:55 PM does not corrupt the database at 8 PM.
The graph for a non-trivial environment
Real environments have more than one app. A typical dev stack might have a frontend app, a backend API, a worker service for batch jobs, a search service backed by Elasticsearch, plus the shared database, cache, and storage.
The dependency edges in such an environment:
| Source | Depends on |
|---|---|
| ALB (web route) | frontend |
| frontend | backend |
| backend | worker, Redis, Aurora |
| worker | Redis, Aurora |
| ALB (search route) | search-service |
| search-service | Elasticsearch |
Shutdown order (depth-first, from the leaves outward):
- ALB drain (both routes)
- frontend drain (no downstream waiting on it)
- worker drain (depends on backend; backend cannot drain until worker stops calling)
- backend drain (now safe)
- search-service drain
- Redis stop
- Elasticsearch stop
- Aurora stop
The graph format makes the order obvious. The cron format hides it. This is why purpose-built schedulers expose the dependency as a first-class graph: you draw the edges; the system computes the order.
This works when the graph is kept up to date as services are added. It breaks when a new service is deployed without being added to the graph; the schedule shuts down the things it knows about, the new service keeps writing to a paused database, and the morning brings a fresh corruption.
The mitigation is to make the graph required at deploy time. A new Deployment that doesn’t have a dependency entry doesn’t deploy. This is admission-time policy enforcement; the Cloud Custodian / OPA / MCP decision matrix covers the surface choice.
The audit trail
Every shutdown and wake event should produce a structured log. The fields the audit row needs:
| Field | Type | Example |
|---|---|---|
ts | ISO timestamp | 2026-05-08T20:00:01.000Z |
schedule | string | dev-eks-weeknight |
phase | shutdown or wake | shutdown |
step | string | drain-alb-targets |
duration_ms | number | 28430 |
wait_condition_met | boolean | true |
result | ok or fail | ok |
The query “how long did last night’s shutdown take” is a one-liner. The query “which step is the slowest” reveals where the dependency timeout is wrong. The query “did any step fail in the last 30 days” is the regression dashboard.
When something corrupts, this is the audit trail that says “the database was paused at 20:05:42, the application’s last commit attempt was at 20:05:39, the gap was 3 seconds, too short, the dependency check did not actually wait.” That is the post-mortem signal.
A 21-day rollout
Days 1-7: Map the graph for one environment. One dev environment. Draw the dependency graph by hand. Confirm with the service owners. The first draw is always wrong; expect 3-5 corrections in the first week.
Days 7-14: Build the dependency-aware shutdown for that environment. Pick the orchestrator: Step Functions, Argo, or a managed scheduler. Implement the 5-step shutdown and 5-step wake. Run it manually (not on a schedule) during business hours; verify the audit log.
Days 14-21: Move it to a schedule. Pick the off-hours window (8 PM to 7 AM is typical). Run for two weeks with monitoring on. Watch for the failure modes: failed wait conditions, unexpected DB connections at shutdown time, cold-cache hit storms. Tune the timeouts and the order based on what you learn.
By day 21 the team has one environment running on dependency-aware shutdown with no corruption events. The pattern then extends to other environments by reusing the orchestrator and re-drawing the graph for each.
The closing math
A corruption event from a wrong-order shutdown costs 4-8 engineer-hours of triage and restoration. Across 30 dev environments shutting down nightly, the failure rate at parallel-cron is roughly 1-2 incidents per quarter. The dependency-aware version brings this to near-zero.
The savings from the schedule itself ($60K-$200K/year for a typical mid-market platform) only land if the team trusts the schedule. One corruption event is enough to roll back the whole rollout. The dependency-aware sequence is the part that keeps the trust intact.
The pattern is not “longer sleep timers.” It is “explicit dependency expression.” The graph format is what survives reorganization, what onboards new services correctly, and what answers the post-mortem question. Pick one environment. Draw the graph. Run the orchestrator manually for a week before scheduling it. The morning of the first weekend the schedule fires correctly is the morning the savings start to compound.
