Outcome
By the end of this lesson, you will be able to configure sequenced execution within a group and choose between auto and custom ordering based on workload dependencies.
| Tier | Operator |
| JTBD | ”Stop my app from trying to start before its database is up.” |
| Personas | Platform Engineer |
| Prerequisites | L1-L3 |
| Time | 10 minutes |
| Bloom verb | Configure (Apply) and Choose (Evaluate) |
1. Concept
When a schedule fires on a group, something has to decide what order things come up in.
Starting everything at once is fine when nothing depends on anything. It is not fine the moment one thing needs another: an application that starts before its database is ready does not start, it fails.
So a group can set an order. Storage first, then the machines, then the applications on top. Stopping runs the other way.
The default order: auto-sequencing
ZopNight ships with sensible defaults based on resource type:
TIER TYPICAL RESOURCES START ORDER STOP ORDER─────────────────────────────────────────────────────────────────────1 Databases (RDS, CloudSQL, FIRST LAST Azure SQL, Cosmos), Persistent storage (EBS, Persistent Disks, Managed Disks), Caches (ElastiCache, Memorystore)
2 Compute (EC2, GCE, Azure VMs, SECOND SECOND ECS, EKS nodes, GKE nodes, AKS nodes)
3 Application / orchestration THIRD FIRST (Lambda concurrency, container services, load balancers, autoscalers)Starting goes bottom up: storage, then machines, then applications. Stopping goes top down, so traffic drains out of the application before the machines underneath it disappear.
That order assumes the usual arrangement, where the application needs the machines and the machines need the storage. Most systems are built that way, which is why it is the default.
That default is right about 80% of the time, and most groups never need anything else.
Custom sequencing
For workloads with non-standard dependencies:
EXAMPLE: A group with 12 resources. - 2 RDS instances (db-orders, db-users) - 6 EC2 instances (4 app servers, 2 batch workers) - 2 ECS services (api-service, worker-service) - 1 Lambda function - 1 ALB
DEFAULT auto-sequencing would group them by tier (storage / compute / app).But the team knows: - db-orders MUST start before db-users (replication dependency) - app servers MUST start before batch workers (registry dependency) - api-service waits for app servers; worker-service is independent
CUSTOM SEQUENCE (start order): Position 1: db-orders (RDS, tier 1) Position 2: db-users (RDS, tier 1, depends on db-orders) Position 3: app-server-1 (EC2, tier 2) Position 4: app-server-2 (EC2, tier 2) Position 5: app-server-3 (EC2, tier 2) Position 6: app-server-4 (EC2, tier 2) Position 7: batch-worker-1 (EC2, depends on app servers) Position 8: batch-worker-2 (EC2, depends on app servers) Position 9: api-service (ECS, depends on app servers) Position 10: worker-service (ECS, independent: could move earlier) Position 11: lambda-X (Lambda) Position 12: alb-main (ALB, last)The custom sequence runs strictly in order. Each step waits for the previous to complete (with timeout) before proceeding.
Per-step delay
For workloads that need warm-up time even after a successful start:
SEQUENCE WITH DELAYS Position 1: db-orders (no delay) Position 2: db-users Wait 60s after db-orders before starting db-users Position 3: app-server-1 Wait 120s after db-users (DB warm-up)...Per-step delay specifies how long ZopNight waits after the previous resource is confirmed-started before starting the next. Useful for warm-cache scenarios, DB replication setup, or service-mesh propagation.
Stop sequence
For stop, the default sequence reverses (app → compute → storage). Custom sequence is also reversible: ZopNight applies the reverse of the start order on stop, unless the user explicitly defines a separate stop sequence.
When auto vs custom
WORKLOAD PROFILE SEQUENCING─────────────────────────────────────────────────────────Independent resources Auto (tier-based)Layered architecture (web → app → db) Auto worksComplex dependencies (replica chains, Custom service mesh ordering)Warm-up requirements (>30s after start) Custom with delaysPure compute groups (no DBs, no app tier) AutoMost teams start with auto. Move to custom when a specific failure (“the app tried to start before the cache was ready”) motivates it.
Failure handling
If a step in the sequence fails (e.g., a DB times out during start):
- The sequence pauses- The action status panel shows the failure- Subsequent steps are NOT attempted- The team is notified (per notification routing)- The team investigates the root causeFailures do not cascade silently. The intent is to halt and let humans diagnose rather than mass-start everything despite a missing dependency.
For groups with retry tolerance, the schedule can be configured to retry the failed step up to N times with exponential backoff (default: 3 retries). After exhaustion, the failure is logged and the sequence continues from the next step.
Tag-based custom ordering
For teams that prefer declarative sequencing (vs UI-based drag-drop), resources can carry a sequence tag:
Tag: zopnight:start-order = "1" for db-ordersTag: zopnight:start-order = "2" for db-usersTag: zopnight:start-order = "3" for app-server-1...ZopNight reads the tag and orders the sequence accordingly. Tag-driven sequencing scales better than UI clicks for large groups.
2. Demo
A team’s workflow for sequencing a multi-tier app:
WORKLOAD: prod-billing-stack Members: 1× RDS, 1× ElastiCache Redis, 3× EC2 app servers, 1× ALB
INITIAL APPROACH: Auto-sequencing - Tier 1 (storage): RDS, ElastiCache start first (parallel) - Tier 2 (compute): 3 EC2 servers start in parallel - Tier 3 (app): ALB activated last
TESTED: Schedule fires for the first time. Works on Day 1.
DAY 14: An ElastiCache restart takes 90 seconds (cold cache warm-up).The EC2 app servers start at +30 seconds (default tier delay) but failbecause the cache isn't warm yet.
DIAGNOSIS: The default sequencing is right (cache before app), but thedefault delay between tiers (30s) is too short for this workload'scache warm-up.
FIX: Switch to custom sequencing with per-step delay: Position 1: RDS (no delay) Position 2: ElastiCache (parallel with RDS) Position 3: app-server-1 Wait 120s after ElastiCache Position 4: app-server-2 Wait 120s after ElastiCache (concurrent with #3) Position 5: app-server-3 Wait 120s after ElastiCache (concurrent) Position 6: ALB Wait 30s after app servers
Re-test. Day 15 schedule fires cleanly. No more cold-start failures.The hours did not change. What changed was the order things start and stop in.
3. Hands-on (6 min)
For a multi-tier group you can experiment with (use a sandbox):
1. Open the group → Settings → Execution Order tab.2. Note the current setting (Auto or Custom).3. If Auto: review what tiers ZopNight inferred from the member resource types. Confirm the order makes sense for your workload.4. If your workload has non-default dependencies: - Switch to Custom - Drag-drop the members into the right order - Add per-step delays where needed - Save5. Trigger the schedule (or wait for the next firing).6. Watch the action status panel to verify sequencing fires in order.
If a step fails, read the error. Most failures are: - Permission missing (fix via M1.1) - Cloud-side timeout (extend the per-step delay) - Dependency mismatch (reorder)4. Knowledge check
Q1
The default sequencing order (auto) starts:
A. Storage first (DBs, persistent disks, caches), compute second (VMs, container nodes), app last (load balancers, autoscalers, Lambda concurrency)
B. Compute first, then the storage, and then the app tier last of all, which is the exact reverse of the documented default order
C. In random order
D. All at once
Show answer
Correct: A. Stop reverses the order. Storage → compute → app for start. Reverse for stop. Matches the typical layered-architecture dependency assumption.
Q2
A team’s app servers fail to start because their ElastiCache takes 90 seconds to warm up. The right fix is:
A. Auto-sequencing is broken
B. Switch to custom sequencing with a per-step delay of >90 seconds between cache and app servers
C. Disable the cache
D. Restart all of the app servers by hand after each and every scheduled start execution
Show answer
Correct: B. Auto handled the tier order correctly; the warm-up requirement was the gap. Auto-sequencing handled tier order. The fix is per-step delay tuning, which is a custom-sequence feature.
Q3
A step in a sequence fails. The expected behavior is:
A. The sequence simply continues straight on with the next step, ignoring the failed step entirely on each occasion
B. The entire group is stopped
C. The schedule deletes itself
D. The sequence pauses, the failure is logged with diagnostic detail, subsequent steps are NOT attempted, and the team is notified
Show answer
Correct: D. Halt-and-diagnose is the default. Halt-and-diagnose prevents cascading failures where the system would attempt to start dependent resources without their dependencies. The team investigates rather than mass-failures auto-cascading.
5. Apply
Sequencing is configured per-group:
- Group detail → Execution Order tab: auto or custom
- Per-step delays: configurable per position in custom mode
- Tag-driven ordering: via
zopnight:start-ordertag for declarative control - Action status panel: shows sequence progress during firing
For groups with retry configuration, see the per-rule retry settings under organization settings.
Module quiz
You have now completed all four lessons of M1.4. The module quiz (10 questions, 80% pass) is on the Operator certification page.
Related lessons
- M1.5: Overrides (next module)
- T5.M5.2: Schedule design patterns
Glossary terms touched
Sequenced execution · Auto-sequencing · Custom sequence · Per-step delay · Tier (resource)