Sequenced execution: storage / compute / app

Outcome

By the end of this lesson, you will be able to configure sequenced execution within a group by setting each member’s tier and start and stop delays to match workload dependencies.

1. Concept

When a schedule fires on a group, something has to decide what order things come up in.

Starting everything at once is fine when nothing depends on anything. It is not fine the moment one thing needs another: an application that starts before its database is ready does not start, it fails.

So a group can set an order. Storage first, then the machines, then the applications on top. Stopping runs the other way.

The default order: tiers

ZopNight assigns each member a tier from its resource type when you add it to the group:

TIER        TYPICAL RESOURCES                       START ORDER   STOP ORDER
──────────────────────────────────────────────────────────────────────────
Storage     Disks and buckets (EBS, Persistent      FIRST         LAST
            Disks, Managed Disks, S3, GCS)

Database    RDS, Cloud SQL, Azure SQL and other     SECOND        THIRD
            types with "sql" or "database" in
            the name

Compute     VMs (EC2, GCE, Azure VMs) and anything  THIRD         SECOND
            that matches no other tier

App         Container and serverless platforms      LAST          FIRST
            (EKS, GKE, AKS, ECS, Cloud Run,
            Azure Function Apps)

Starting goes bottom up: storage, then databases, then machines, then applications. Stopping goes top down, so traffic drains out of the application before the machines underneath it disappear.

The tier picker on the group page offers Storage, Compute and Apps, so you can move a member between those by hand. The Database tier is only assigned automatically.

Each tier that has members fires 30 seconds after the one before it. Within a tier, members fire in their execution order (the order they were added).

That order assumes the usual arrangement, where the application needs the machines and the machines need the storage. Most systems are built that way, which is why it is the default.

That default is usually right, and most groups never need anything else.

Changing the order: tier and delay

There is no separate “custom” mode and no drag-and-drop list. Each member has two settings you can change on the group page, in the member table:

• Tier: Storage, Compute or Apps (or no tier). Moving a member to another tier moves it to that tier’s slot in the order.
• Delay (seconds): a separate start delay and stop delay, added to the member’s tier offset.

A member’s action fires at its tier’s offset plus its own delay:

EXAMPLE: start of a group with a database, 3 app servers and a container service

TIER         MEMBER            TIER OFFSET   START DELAY   FIRES AT
────────────────────────────────────────────────────────────────────
Database     db-orders         +0s           0s            +0s
Compute      app-server-1      +30s          0s            +30s
Compute      app-server-2      +30s          0s            +30s
Compute      app-server-3      +30s          0s            +30s
App          api-service       +60s          0s            +60s

Only tiers that have members count, so the first occupied tier fires at +0s, the next at +30s, and so on. Set a 120-second start delay on the three app servers and they fire at +150s instead.

A delay on one member does not push back later tiers. In the example above, api-service still fires at +60s, now before the app servers. If a later tier depends on a delayed member, give it a delay too.

Nothing waits for the previous resource to finish starting. Each resource fires at its tier’s offset plus its own delay, whether or not the one before it succeeded.

Start delay and stop delay

The delay is a fixed time gap in seconds, not a wait for the previous resource to be confirmed started. Each member has a separate start delay and stop delay: a database that needs time to come up wants a start delay on the things that use it, while an application that needs time to drain connections wants a stop delay on the things beneath it. Useful for warm-up after start, replication catch-up, or service-mesh propagation.

Stop sequence

For stop, the tier order reverses (app → compute → database → storage). There is one order per member, not a separate stop sequence; what you can set separately is each member’s stop delay.

When to change the defaults

WORKLOAD PROFILE                              WHAT TO SET
─────────────────────────────────────────────────────────
Independent resources                         Nothing (tiers as assigned)
Layered architecture (web → app → db)         Nothing, usually
A type landed in the wrong tier               Change that member's tier
Warm-up requirements (>30s after start)       Start delay on the dependants
Slow drain before the layer below stops       Stop delay on the layer below

Most teams start with the tiers ZopNight assigns. Add a delay when a specific failure (“the app tried to start before the database was ready”) motivates it.

Failure handling

Sequencing is time spacing, not a dependency check. If one resource fails (e.g., a DB times out during start):

- The other members still fire at their own offsets
- Later tiers are NOT held back
- The failed action is retried on its own: after 2 min, 5 min, then 10 min
- After 3 failed retries it is marked retry_exhausted
- The team is notified (per notification subscriptions)

So a missing dependency does not stop the rest of the group from starting. If an app cannot run without its database, give the app enough delay to cover the database’s start time, and watch for failure notifications.

2. Demo

A team’s workflow for sequencing a multi-tier app:

WORKLOAD: billing-stack (non-prod)
  Members: 1× RDS, 3× EC2 app servers, 1× ECS service

DEFAULT TIERS (assigned when the members were added):
  - Database: RDS fires at +0s
  - Compute:  3 EC2 servers fire at +30s
  - App:      ECS service fires at +60s

TESTED: Schedule fires for the first time. The app servers log
connection errors at start, then recover only after a manual restart.

DIAGNOSIS: The order is right (database before app servers), but the
database takes about 2 minutes to accept connections after it starts.
The 30-second tier gap is too short for this workload.

FIX: On the group page, set a 120-second start delay on each of the
three app servers (they now fire at +150s) and a 150-second start
delay on the ECS service (it now fires at +210s, after the servers).

Re-test. The next morning's start fires cleanly. No more cold-start failures.

The hours did not change. What changed was the order things start and stop in.

3. Hands-on (6 min)

For a multi-tier group you can experiment with (use a sandbox):

1. Open the group from the Resource Groups page.
2. In the member table, read the Tier column: these are the tiers
   ZopNight assigned from each resource type. Use the tier tabs
   to see the members of one tier at a time.
3. Confirm the order makes sense for your workload. Move any member
   that landed in the wrong tier using its tier picker.
4. If something needs warm-up time, set a start delay (seconds) on
   the members that depend on it. Set stop delays where a layer
   needs time to drain.
5. Trigger the schedule (or wait for the next firing).
6. Open History, filter to the group's members, and check the times
   on the started rows match tier offset plus delay.

If a step fails, read the error. Most failures are:
  - Permission missing (fix via M1.1)
  - Cloud-side timeout (increase the dependants' delay)
  - Dependency mismatch (change the member's tier or delay)

4. Knowledge check

Q1

The default tier order starts:

A. Storage first, then databases, then compute (VMs), with the app tier (EKS, ECS, Cloud Run) last
B. Compute first, then the storage, and then the app tier last of all, which is the exact reverse of the documented default order
C. In random order
D. All at once

Q2

A team’s app servers fail at start because their database takes 90 seconds to accept connections. The right fix is:

A. The tier order is broken and needs rebuilding
B. Set a start delay of over 90 seconds on the app servers
C. Disable the cache
D. Restart all of the app servers by hand after each and every scheduled start execution

Q3

A step in a sequence fails. The expected behavior is:

A. The other members still fire on time, and the failed action is retried on its own
B. The entire group is stopped, and the schedule waits for a person to restart it
C. The schedule deletes itself, and the group must be attached to a new one
D. The sequence pauses, and later tiers are not attempted until the step succeeds

5. Apply

Sequencing is configured per-group:

• Group detail page → member table: each member’s tier and its start and stop delay in seconds
• History: the actual time each member started or stopped

Retries are not configurable: a failed action retries after 2, 5 and 10 minutes, then is marked retry_exhausted.

Module quiz

You have now completed all four lessons of M1.4. Take the M1.4 module quiz: 10 questions, 80% to pass.

Related lessons

• M1.5: Overrides (next module)
• T3.M3.2: Schedule design patterns

Glossary terms touched

Sequenced execution