The 3-step workflow (precondition → action → validate)

Outcome

By the end of this lesson, you will be able to trace any auto-remediation through its 3-4 step workflow, understand why each step exists, and reason about timing and failure modes.


Prerequisites	M2.1 · M2.2
Time	9 minutes

1. Concept

Every automatic fix runs three or four steps, always in the same order.

Each step either succeeds or fails, with nothing in between, and the first failure stops the whole thing. That is deliberate: it means there is never a half-finished change for somebody to unpick afterwards.

STEP 1: PRECONDITION
  Confirm the recommendation is still valid (resource state hasn't changed)
  Run the checks the remediation template declares (e.g. the resource
  still exists, read live from the discovery cache)
  If any check fails: the step fails with "precondition failed: ..."

STEP 2 (OPTIONAL): APPROVAL
  For destructive operations (delete, resize, pause)
  Inserted when the person who clicked is not an admin
  Wait for an admin to approve
  If declined: halt. A job left waiting 48h is cancelled automatically

STEP 3: ACTION
  Execute the cloud action via the provisioner
  For Start/Stop: executor handles directly
  For pause/delete: provisioner orchestrates with cascading

STEP 4: VALIDATE
  Re-check the cloud-side state to confirm the action took effect
  Within 60 seconds typically; longer for K8s cluster operations
  If the observed state does not match: the step fails
  (some auto templates skip this step because the action step
  already confirms the result)

The workflow is deterministic + observable. Every step’s progress shows in the UI.

Why 4 steps: the rationale

PRECONDITION exists to prevent acting on stale findings
  Scenario: rec fired 3 hours ago; someone already terminated the resource
  Without precondition: ZopNight tries to delete an already-gone resource
  With precondition: re-check fails; the workflow stops before any action

APPROVAL exists to route high-blast-radius operations through humans
  Some actions are reversible (start/stop); some aren't (delete)
  The initiator's role (and, for some actions, the approval setting) decide
  Deletes need approval unless an admin starts them

ACTION is the actual cloud API call
  EIP: release-address
  EBS: delete-volume (no snapshot is taken first)
  GCP / Azure disk: delete the unattached disk

VALIDATE exists because cloud APIs are eventually consistent
  "Success" response sometimes means "queued" not "done"
  Validation re-reads the state to confirm
  Catches the rare case where API claims success but action failed

Each step has a purpose. None is redundant.

Step execution model

EACH STEP HAS STATUS:
  pending     (not started yet)
  running     (in progress)
  succeeded   (done; can proceed to next)
  failed      (halted; workflow stops)
  skipped     (not applicable for this rule type)

UI SHOWS PROGRESS IN REAL TIME:
  REMEDIATION: release-eip-0abc (RC-016)
  ─────────────────────────────────────────────────
  [●] STEP 1: Precondition       succeeded  (1.2s)
  [●] STEP 2: Approval           skipped (admin initiator)
  [◐] STEP 3: Action             running   (4.8s elapsed)
  [ ] STEP 4: Validate           pending

Visible progress = trust. Engineers see what’s happening.

How long the workflow takes

TYPICAL TIMING per step:

  Precondition:                       1-3 seconds
  Approval (when required):           0 sec to 48 hr (human-bounded)
  Action: EC2/EBS:                    5-30 seconds
  Action: EKS/GKE/AKS cluster:       30 sec to 5 min
  Validate:                            5-60 seconds

TOTAL DURATIONS:
  No approval, fast resource (EC2 stop):       10-15 sec
  No approval, K8s cluster operation:           3-6 min
  With approval: up to 48 hrs (cancelled after that)

Most actions complete within a minute; K8s operations take longer; approval-gated actions can be days.

Idempotency: workflow can re-run safely

RE-RUNNING A COMPLETED REMEDIATION:
  Precondition re-checks the resource
  If it is already gone: the precondition fails and nothing runs
  No cloud action is attempted twice

RE-RUNNING A FAILED REMEDIATION:
  "Remediate again" starts a new workflow
  The precondition re-checks the resource first
  Nothing resumes the failed job mid-way

GUARANTEE: workflow can be re-run without harm
  Idempotency is the safety property

The idempotency means engineers can re-try with confidence.

What happens at each failure mode

FAILURE MODE                                  WORKFLOW BEHAVIOR
────────────────────────────────────────────────────────────────────
Precondition fails (state changed)            Halt; "precondition failed: ..."
                                              No cloud action runs

Approval rejected                              Job cancelled; reject reason recorded
                                              Audit Logs names the rejecting admin

Approval timeout (48h elapsed)                 Job cancelled automatically
                                              Start it again from the recommendation

Action returns user_action error              Halt; show fix hint + console link
                                              (Customer fixes; can retry)

Action returns transient error                 Halt; failure marked retryable
                                              Wizard offers "Remediate again"

Action returns system error                    Halt; copy diagnostic + contact support
                                              (ZopNight engineering investigates)

Validation fails after action                  Validate step fails; workflow stops
                                              Diagnostic names observed and
                                              expected state

Three error classes (user_action, transient, system) are explored in L4.

Halt-on-first-failure semantics

WHY HALT (vs continue with errors):
  Each step depends on prior steps succeeding
  Partial completion = unclear state
  Halt = clean, observable point to investigate

ALTERNATIVE (continue with errors):
  Risk: cascading wrong actions
  Risk: untraceable partial state
  Risk: harder to debug

ZOPNIGHT CHOOSES HALT:
  Conservative; observable
  Engineer chooses retry strategy
  Audit log clearly shows the failure point

The conservative choice. Better to halt than to wreck.

Audit log per workflow

EVERY WORKFLOW RUN logs:
  Recommendation ID
  Rule ID + version
  Resource UID
  User who triggered (or "auto-remediation cron")
  Timestamp of each step
  Step results (succeeded/failed; duration)
  Cloud API responses
  Approval grants/declines (if applicable)
  Final status + duration

RETRIEVAL:
  Audit Logs page, plus the step timeline on the remediation job
  Filter by user, resource, date, status
  Export for compliance

The audit log is complete; every action is reproducible.

Workflow variations by rule type

SIMPLE RULES (e.g., RC-016 unassociated Elastic IP):
  Precondition → Action → Validate (3 steps)
  Approval skipped for an allowlisted rule when the initiator
  is an admin; a non-admin initiator still hits the gate

GUIDED RULES (e.g., RC-021 orphan EBS snapshot):
  Review & confirm → Precondition → Approval → Action → Validate
  You type to confirm; approval is skipped for an admin initiator

K8S RULES (e.g., cluster termination):
  Precondition → Approval → Action (cascading) → Validate
  Cascading: terminate dependent resources first
  Long validation: cluster takes minutes to fully terminate

SCALE OPERATIONS (e.g., scale-to-1):
  Precondition → Action (scale change) → Validate
  Approval typically skipped
  Reversible action

The variations are determined by the rule’s blast radius.

2. Demo

A successful orphan Elastic IP remediation walked through:

T+0       An admin clicks Remediate on an RC-016 card
          (unassociated Elastic IP)
          Workflow begins

T+0.8s    STEP 1 (Precondition) starts
          Checking the discovery cache for the resource

T+1.2s    STEP 1 (Precondition) SUCCEEDED
          Resource still exists (state is not "deleted")

T+1.2s    STEP 2 (Approval) SKIPPED
          Not inserted: the initiator is an admin

T+1.3s    STEP 3 (Action) starts
          ec2:DescribeAddresses on eipalloc-0abc123
          Confirms the address is not associated

T+2.5s    ec2:ReleaseAddress on eipalloc-0abc123

T+4s      STEP 3 (Action) SUCCEEDED

T+4s      STEP 4 (Validate) starts
          Expected state: deleted

T+6s      STEP 4 (Validate) SUCCEEDED

T+6s      WORKFLOW COMPLETE
          Status: succeeded
          Savings: about $3.65/mo ($0.005/hr unassociated EIP charge)
          Audit log entry created

Six seconds from intent to verified action. Fully audited; reproducible.

3. Hands-on (5 min)

In a sandbox (or low-stakes resource), apply an auto-remediation:

□ STEP 1: Pick a low-stakes recommendation
  Recommendation: __________
  Resource: __________
  Severity: __________

□ STEP 2: Click Apply Remediate
  Note start time

□ STEP 3: Watch the workflow
  Precondition duration: ___ sec
  Approval status: ___ (succeeded / skipped)
  Action duration: ___ sec
  Validate duration: ___ sec
  Total duration: ___ sec

□ STEP 4: Verify outcome
  Notification fired? □ Yes □ No
  Audit log entry? □ Yes □ No
  Cloud-side action visible? □ Yes □ No

□ STEP 5: Re-trigger
  Try applying again
  Expected: the precondition fails (resource already gone)
  Confirm nothing runs twice

A 10-minute hands-on builds intuition for the workflow timing.

Do it through MCP. The same task you just did in the console, asked in one sentence.

BEFORE  A ZopNight account with one cloud connected. One open recommendation on something you are free to change: a non-prod EC2 instance, Compute Engine VM or Azure Virtual Machine.
ASK     "Preview the remediation for this recommendation and tell me exactly what it would change."
CHECK   that you previewed before anything ran. Preview is not politeness, it is the step that shows the blast radius.

Tools behind it: preview_remediation (read, Optimize), get_action (read, Operate), start_remediation (write, tier 3, irreversible). Connection details are in the MCP integration docs.

4. Knowledge check

Q1

The precondition step’s purpose:

A. Authorize the user
B. Notify the team
C. Encrypt the credentials before the cloud API call is ever actually made
D. Confirm the recommendation is still valid (resource state hasn’t changed)

Show answer

Correct: D. Prevents acting on stale findings. If someone already deleted the resource manually, the precondition check fails and the workflow stops before any cloud action runs.

Q2

A remediation’s action step succeeds but validation fails. What happens:

A. Workflow restarts from precondition
B. Reverted automatically
C. The validate step fails and the workflow stops
D. Logged silently, with no notification sent to anyone

Show answer

Correct: C. A state mismatch fails the validate step, and the diagnostic names both the observed and the expected state. Nothing is reverted automatically; a human investigates.

Q3

The total typical time for an auto-remediation without approval:

A. 10-15 seconds for simple resources (EC2 stop)
B. <1 second
C. Always under 60 seconds from start to end
D. Several hours

Show answer

Correct: A. K8s cluster operations take 3-6 minutes. Resource type drives duration; cloud API latency is the bottleneck.

5. Apply

Remediation workflow visible in real time on the recommendation card during apply. Per-action audit log entry covers the full step trace.

For your team: trust comes from observability. Watch a few remediations end-to-end; understand the timing; rely on the audit trail.

Glossary terms touched

Auto-remediation · Idempotency · Validation

Outcome

1. Concept

Why 4 steps: the rationale

Step execution model

How long the workflow takes

Idempotency: workflow can re-run safely

What happens at each failure mode

Halt-on-first-failure semantics

Audit log per workflow

Workflow variations by rule type

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.