Apply, pause, resume, remove: the lifecycle

Outcome

By the end of this lesson, you will be able to walk an autoscaler policy through its full lifecycle, understand what each operation does at the cloud layer, and execute safe rollback when needed.


Prerequisites	M2.4.L1 - L4
Time	9 minutes

1. Concept

A policy has four things you can do to it. Apply and Remove change your cloud account; Pause and Resume only change the policy’s status inside ZopNight.

APPLY     Push the policy to the cloud (create cloud-side autoscaler)
PAUSE     Mark the policy paused in ZopNight (no cloud call)
RESUME    Mark a paused policy active again (no cloud call)
REMOVE    Roll back: delete cloud-side autoscaler; restore previous config

The lifecycle is the full management surface. All four operations are idempotent + reversible.

APPLY: pushing to the cloud

WORKFLOW STEPS:

1. ZopNight queues an apply intent; the executor picks it up
   asynchronously and validates the policy against the cloud
   - Target exists?
   - Permissions present?
   - Min/max sensible (max > min)?

2. Pre-checks pass; proceed

3. Cloud API call (provider-specific):
   AWS ASG: PutScalingPolicy + UpdateAutoScalingGroup
   ECS:     RegisterScalableTarget + PutScalingPolicy
            (prefixed "zopnight-ecs-")
   Azure VMSS: PutAutoScaleSettings
   GCP MIG:    insertAutoscaler

4. Previous config saved on policy event row
   For later rollback (Remove operation)

5. Cloud policy now active
6. From here the cloud's native autoscaler scales per the policy;
   ZopNight does not run a scaling loop of its own

The previous-config save is critical. Without it, Remove couldn’t restore the original setup byte-for-byte.

PAUSE: disable temporarily

PAUSE STEPS:

1. ZopNight sets the policy status to "paused" (database only)
2. No cloud call is made: the scaling rules already applied stay
   active and keep scaling until you Remove the policy
3. Policy event row records the pause action; a lifecycle
   notification is sent

WHEN TO PAUSE:
  Signal that a policy is under review
  Investigating an anomaly without touching the cloud
  NOT to halt cloud scaling: use Remove for that

Pause is a status flag, reversible without re-doing any configuration. It does not stop scaling.

RESUME: re-enable

RESUME STEPS:

1. ZopNight sets the policy status back to "active" (database only)
2. No cloud call is made; the cloud rules were never touched
3. Resume event recorded in the policy event log

GUARANTEE:
  Resume is non-destructive
  Cloud-side state was never changed by the pause

Resume is the simple counterpart to Pause.

REMOVE: complete rollback

REMOVE STEPS:

1. ZopNight queries the previous_config from the policy event row
   (Saved during Apply)

2. Cloud API calls:
   - Delete ZopNight's added scaling policy
   - Restore the previous autoscaling group / scalable target config
   - For ECS: deregister the scalable target if previously not registered

3. The cloud autoscaling state returns to pre-ZopNight (byte-accurate)

4. Event row records the remove with restoration confirmation

EFFECTIVE OUTCOME:
  Cloud-side autoscaling = exactly what it was before ZopNight applied
  No residual ZopNight artifacts
  Audit log shows the full sequence

Remove is the “undo”: fully reversible to the pre-ZopNight state, byte-accurate.

What happens if Remove fails midway

SCENARIO: ZopNight tried to remove a policy
  Delete Policy call failed (rate-limited by AWS)
  The Restore step didn't execute

RESULT:
  Cloud is in an inconsistent state
  ZopNight's policy still exists
  ZopNight has logged it as "removing"
  Manual intervention needed

UI RESPONSE:
  Policy shown in "remove failed" status
  Restore step pending
  Operator clicks "Retry Remove" to complete

IDEMPOTENT RESTORE:
  Remove can be retried until it succeeds
  No partial-state corruption

The idempotency of Remove means engineers can recover gracefully from failures.

Adopted policies: different rules

Recall from M2.5: some policies are “adopted” (pre-existing in the cloud, observed but not managed by ZopNight). For adopted policies:

ADOPTED POLICY LIFECYCLE:

APPLY    No cloud mutation
         Records existing config as "previousConfig"

PAUSE    Effectively a no-op (ZopNight wasn't acting anyway)

RESUME   Effectively a no-op

REMOVE   Same as no-op (cloud-side state unchanged)

EDIT     Auto-promotes the policy from "adopted" to "recommended"
         (the ownership flag changes)

The user explicitly upgrades to ZopNight management when ready.

Adopting a policy is the safe default for existing cloud autoscaling. The customer chooses when to take over management.

The Edit → auto-promote flow

T+0      Customer is observing an adopted policy:
         min=4, max=12, target=60%

T+0      Customer wants to change target to 65%. Clicks Edit.
T+0      Modifies the value. Saves.
T+0      ZopNight: source flag flips from "adopted" → "recommended"
         The policy is now owned by ZopNight

T+5s     UI prompts: "Re-apply policy to push the new target value
                       to the cloud?"

T+15s    Customer clicks Re-apply
T+30s    Cloud-side autoscaler updated. Cooldown clock resets.

OUTCOME:
  Adopted → recommended (irreversible)
  Cloud-side now reflects ZopNight management

NOTE: Once recommended, policy stays recommended.
       Don't accidentally edit adopted policies if you want to
       preserve observation-only status.

Auto-promote is a one-way change. Once a policy is recommended (ZopNight-owned), it stays recommended.

Lifecycle audit trail

Every lifecycle operation is logged:

POLICY EVENT LOG:
  Apply:    "apply" queued, with the user; the executor's outcome
            follows as a "status_update" (completed or failed,
            error message, previousConfig saved)
  Pause:    timestamp, user (there is no reason field)
  Resume:   timestamp, user
  Remove:   "remove" queued, with the user; outcome follows as a
            "status_update"

RETRIEVAL:
  get_autoscaler_events MCP tool (or the policy's events endpoint)
  Or: Activity → Audit Logs (filter: Autoscaler Policy)

USE CASES:
  Debugging (when did this policy change?)
  Compliance (audit trail of automated actions)
  Rollback investigation (what was the previous config?)

The audit log preserves the complete history.

Common lifecycle mistakes

MISTAKE                              MITIGATION
──────────────────────────────────────────────────────────────────
Apply without testing on dev first    Use staging account; test before prod

Pause as a substitute for fixing      Pause is temporary; investigate
the underlying issue                  root cause

Remove + Apply (instead of Edit)      Edit + Re-apply is safer
                                      Preserves audit + history

Frequent policy churn                 Tune existing policies vs recreating
                                      Audit log gets cluttered

Forgetting that adopted Edit         Read M2.5 before editing adopted
auto-promotes

Race conditions during Remove          Idempotent retry; investigate
                                      cloud-side state

Most mistakes are about misunderstanding what operation to use. The decision matrix above clarifies.

2. Demo

A team’s policy lifecycle in 6 months:

MONTH 0    APPLY: created policy on prod-api ASG
           Mode: recommend
           Result: working as expected
           Audit log: policy created; previous_config = ASG had no scaling

MONTH 1    Team takes over approvals; mode edited to Autopilot
           Event log: update (no cloud change)

MONTH 3    Anomaly detected. Workload pattern changed.
           EDIT: tuned target value from 60% → 70%
           Re-apply to push to cloud
           Audit log: edit + re-apply

MONTH 4    Compliance event. Auto-scaling temporarily concerning.
           PAUSE: policy marked paused in ZopNight
           Result: the applied cloud rules keep scaling (Pause
                   makes no cloud call; halting needs Remove)
           Audit log: pause

MONTH 4.5  Compliance event resolved.
           RESUME: policy marked active again
           Audit log: resume

MONTH 6    Workload retired. Time to remove ZopNight's policy.
           REMOVE: cloud restored to pre-ZopNight config
           Policy archived in the policy event log
           Audit log: remove + restoration confirmation

Each lifecycle operation has a specific purpose. Together they cover the full management cycle.

3. Hands-on (5 min)

Walk a policy through the lifecycle in a sandbox:

□ STEP 1: APPLY a new policy
  Create policy via Quick Setup
  Observe cloud-side change
  Verify previous_config saved

□ STEP 2: PAUSE the policy
  Click Pause
  Verify the status shows Paused
  Verify the cloud-side policy is unchanged (it keeps scaling)

□ STEP 3: RESUME the policy
  Click Resume
  Verify the status shows Active
  Verify no cloud-side change

□ STEP 4: REMOVE the policy
  Click Remove
  Verify cloud-side restored to previous_config
  Verify ZopNight policy artifact gone

□ STEP 5: Review audit log
  Activity → Audit Logs (filter: Autoscaler Policy)
  Verify all 4 events logged

A 15-minute exercise reveals the lifecycle mechanics.

4. Knowledge check

Q1

A policy’s previousConfig is captured on Apply because:

A. Remove uses it to restore the cloud-side state to its pre-ZopNight configuration
B. For backups
C. As a performance optimization for the policy read path in the main API
D. Cloud requires it

Show answer

Correct: A. Byte-accurate restoration is the design goal; without previousConfig, rollback would be incomplete.

Q2

Editing an adopted policy:

A. It does not affect the ownership of that policy at all afterwards either
B. Pauses it
C. Deletes the policy
D. Auto-promotes the policy from “adopted” to “recommended”: ZopNight now owns it

Show answer

Correct: D. It is a one-way transition: the Edit changes ownership, and later operations treat the policy as recommended.

Q3

Pause and Remove differ how:

A. Pause only flags the policy in ZopNight; the cloud-side rules keep running
B. Same action
C. Pause also restores the previous cloud configuration, every single time
D. Pause is faster

Show answer

Correct: A. Pause is a status flip with no cloud call, so the applied rules keep scaling. Remove deletes ZopNight’s policy and restores the previous cloud-side configuration.

5. Apply

Automate → Auto Scaling → open any policy → Apply / Pause / Resume / Remove actions. Each is one click.

For your team: understand the full lifecycle before applying first policy. Reversibility is the safety net.

Glossary terms touched

Idempotency · Adopt-or-replace

Outcome

1. Concept

APPLY: pushing to the cloud

PAUSE: disable temporarily

RESUME: re-enable

REMOVE: complete rollback

What happens if Remove fails midway

Adopted policies: different rules

The Edit → auto-promote flow

Lifecycle audit trail

Common lifecycle mistakes

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.