The Recommendation Safety Gate · ZopDev University

Outcome

By the end of this lesson, you will be able to explain the two-tier model that decides whether a lever runs automatically, name the four authoritative evidence keys the gate accepts, and predict which rules are demoted to advisory and why.


Tier	Engineer
JTBD	”Know why a rule that says it supports auto-remediation still shows no Apply button on this resource.”
Personas	Platform Engineer · Security/Compliance · SRE
Prerequisites	M2.3.L2 (the allowlist) · M2.3.L5 (the database denylist)
Time	9 minutes
Bloom verb	Explain (Understand), Name (Remember), Predict (Analyze)

1. Concept

The allowlist from L2 has one blind spot, and it is built into how it works: it only knows about rules.

It can answer “may this rule act on its own”. It cannot answer “should it act on this resource”, because it never looks at the resource at all.

That is a real gap, not a theoretical one. A rule marked auto could one-shot-stop a bastion host, delete a still-attached MySQL data disk, or delete a 90-day production snapshot whose source volume still exists. Per-rule guards were tried and were inconsistent, and each new rule re-introduced the risk.

The Safety Gate is the answer. Every recommendation passes through one central check, so being safe is a property of the system rather than something each rule’s author has to remember to do.

The two-tier model

TIER 1: THE ALLOWLIST          "does this RULE support auto?"
  Keyed on rule id. Resource-blind.
  Surfaces as the rule-grouped summary pill.

TIER 2: THE GATE               "is it safe on THIS resource?"
  Resource-aware. Reads the recommendation's own Specs.
  Surfaces as the per-recommendation pill.

Both must pass for a one-click Apply to appear. This is why the two pills can disagree, and why “the rule supports auto-remediation” is never a complete answer to “why is there no Apply button here.”

Three principles, and they are the whole design

1. AUTHORITATIVE FIELDS ONLY
   The gate reads control-plane fields from rec.Specs.
   It NEVER reads tags or resource-name tokens. Those are
   customer-set and opaque, so they are not authoritative
   for a destructive action. A resource named "test-db" may
   be production; a tag saying env=dev may be six months stale.

2. FAIL CLOSED
   For a destructive lever, unknown or absent evidence means
   NOT auto. The recommendation is demoted to advisory and a
   human confirms. IN-USE is cheap to prove; ORPHAN must be
   proven.

3. CLOUD-OPAQUE MEANS ADVISORY
   No cloud API exposes the workload running inside a generic
   compute instance. Database, SAP, Kafka, bastion, privileged
   access workstation: they all look identical from outside.
   So a one-shot stop on compute can never be proven safe.

The matrix

STEP KIND                       GATE DECISION
──────────────────────────────────────────────────────────────
stop_compute                    DEMOTE, always. A one-shot
                                stop is an outage and the
                                workload is opaque. Use a
                                recurring schedule instead.

delete_resource                 DEMOTE if the resource is
                                stateful, OR if there is no
                                authoritative orphan evidence.
                                Otherwise allow.

resize_with_stop                DEMOTE if the resource is
                                stateful. Do not auto-reboot
                                a database.

resize_resource (hot)           ALLOW
pause_service                   ALLOW
set_lifecycle_policy            ALLOW    non-destructive,
edit_k8s_manifest               ALLOW    reversible, or
                                         already guarded

The four authoritative orphan-evidence keys

For delete_resource, the gate allows auto only when the recommendation’s Specs prove the resource is unused. Flags are read truthily (true/yes/1), and false must be explicit (false/no/0). Absent is not evidence.

ip_associated            explicitly false
                         Elastic IP / Azure Public IP, read
                         from the provider's own association
                         back-references

unattached  AND          block storage, where the dwell uses
detached_since ≥ 7 days  the discoverer's REAL last-detach
                         time

source_disk_exists       explicitly false, or
/ source_exists          source_disk_gone truthy: a snapshot
                         whose source is provably gone

protected_items_count    == "0": an empty Azure recovery
                         vault, so deleting it loses no backups

The detached_since versus orphan_since distinction is worth holding on to. orphan_since is producer-ambiguous: AWS EBS stamps it with the volume’s CreateTime, so trusting it would auto-delete a volume detached ten minutes ago in the middle of a migration. The gate reads the dedicated detached_since or it abstains.

Stateful classification, without name tokens

A resource is STATEFUL if any of:
  - its TYPE is a managed data service
    (rds / aurora / elasticache / redshift / documentdb /
     neptune / memorydb / cloudsql / alloydb / memorystore /
     bigtable / spanner / azure-sql / *-flexible / cosmosdb /
     redis ...)
  - it carries a non-empty `engine` spec
  - its `tier` is one of data, database, db

Type, engine and tier are all control-plane facts. Nothing here reads a name.

What this changed in practice

EARNED AUTO (evidence-backed)
  RC-112 (GCP) / RC-212 (Azure) disk  via unattached +
                                      detached_since
  RC-016 EIP / RC-1363 Azure public   via ip_associated
  RC-1392 empty recovery vault        via protected_items_count
  plus every non-destructive lever

DEMOTED TO ADVISORY (fail-closed)
  RC-201, RC-1212, RC-149    auto stop_compute: the guest
                             workload is cloud-opaque
  RC-002 orphan EBS          AWS gives no real last-detach
                             signal from one DescribeVolumes
                             call, so detached_since is
                             unavailable
  RC-143 GCP snapshot        age-only, no source-gone check

GUIDED RATHER THAN AUTO
  RC-021 EBS snapshot        earns the evidence-backed delete
                             lever, but ships guided: deletion
                             is irreversible and source_disk_gone
                             cannot see an AMI that still
                             references the snapshot

A demoted recommendation is still surfaced. It renders as advisory with autoRemediable=false, so the customer reviews and applies manually. Nothing is suppressed; only the automation is withheld.

Extending the gate

The wrong way to re-enable auto for a rule is to relax the gate. The right way is to make the rule’s discoverer producer emit the authoritative evidence key from a documented control-plane field. RC-002 is the open case: AWS EBS detached_since needs cross-run detach tracking because no single-call field exists, and until that lands, RC-002 stays advisory.

2. Demo

The same rule, two resources, two verdicts:

RULE: RC-021 Orphaned EBS Snapshot
ALLOWLIST: yes, with Guided=true

RESOURCE A: snap-0a91f3c2
  Specs:  source_disk_gone = "Yes"
          age_days = 412
  ALLOWLIST: pass
  GATE:      delete_resource, not stateful, authoritative
             orphan evidence present -> lever allowed
  RENDERED:  Remediate button, type-to-confirm modal

RESOURCE B: snap-7f22b8e1
  Specs:  age_days = 380
          (no source_disk_exists, no source_disk_gone)
  ALLOWLIST: pass          <- the rule-grouped pill still
                              says this rule supports auto
  GATE:      delete_resource, no authoritative orphan
             evidence -> FAIL CLOSED
  RENDERED:  advisory. "How to Fix" playbook, no button.

─────────────────────────────────────────────────────────
THE SUPPORT TICKET THIS PREVENTS

  "Why does RC-021 have an Apply button on one snapshot
   and not the other? Is it broken?"

  It is not broken. Snapshot B has no proof its source is
  gone. Age alone is not evidence of orphan status: a
  400-day-old snapshot can be the only copy of something.
  The gate would rather make you look than guess.

3. Hands-on (6 min)

1. Open Recommendations, filter to category `orphan`.
   Find one recommendation showing a Remediate button and
   one advisory recommendation from the SAME rule.
   Rule id: ____________

2. For the advisory one, open the Evidence panel. Which
   of the four evidence keys is absent?
   ______________________________________________

3. Find any recommendation whose step kind is
   stop_compute. Is it ever auto?          Y / N
   Explain in one line why not:
   ______________________________________________

4. Pick a stateful resource (RDS, Cloud SQL, Azure SQL)
   with a resize recommendation. Is it resize_resource
   or resize_with_stop?          ________________
   Which of those does the gate demote, and why?
   ______________________________________________

5. Name the one thing you could ask a ZopNight engineer
   for that would legitimately turn an advisory RC-002
   into an auto one:
   ______________________________________________

Answer to 5: a discoverer producer that emits a real detached_since for AWS EBS, which needs cross-run detach tracking. Not a gate change.

4. Knowledge check

Q1

A rule appears on the auto-remediation allowlist, but a specific recommendation from it renders as advisory with no Apply button. The explanation:

A. The recommendation is stale and should simply be regenerated from the current inventory data instead of using this one
B. The user lacks the remediation permission
C. The allowlist is keyed purely on rule id and is resource-blind; the Safety Gate is resource-aware and evaluated it separately
D. The rule was recently retired

Show answer

Correct: C. The rule-grouped pill reflects capability, the per-recommendation pill reflects the gate’s verdict on that resource. Both tiers must pass. This two-pill disagreement is the single most common support question about auto-remediation, and “the rule supports auto” is never a complete answer to it.

Q2

Why can a stop_compute step never be auto-remediated, even on a resource tagged env=dev?

A. Because stopping is irreversible
B. Because no cloud API exposes the workload running inside a generic compute instance, so a one-shot stop can never be proven safe
C. Because compute resources are always stateful, and stopping one always risks losing the data held on any of them
D. Because it requires an approval gate instead

Show answer

Correct: B. The gate reads authoritative control-plane fields only and never tags, which are customer-set and can be stale or wrong. The intended alternative is a recurring schedule, which is reversible by design and is the path the schedule-category recommendations deep-link to. A tag is exactly the kind of evidence the gate deliberately refuses.

Q3

For a delete_resource step, the gate finds no unattached flag at all in the recommendation’s Specs. It treats this as:

A. Equivalent to unattached = false, so the resource is in use
B. A reason to re-run discovery before deciding
C. Equivalent to unattached = true, since orphans are always the common case
D. Not evidence either way, and fails closed: the recommendation is demoted to advisory

Show answer

Correct: D. In-use is cheap to prove; orphan must be proven, and absent is never proof. Note the asymmetry in how flags are read: truthy values are true/yes/1, and false must be explicit. Absence is a third state, and it always loses.

5. Apply

When triaging a rule whose recommendations behave inconsistently, check the per-recommendation pill before assuming a defect. Then open the Evidence panel and look for the four keys: ip_associated, unattached with detached_since, source_disk_gone, protected_items_count.

If a rule in your estate is stuck advisory and you believe it is safe, the ask to bring to ZopNight is a producer change that emits the evidence key, not an exemption.

Glossary terms touched

Safety Gate · Fail closed · Orphan evidence · Stateful classification

Outcome

1. Concept

The two-tier model

Three principles, and they are the whole design

The matrix

The four authoritative orphan-evidence keys

Stateful classification, without name tokens

What this changed in practice

Extending the gate

2. Demo

3. Hands-on (6 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.