Outcome
By the end of this lesson, you will be able to identify which rules can be one-click auto-remediated, explain the two independent checks a recommendation passes before an Apply button appears, and plan progressive enablement in your org.
| Tier | Engineer |
| JTBD | ”Know which rules support one-click apply; enable them progressively; understand why the denylist exists.” |
| Personas | Platform Engineer · FinOps Lead · SRE |
| Prerequisites | M2.1 · M2.2 · M2.3.L1 |
| Time | 9 minutes |
| Bloom verb | Identify (Apply), Explain (Understand), Plan (Create) |
1. Concept
Auto-remediation requires high confidence that the cloud action is safe, reversible-or-undoable, and tested. Eligibility is an allowlist in code (contract/autoremediation_allowlist.go), not a product feature called “certification.” The allowlist wires about 124 rules: roughly 28 as one-click auto and 96 as guided (a type-to-confirm review step). Every other rule is advisory: the recommendation shows, but no Apply button. (“Certified” in this lesson is just shorthand for “on the auto-remediation allowlist.”)
THE AUTO-REMEDIATION ALLOWLIST:
ON THE ALLOWLIST, AUTO (~28, one-click Apply): Tested end-to-end on real cloud accounts Failure modes catalogued Idempotent + safe Audit log verified
ON THE ALLOWLIST, GUIDED (~96, type-to-confirm): Same safety bar, plus a human confirmation step
NOT ON THE ALLOWLIST (advisory, manual execution): Rule logic verified Remediation steps shown Apply button NOT shown Customer executes manuallyThe allowlist grows over time as more rules are validated end-to-end.
What gets a rule onto the allowlist
A rule joins the allowlist when its lever has been proven end to end against real cloud, not just fixtures. Specifically:
ALLOWLIST CRITERIA (all must pass):
1. RULE LOGIC VERIFIED Across multiple sample resources Edge cases tested
2. REMEDIATION WORKFLOW TESTED ON REAL AWS / GCP / Azure Not just sandbox; real cloud accounts Real resources; real cleanup
3. FAILURE MODES CATALOGUED Every error class identified Right user_action / transient / system classification Fix hints documented for each
4. IDEMPOTENCY PROVEN Re-runs are safe Tested with concurrent calls No partial-state scenarios
5. ROLLBACK PATH DOCUMENTED Snapshot before destructive action Restore procedure tested Recovery time measured
6. AUDIT LOG ENTRIES VERIFIED Every action recorded correctly Reproducible from audit trail
7. SAMPLE CUSTOMER ADOPTION TRACKED Run in pilot customers No incidents over observation period Edge cases caught + fixedOne more requirement is easy to miss and is the reason the bar holds: every allowlisted rule ships a paired dispatch test that drives the literal JSON the recommender emits through the real dispatcher with real handlers registered. Producer and consumer live in separate modules, so capturing the wire bytes on one side and replaying them on the other is the only durable way to stop the two drifting apart. A rule without that paired test does not go on the list.
The allowlist is only the first of two checks
This is the part that surprises people. Being on the allowlist means the rule supports automation. It does not mean the Apply button appears on a given card. A second gate re-decides per recommendation, against that specific resource:
TWO TIERS:
TIER 1 allowlist "does this RULE support auto?" keyed on rule id, nothing else
TIER 2 safety gate "is it safe on THIS resource?" reads authoritative control-plane fields on the recommendation, never tags and never name tokens
Tier 2 disagrees -> the recommendation is DEMOTED to advisory. It is still shown. Nothing is suppressed.What the gate demotes, every time:
one-shot stop of compute always demoted. The workload inside a generic instance is opaque to every cloud API, so "this is not a bastion" can never be proven. Use a recurring schedule instead.
delete of a stateful resource demoted. Managed database types, a non-empty engine field, or a data tier.
delete without proof demoted unless an authoritative field proves the thing is unused: an elastic IP explicitly not associated a volume flagged unattached with a real detach time at least 7 days old a snapshot whose source is provably gone a backup vault with zero protected items
resize needing a stop demoted on a stateful resource.Absent evidence is not evidence. The gate fails closed, so a new rule inherits this protection without anyone remembering to add it. It is also why the summary pill on a rule group and the pill on an individual recommendation can disagree: the first reflects capability, the second reflects this decision.
What’s on the allowlist
The authoritative auto and guided sets live in contract/autoremediation_allowlist.go (about 28 auto + 96 guided). Treat that file as the source of truth; the entries below are illustrative of the categories, not a fixed list.
AUTO (one-click), illustrative: RC-002 Orphan EBS volume delete (only with authoritative orphan evidence) K8s scale-to-zero: idle Deployments, suspend CronJobs (EKS / GKE / AKS) Pause services: App Runner, Beanstalk, Azure Databricks cluster
GUIDED (type-to-confirm), illustrative: RC-021 Orphan EBS snapshot delete (irreversible, so human-confirmed) Single-replica K8s bump 1->2 (RC-1706 / 1806 / 1906) RC-214 Azure Hybrid Benefit (a reversible licensing setting)
NOT auto-remediable (advisory only): RC-001 Idle EC2: advisory. It fires on already-stopped instances and recommends termination; it is explicitly OFF the allowlist. Rightsizing, compliance, network, and IAM changesThe allowlisted rules cluster around the simplest, safest categories: orphan deletes backed by evidence, scale-to-zero, and pause/resume. Rightsizing and complex changes are deliberately advisory.
Why advisory rules don’t have Apply
For advisory rules, the recommendation card hides the Apply Remediate button. The remediation steps are still shown: but as instructions for manual execution.
ADVISORY RULE CARD example: RC-006 Oversized EC2─────────────────────────────────────────────────────────RC-006 · Oversized EC2 Instance[no Apply button]
REMEDIATION (manual)1. Review CloudWatch CPU and memory utilization2. Use AWS Compute Optimizer for sizing recommendations3. Stop the instance, change instance type, restart4. Monitor performance after the change for 48 hours
[Console URL: https://us-east-1.console.aws.amazon.com/ec2/...][Dismiss] [Mark Applied] [Snooze]The “Mark Applied” button lets the customer note they’ve completed the action manually, so savings tracking works correctly.
What it takes to add a rule to the list
ADDING A RULE:
PHASE 1: REAL-CLOUD TESTING (4-8 weeks): Sandbox accounts don't count Pre-prod customer accounts (with consent) used Multiple regions; multiple resource configurations
PHASE 2: FAILURE HANDLING (1-2 weeks): Catalog every failure mode Map to error class (user_action / transient / system) Write fix hints
PHASE 3: ROLLBACK DOCUMENTATION (1 week): For idle EC2: restart from snapshot For pause: resume For irreversible (delete): final-snapshot policy Test rollback procedure
PHASE 4: SAMPLE ADOPTION (2-4 weeks): Run with consenting pilot customers Track all incidents Iterate until clean
PHASE 5: DENYLIST REMOVAL: Update certification status Apply button appears Documentation updated Customer notification
TOTAL: 8-16 weeks per rule typicalThe slow pace is intentional. Quality > velocity for auto-remediation.
Customer-requested additions
A TEAM CAN REQUEST a specific rule be allowlisted:
Process: 1. File request with ZopNight (support / account team) 2. ZopNight engineering adds to certification queue 3. Runs end-to-end tests with consenting customers 4. Catalogs failure modes 5. Removes from denylist; Apply button appears
TYPICAL TIMELINE: 2-6 weeks
COMMON REQUESTS: Rules that the customer manually applies frequently Rules with high $ savings impact Rules where manual execution has been cleanThe certification queue is responsive to customer demand.
When uncertified ≠ unsafe
UNCERTIFIED rule isn't "unsafe to apply" Just hasn't passed ZopNight's bar for one-click execution The remediation is still shown Customer executes manually with full context
COMMON UNCERTIFIED CATEGORIES: Rightsizing (instance class changes) Risk: workload-specific; needs validation Reason: tested only in narrow workloads
Compliance configuration changes Risk: policy interpretation varies Reason: customer policy varies
Network changes Risk: high blast radius Reason: cross-team dependencies
IAM changes Risk: lockout potential Reason: hard to test without breakingCustomer judgment fills in. Manual execution is still safe.
Progressive enablement pattern
MONTH 0: NO AUTO-REMEDIATION Manual triage on all findings Average remediation latency: 4 weeks Reason: building trust + auditing scope
MONTH 1: ENABLE LOW-RISK CERTIFIED (RC-001, RC-002) ~30% of findings now auto-remediate Latency on those: <1 hour Trust validated
MONTH 3: ENABLE K8S SCALE-TO-ZERO (RC-1701, RC-1702) ~50% of findings auto-remediate
MONTH 6: ENABLE PAUSE RULES ~65% of findings auto-remediate Remaining 35% (rightsizing, complex) still manual
ONGOING: AS CERTIFICATION GROWS Additional rules enabled per quarter Manual residual shrinks Customer team focuses on judgment callsThe pattern: enable conservatively; trust grows; rate of enablement increases.
Default state: opt-in
DEFAULT for ALL CERTIFIED rules: Auto-remediation OFF Customer-controlled per-rule per-org
ENABLING a rule: Settings → Auto-Remediation Toggle ON for the specific rule Approval policies configurable
WHY OPT-IN BY DEFAULT: Conservative Customer-controlled Build trust through visibility first Customer chooses when readyDefault off; opt-in deliberately. Trust comes from control.
2. Demo
A team’s progressive adoption pattern over 6 months:
MONTH 0: STARTING POINT: No auto-remediation enabled 100% manual triage Average latency: 4 weeks finding → action ~10% of findings actually acted on (backlog grows) Savings realization: poor
MONTH 1: ENABLE LOW-RISK: Enable RC-001 (idle EC2), RC-002 (orphan EBS) Pre-approval: NOT required (low blast radius) Conservative Results after 1 month: 30% of findings now auto-remediate Latency on those: <1 hour Zero incidents
MONTH 3: ENABLE K8S: Add RC-1701, RC-1702 (K8s scale-to-zero / suspend) Pre-approval: still skipped (reversible) Results after 1 month: 50% of findings auto-remediate Cross-team adoption (platform + workload teams)
MONTH 6: ENABLE PAUSE RULES: Add RC-303, RC-304, RC-401, RC-402 (pause services) Pre-approval: required for prod (configurable) Results after 1 month: 65% of findings auto-remediate Manual residual: rightsizing, compliance (always manual)
MONTH 12: STEADY STATE: All available certified rules enabled ~70% of findings auto-remediate Median latency: <1 day Engineer time on routine actions: dropped 80%
REALIZED EFFECT: Time-to-remediation: 4 weeks → <1 day Savings realization: from 30% to 85% Engineer satisfaction: high Zero incidents from auto-remediationThe progressive enablement is the safe path. Trust through demonstrated reliability.
3. Hands-on (5 min)
Audit your auto-remediation settings:
□ STEP 1: Open Settings → Auto-Remediation
□ STEP 2: List enabled rules Currently enabled: _____ Total certified available: _____ Adoption: _____%
□ STEP 3: Identify candidates to enable Low-risk certified rules off: RC-_____: __________ RC-_____: __________
□ STEP 4: Plan enablement Pick 1-2 to enable this month Pre-approval policy: __________ Monitor for: ___ weeks Owner: __________
□ STEP 5: Set quarterly review Date: __________ Owner: __________ Goal: track # of certified rules enabled over quartersA 15-minute audit reveals adoption gap. Quarterly cadence drives progressive enablement.
4. Knowledge check
Q1
A recommendation card doesn’t show Apply Remediate. Most likely cause:
A. The user lacks permission
B. Either the rule is advisory, or it is allowlisted but the per-recommendation safety gate demoted it on this resource. Either way the remediation steps and console link are shown for manual execution, and the card still offers “Mark Applied” for tracking. Manual does not mean unsafe.
C. The cloud is down
D. Bug
Show answer
Correct: B. Two different causes produce the same missing button, and the distinction matters when a customer asks why one instance has Apply and another does not.
Q2
About 124 rules are on the auto-remediation allowlist (~28 auto + ~96 guided). The rest of the 450+ rules:
A. Don’t exist
B. Are advisory: shown with remediation steps but no one-click apply. The allowlist grows over time as more rules are validated end-to-end; a customer can request a specific rule be added.
C. Are buggy
D. Will never be remediated
Show answer
Correct: B. The list grows as levers are proven, and it shrinks too: rules have been removed from it when an audit showed the lever was unsafe or the saving was not concrete.
Q3
Auto-remediation is enabled per-rule per-org. The default state:
A. All allowlisted rules on
B. All rules off (opt-in). Customer-controlled. Default conservative: trust built through visibility before action. Customer enables when ready, progressively, per their risk model.
C. Random
D. Critical rules on by default
Show answer
Correct: B. Default off; customer chooses when ready.
5. Apply
Settings → Auto-Remediation lists certified rules with toggle. Enable progressively as trust builds.
For your team: 1-2 new rules enabled per quarter is a sustainable pace.
Related lessons
- L1: 3-step workflow
- L3: Approval gate (next)
- L4: Error classes
- L5: Database denylist
- L6: Terminal notifications
Glossary terms touched
Certified rule · Denylist · One-click apply · Mark applied