Outcome
By the end of this lesson, you will be able to detect tag drift before it accumulates, classify drift events by source, and resolve each event with the right pattern (trust cloud / trust code / re-derive).
| Prerequisites | M3.1.L1 · M3.1.L2 · M3.1.L3 |
| Time | 9 minutes |
1. Concept
Drift is when the tags in your cloud stop matching what your policy or your code says they should be.
It happens even with inheritance set up well. Somebody edits a tag in the console. A script rewrites one. A deployment is rolled back and takes the tags with it.
None of it is announced, so unless something is looking, it accumulates.
WHERE DRIFT COMES FROM:
1. MANUAL CLOUD CONSOLE CHANGES Engineer fixes a tag in a hurry via console IaC doesn't know; next plan shows a diff Or: console wins; IaC silently overwritten on next apply
2. TERRAFORM/CDK APPLY WITH NEW VALUES Tag policy updated; old resources retain old values Until re-applied, drift exists
3. CLOUD-SIDE AUTOMATION Auto-scaling adds resources without tags Service Catalog deployments use different conventions Backup automation copies resources with original tags
4. MULTI-TEAM CONFLICTS Team A renames "team=growth" to "team=monetization" Team B's resources still tagged team=growth (forgot to sync) Org-wide rename incompleteThe 6-hour discovery cadence catches drift within the day; weekly review keeps it from accumulating.
How to detect drift yourself
Drift detection is a comparison you run on a schedule. You compare what the cloud says with what your code and your tag policy say, and you keep the result.
EVERY DAY (a scheduled job in your CI or a cron runner): 1. Read live tags AWS: AWS Config rule required-tags, or the Resource Groups Tagging API (get-resources) Azure: Azure Resource Graph query over tags GCP: Cloud Asset Inventory (gcloud asset search-all-resources) 2. Read expected tags IaC: terraform plan -refresh-only (shows tags changed outside code) Policy: your tag-policy.yaml (required keys, allowed values) 3. Diff Tag changed outside IaC → drift event Value not in the allowed list → policy violation Required key missing → coverage gap 4. Store the result (a table or a CSV per run) and post a summary to your channel from the same pipelineThe detection is cheap; the resolution is the work.
ZopNight helps with two of the three outputs. Cost Reports, then the Tags tab, has a Tag Coverage card that shows how much spend is tagged, and tagging policies (Governance, then Policies, then Tagging) report violations after each 6-hour discovery sync. ZopNight does not keep an expected-tag baseline or compare against your IaC, so the drift diff itself is your job.
What a drift event looks like
DRIFT EVENT (one row from your daily diff): Resource: i-0abc123 Account: prod-us Region: us-east-1 Resource type: aws_instance Resource name: payment-api-server-3
Tag in drift: environment IaC declares: dev Cloud actual: prod Detected: 2026-05-21 daily run (06:00 UTC) Previous run: 2026-05-20 (was dev)
Context (pulled by the job or by the reviewer): CloudTrail: jane@platform called CreateTags at 12:15 UTC IaC last apply: 2026-05-20; declared "dev" Likely cause: Manual console edit; either intentional fix or mistakePull the “who changed it” line from the provider’s own audit trail (AWS CloudTrail, Azure Activity Log, GCP Cloud Audit Logs). That is what makes an event quick to resolve.
Resolution patterns: three options
PATTERN A: TRUST CLOUD (cloud is now source of truth) WHEN: Engineer made a deliberate change Cloud-side value is correct IaC was outdated
ACTION: Open a PR so the IaC declares the new value Document the reason in the PR
EXAMPLE: tagging fix discovered during incident; cloud-side correct
PATTERN B: TRUST CODE (revert cloud) WHEN: Unauthorized change Cloud-side value violates policy IaC source-of-truth should win
ACTION: Re-apply the IaC (or revert the tag by hand) Investigate the source of the change (cloud audit trail) Reach out to the engineer if it was a mistake Update process if it was a gap
EXAMPLE: stray "environment=qa" tag (not in allowed values)
PATTERN C: RE-DERIVE (both sides are stale) WHEN: Neither IaC nor cloud holds the right value Common after a reorg
ACTION: Decide the correct value from ownership records If a ZopNight tagging policy covers the key, Smart Tags derives a value from the resource's fields; a person accepts it and can write it back with Apply to cloud Update the IaC so the next apply keeps it
EXAMPLE: post-reorg drift; both declared and actual are staleThe choice depends on the source. Reading the cloud audit trail and change timestamps usually makes it obvious.
Detection cadence
EVERY 6h: ZopNight discovery refreshes tags, coverage and tagging-policy violations
EVERY DAY: Your drift job runs and stores its diff It posts to Slack only if new events > 10 (your threshold; cap it to avoid alert fatigue)
EVERY WEEK: Friday Operate review: 15-30 minutes Triage week's drift; resolve each
EVERY QUARTER: Drift trend review If consistent 5%+ drift: structural issue; investigate root causeThe weekly cadence is the workhorse. Most drift gets resolved within 7 days of occurrence.
Drift trending
Store each run, then cohort the drift events by week. Healthy orgs stay flat; unhealthy orgs trend up.
DRIFT RATE TRENDING (resources with drift / total resources):
LOW (<2% of resources): Healthy Normal weekly fluctuation
MEDIUM (2-5%): Manageable Investigate sources; one-off issues common
HIGH (>5%): Tagging discipline breakdown Structural issue Investigate: bad IaC, manual changes, auto-policies, multi-team conflict
RISING (week over week): Recent regression New automation? Recent reorg? Investigate immediatelyThese bands are a working rule of thumb, not an industry standard. The trend matters more than the absolute number. 3% steady is fine; 3% rising is not.
Tag-policy drift (a level up)
Drift isn’t only cloud-side: the policy itself can drift:
POLICY-LEVEL DRIFT: Informal changes ("we agreed to add 'project' tag in standup") New approved values added without updating the canonical list Deprecated values not sunset Team-level deviations from org policy
PROTECT THE POLICY: Documented version in repo (tag-policy.yaml) Changes via PR + review (1-week review minimum) Communicate changes to teams via #eng-platform Quarterly review of policy itselfThe tag policy is itself a long-lived artifact. Treat it like a contract.
Prevention beats detection
Detection finds drift after the fact. Three controls stop much of it at the source:
PREVENT AT THE CLOUD: AWS Organizations tag policies (allowed values per key) Azure Policy "Require a tag" / "Inherit a tag" effects GCP organization policy plus label checks in CI
PREVENT IN CODE: Provider default_tags (Terraform AWS provider) or module defaults A policy check on terraform plan in the PR (ZopNight IaC governance can run as that check)
BACKFILL WHAT IS LEFT: ZopNight Smart Tags derives values for resources a tagging policy covers; values stay pending until a person accepts them Smart Tags does not normalize existing values ("PROD" → "prod") or resolve drift; that is a policy plus IaC changePrevention plus a daily diff is the steady state for mature orgs.
Anti-patterns: what kills tag hygiene
ANTI-PATTERN FIX──────────────────────────────────────────────────────────────────Nobody reads the drift report Weekly Friday review 15-30 minutes
Resolving without root cause Investigate WHY drifted Fix the source, not just the symptom
Mass "trust cloud" without review Pattern A is fine sometimes But accepting everything = giving up on tag policy
Tags fixed but IaC not updated Open PR to sync IaC Otherwise next apply reverts your fix
No alert for high drift Have the drift job post to Slack on >10 new events/day or >5% rateMost of these are habits, not technical problems. The discipline of weekly review is the main investment.
Drift in regulated environments
For SOC 2 / ISO 27001 / regulated workloads:
ADDITIONAL CONTROLS: Drift events logged for audit Resolution recorded (who decided, why) Quarterly review by FinOps + Security Annual third-party audit verifies process
COMPLIANCE FRAME: Drift = control deficiency until resolved Reasonable resolution time (e.g., 7 days) Documented process for unauthorized changes
Keep the stored drift runs and the PRs that resolved them; togetherthey are the audit trail.For unregulated orgs: weekly review is sufficient. For regulated: keep the trail and review evidence quarterly.
2. Demo
A weekly Operate cadence at a 30-team org:
FRIDAY 2pm: TAG DRIFT REVIEW (Platform team + FinOps lead)
OPEN THE WEEK'S DRIFT REPORT (from the daily job): This week: 18 drift events Last 4 weeks: 22, 19, 21, 18 events (flat trend; healthy) Current drift rate: 1.8% (target <2%)
CATEGORIZE the 18 events:
Source breakdown: Terraform-driven (legitimate): 12 Manual console changes: 5 - 4 intentional fixes - 1 unknown source (investigate) Smart Tags write-back: 1
RESOLUTIONS:
TRUST CLOUD (Terraform applied a merged change; baseline updated): 10 events resolved 2 events deferred: the merged PR is not applied in every workspace yet; they clear at the next IaC sync
TRUST CLOUD (manual fixes that were correct): 4 events resolved Engineer documents the change in the PR description Open PR to sync IaC where applicable
TRUST CODE (unauthorized change): 1 event: jane@platform changed environment=prod → dev on a prod resource via console CloudTrail: jane was experimenting; left wrong tag Resolution: revert to environment=prod Followup: 5-min Slack DM to jane; document for future
TRUST CLOUD (Smart Tags write-back was right): 1 event: an accepted Smart Tag for "team" was applied to cloud Derived from the resource name; visually confirmed Resolution: add the value to the IaC
TOTAL TIME: 22 minutes for 18 eventsRESOLVED: 16 of 18 (2 Terraform events deferred to the next IaC sync)FOLLOWUPS: 3 (jane reminder + 2 IaC sync PRs)
LOG entry to #finops-weekly: "Week 21 tag drift review complete. 18 events: 16 legit changes, 1 unauthorized (resolved + educated), 1 Smart Tags write-back. Drift rate stable at 1.8%; no structural issues."The 22-minute weekly investment keeps drift bounded. Skip it and drift accumulates quietly, one console edit at a time.
3. Hands-on (5 min)
Check your tag drift posture:
□ STEP 1: Run one drift check by hand AWS: AWS Config required-tags compliance, or terraform plan -refresh-only on one workspace Azure / GCP: a Resource Graph or Asset Inventory query on tags Resources with drift: _____ Drift rate: _____ %
□ STEP 2: Read coverage in ZopNight Cost Reports → Tags → Tag Coverage card: _____ % of spend tagged
□ STEP 3: Triage the top 5 drift events Categorize by source: 1. __________ Source: __________ Resolution: __________ 2. __________ Source: __________ Resolution: __________ 3. __________ Source: __________ Resolution: __________ 4. __________ Source: __________ Resolution: __________ 5. __________ Source: __________ Resolution: __________
□ STEP 4: Make it a habit Daily drift job owner: __________ Weekly review day / time: __________ Duration: 20-30 minOne manual run shows the drift posture. A daily job and a weekly review keep it stable.
Do it through MCP. A task from this lesson, asked in one sentence.
BEFORE A ZopNight account with one cloud connected. A tag convention you already use, even an informal one.ASK "Show me tagging violations for the last week and set a policy for the missing owner tag."CHECK the violation count a week later. A policy nobody is measured against produces a stable, ignorable number.Tools behind it: list_tagging_violations (read, Govern), get_tagging_policy_catalog (read, Govern), create_tagging_policy (write, tier 2, reversible). Connection details are in the MCP integration docs.
4. Knowledge check
Q1
The most reliable way to detect tag drift on IaC-managed resources is:
A. Wait for the monthly invoice to show it
B. Ask each team to self-report changes
C. Re-tag everything by hand each quarter
D. A scheduled diff of live tags against IaC
Show answer
Correct: D. A daily job that reads live tags (AWS Config, Resource Graph, Asset Inventory) and compares them with the IaC (terraform plan -refresh-only) catches drift within a day. Invoices and self-reporting are too late or too patchy; manual re-tagging fixes nothing at the source.
Q2
Your drift job flags a tag that changed because Terraform CI applied a value from a merged PR:
A. Revert the cloud tag to last week’s value
B. Block the whole CI pipeline until the drift has been fixed properly first
C. Likely “Trust cloud”: the merged Terraform change is the legitimate source
D. Reject the change
Show answer
Correct: C. Update the job’s baseline to match. If the CI change was wrong, fix the IaC; but generally CI-driven changes are legitimate. Source of truth wins. Terraform = legitimate source.
Q3
A team with a consistent 5%+ drift rate week over week:
A. Random fluctuation
B. Expected on any large estate
C. Fine; tags drift
D. Tagging discipline breakdown
Show answer
Correct: D. Structural issue; investigate sources: bad IaC, frequent manual changes, missing cloud policy, multi-team conflict. The 5% rate compounds; address root cause not symptoms. Structural issue at 5%+. Address root cause.
5. Apply
Run a daily drift diff and a weekly review (15-30 minutes typically). Resolve drift events with the right pattern. Investigate the source for unauthorized changes. Alert from your own pipeline at the thresholds you choose.
For ZopNight: Cost Reports → Tags shows the Tag Coverage card; tagging policies (Governance → Policies → Tagging) and Smart Tags cover violations and backfill. The drift diff and its alerts are yours to build.
Related lessons
- L1: Tags as organizational debt
- L2: MVT
- L3: Inheritance + propagation
- L5: Reorg-proof tagging (next)
- T6: AI-Powered Cloud Ops (query tag coverage via MCP)