Outcome
By the end of this lesson, you will be able to explain why an accepted Smart Tag can disappear without anyone revoking it, diagnose the three causes, and decide whether a given drift event is the tag’s problem or the estate’s.
| Prerequisites | M2.8.L3 |
| Time | 9 minutes |
1. Concept
The rule that governs everything in this lesson is one sentence:
An accepted tag is kept only while it is still derived, with the same value.
Reconciliation runs on every refresh. It is not a periodic cleanup job you can schedule around; it is a condition that is continuously true.
ON EVERY REFRESH, FOR EVERY ACCEPTED TAG: is the policy still deriving this key on this resource? no -> the accepted tag is dropped is it still deriving THE SAME VALUE? no -> the accepted tag is dropped; the new value appears as pending yes to both -> the accepted tag survivesThis is the opposite of how a cloud tag behaves. A cloud tag persists until someone changes it. A Smart Tag persists only while its derivation still holds. That difference is the source of every surprise in this lesson.
The three ways an accepted tag disappears
1. THE RESOURCE CHANGED The policy derives from the resource's fields (and matches on them in its selector). Change one of those and the derived value changes with it.
Renamed payments-api-03 -> checkout-api-03 team = payments (accepted) is dropped team = checkout appears as pending Nobody did anything wrong. The derivation followed the rename, which is what it is for.
2. THE POLICY CHANGED Editing the policy re-derives everything it touches. An accepted tag drops only if its derived value changes, or stops being derived; a value that comes out the same survives.
An edit that changes EVERY value the policy produces (payments -> payments-api, platform -> platform-eks) drops every accepted tag under that key at once.
3. A TAG THE POLICY READS CHANGED A policy can derive its value from an existing tag (tag.<key>). Edit that tag in the cloud and the derived value changes, so the accepted tag drops. Editing the cloud copy of the Smart Tag's own key does NOT, on its own, return it to pending: reconciliation compares accepted against derived, not against the live cloud tag.The one that looks like a bug and is not
Cause 2 produces the alarming version: an admin makes a small policy edit and thousands of accepted tags vanish, taking the attribution with them and pushing spend back into Unattributed.
WHAT IT LOOKS LIKE Monday: Unattributed 9% Tuesday: Unattributed 34%, nobody revoked anything
WHAT HAPPENED Someone edited the team policies' values. The edit changed every value they produced, so every accepted tag under that key dropped and reappeared as pending under a new value.
WHY IT IS CORRECT The alternative is worse: keeping the old accepted values would mean your reports attribute spend using a rule that no longer exists, and nothing in the UI would tell you.
WHAT TO DO Re-accept by value (a few bulk accepts, per L3), not by resource. Then treat policy edits as a change with a blast radius, and make them deliberately.Treat a policy edit the way you would treat a schema migration: it is a fleet-wide operation, not a settings tweak.
Whose problem is it
The useful diagnostic question is not “how do I stop the tag drifting.” It is “is the derivation wrong, or is the estate telling me something.”
DRIFT SIGNAL USUALLY MEANS─────────────────────────────────────────────────────────────One resource renamed, tag Normal. Re-accept, or acceptfollowed it the new value group.
A whole group's value A team reorganised, or achanged at once naming convention shifted. The tag is reporting a real organisational change.
Values churn every refresh The policy axis is unstable. Usually a name policy on an estate with generated names. Fix the POLICY (L2), not the queue.
A cloud tag was edited away Someone changed the tag in thefrom the derived value console. Worth asking why before assuming the policy is right and the human is wrong.Only the third row is a defect, and the fix is in the policy.
Reconciliation is not drift detection
Worth separating, because the vocabulary collides with a different feature.
SMART TAGS RECONCILIATION Keeps derived state honest against current resource fields. Runs on every refresh. Has no notion of an "expected" tag: whatever the policy derives now IS the expectation.
IaC / TAG-POLICY DRIFT (T3.M3.1.L4) Compares cloud tags against a declared expectation. Reports a violation when they diverge, and someone is expected to act.Reconciliation never reports a violation, because it has nothing to be violated against. If you need “this resource should have tag X and does not”, that is a validate-mode tagging policy: it reports violations (a missing required tag, a value outside the allowed list, a forbidden tag), and the Smart Tags page shows them as a separate non-compliant count.
2. Demo
A month of reconciliation on a real estate, read as signal rather than noise:
MONTH IN REVIEW: team policies (name selector + literal value per team)
WEEK 1 4 accepted tags dropped 4 resources renamed during a service rename (payments-* -> checkout-*) VERDICT: normal. Accepted the new checkout group.
WEEK 2 0 changes.
WEEK 3 1,781 accepted tags dropped in one refresh CAUSE: a platform admin split every team policy by service, trying to separate payments-api from payments-worker RESULT: every value changed shape at once (payments -> payments-api, payments -> payments-worker) Unattributed 9% -> 41% for about an hour VERDICT: correct behaviour, poorly sequenced change. Re-accepted by value: 19 values, 19 bulk accepts. The new split was actually what they wanted; the surprise was the blast radius, not the outcome.
WEEK 4 41 accepted tags dropped, no policy change, no renames CAUSE: an eksctl-managed nodegroup recycles its nodes weekly with generated names VERDICT: the policy axis is wrong for that set. These will churn forever. Left them pending permanently and attributed that nodegroup by resource group instead.
WHAT THE MONTH TAUGHT Weeks 1 and 3 were the system working. Week 4 was the only real finding, and it was about the estate, not the tags.3. Hands-on (6 min)
1. Note today's accepted count: ______ Note it again after the next refresh: ______ A difference with no policy edit and no renames is worth investigating.
2. Pick a resource with an accepted derived tag. What would have to change for that tag to drop? ______________________________________________ (If you cannot answer, re-read the reconciliation rule; it is the whole lesson.)
3. Before your next policy edit, write down: key being edited: ________________ accepted tags under that key: ________________ That number is your largest possible blast radius. On the next refresh, only the tags whose derived value changes (or that stop being derived) drop.
4. Classify one recent drop: [ ] resource changed [ ] policy changed [ ] a tag the policy reads changed Which of the four DRIFT SIGNAL rows does it match? ______________________________________________ Is the fix in the queue, or in the policy? ______________________________________________
5. Name one tag key in your estate that should NOT be a Smart Tag, because it needs to persist independently of any derivation: __________________________4. Knowledge check
Q1
An admin edits a tagging policy’s derivation rule. On the next refresh, several thousand accepted tags disappear and Unattributed jumps. This is:
A. A bug; accepted tags should be immutable
B. A reconciliation job that failed halfway
C. Correct behaviour
D. A permissions failure
Show answer
Correct: C. An accepted tag is kept only while it is still derived with the same value, so a rule change drops every accepted tag whose derived value changed. Keeping the old values would mean reports attributing spend by a rule that no longer exists, with nothing in the UI to say so. The practical lesson is to treat a policy edit as a fleet-wide change with a blast radius of up to the accepted count under that key, and to re-accept by value rather than by resource.
Q2
A Smart Tag key churns on almost every refresh: values change, accepted tags drop, new pending values appear, with no policy edits. The right fix:
A. Re-accept the new values each week
B. Change the policy
C. Disable reconciliation for that key
D. Apply the tags to the cloud so they persist
Show answer
Correct: B. Constant churn means the derivation axis is unstable for that resource set, typically a name-derived policy over cloud-generated or recycled names. Working the queue harder treats the symptom; the axis is the defect. D is a tempting trap: applying to the cloud does make the cloud tag persist, but the derived value keeps churning underneath and the two now disagree. C is not available, and would not be desirable if it were.
Q3
How does Smart Tags reconciliation differ from the tag-drift detection covered in T3.M3.1?
A. Reconciliation keeps derived state honest against the resource’s current fields and has no notion of an expected tag: whatever the policy derives now is the expectation, so it never reports a violation
B. They are the same feature under two names
C. Reconciliation runs weekly while drift detection runs daily, so a tag that changes in between the two is only ever noticed on the slower of the two cycles on each pass through it
D. Reconciliation only applies to AWS
Show answer
Correct: A. Drift detection compares cloud tags against a declared expectation and does report violations for someone to act on. The distinction matters when someone asks reconciliation to tell them which resources are missing a required tag. It structurally cannot: it has nothing to be violated against. That question belongs to a validate-mode tagging policy, whose violations the Smart Tags page shows separately.
5. Apply
Before your next policy edit, check the accepted count for that key on Smart Tags. That number is the most that can drop on the next refresh: only the tags whose derived value changes, or that stop being derived, will drop. Knowing it in advance turns a surprise into a planned change.
Set a monthly habit of reading the drop events as signal: renames and reorganisations are the estate telling you something, and only constant churn is a defect in the policy.
Related lessons
- L3: Accept, revoke, apply to cloud
- L1: What Smart Tags derive
- T3.M3.1.L4: Drift detection and remediation
- T3.M3.1.L5: Reorg-proof tagging
Glossary terms touched
Reconciliation · Tag drift · Derived tag · Blast radius
Complete M2.8 → 10-question module quiz.