IaC drift detection

SCENARIO:
  Terraform says: instance_type = "m5.large"
  Cloud console (engineer changed it): m5.2xlarge
  Result: drift; IaC no longer matches reality

ON NEXT TERRAFORM APPLY:
  Either:
    Terraform changes back to m5.large (engineer's change lost)
    OR
    Engineer remembers to update IaC first

EITHER way: friction. The drift is the symptom of broken discipline.

SOURCES OF DRIFT:

1. MANUAL CHANGES FOR QUICK FIXES
   Engineer fixes something via console during incident
   Doesn't update IaC afterward
   Next apply: lost fix

2. OTHER TOOLS MODIFYING RESOURCES
   Auto-scaling adds tags
   Backup tool modifies properties
   Other teams' automation

3. CLOUD-SIDE AUTOMATION
   AWS auto-scaling adjustments
   Managed service updates (RDS minor version)

4. ACQUIRED COMPANY INTEGRATIONS
   Pre-existing resources brought under your IaC
   May have drift from start

5. MIGRATED OLD IaC PATTERNS
   Resources created before IaC adoption
   Imported with mismatch

6. CONSOLE EXPLORATION GONE WRONG
   Engineer experimenting; saves changes inadvertently

DRIFT OFTEN CAUSES:
  ✓ Resources at wrong size (oversized after manual upgrade)
  ✓ Wrong instance class (engineer "upgraded" to bigger family)
  ✓ Wrong region (deployment error)
  ✓ Wrong account (multi-account confusion)
  ✓ Tag changes (affecting billing attribution)
  ✓ Storage drift (volume size grown manually)
  ✓ Network drift (added security group rules)

EACH DRIFT EVENT = UNINTENDED STATE = unexpected cost OR security risk

TYPICAL COST IMPACT:
  Small team: 5-10 drift events/week
  Medium team: 30-50/week
  Large org: 100+/week

  ~30% are cost-impacting (sizing changes, region differences)
  ~10% are security-impacting
  Rest: configuration drift (still bad, less direct cost)

TERRAFORM:
  $ terraform plan
  - Compares current state to desired state
  - Shows differences (drift)
  - Doesn't auto-fix; shows what WOULD happen on apply

  Better: scheduled drift detection
  $ terraform plan -detailed-exitcode
  - Exit 2 if drift detected
  - Use in CI/CD daily check

CDK:
  $ cdk diff
  - Compares CFN template to deployed
  - Shows drift in CloudFormation context
  - CloudFormation also has built-in drift detection

  AWS Config:
  cfn:DriftDetectionResource = NOT_IN_SYNC
  - Visible in CFN console

PULUMI:
  $ pulumi refresh
  - Updates state from cloud
  $ pulumi up
  - Shows desired vs actual differences

  Workflow:
    1. pulumi refresh (read current)
    2. pulumi up (apply desired or accept drift)

CLOUD-NATIVE (across all IaC):
  AWS Config Rules - "rule-based" drift detection
  GCP Cloud Asset Inventory + Config Validator
  Azure Policy + Compliance scans

  These catch drift even if IaC tool doesn't notice

PATTERN A: ACCEPT (cloud is truth)
  Cloud-side change was intentional and correct
  Action:
    Update IaC to match new cloud-side state
    Document the reason (PR with explanation)
    Future plans converge

  Use when:
    Engineer's manual change was correct
    Cloud-side automation made legit change
    Migration / acquisition with desired state

PATTERN B: REVERT (IaC is truth)
  Cloud-side change was wrong / unauthorized
  Action:
    terraform apply (or equivalent)
    Cloud reverts to IaC state
    Investigate why the change happened
    Educate or harden against repeat

  Use when:
    Unauthorized manual change
    Accidental console change
    Drift caused by misconfigured automation

PATTERN C: INVESTIGATE FIRST
  Don't know yet; need context
  Action:
    Audit log: who/when made the change
    Slack the engineer
    Wait for response
    Then accept or revert based on intent

  Use when:
    Unclear what happened
    Risk of either accept or revert being wrong

HEALTHY IaC discipline:
  Drift events per week per 100 resources: <5
  Resolved within 24-72 hours
  Pattern: mostly intentional (accept; update IaC)

MODERATELY UNHEALTHY:
  Drift events per week per 100 resources: 5-20
  Mix of intentional and accidental
  Investigation cycle: 1-2 weeks

UNHEALTHY:
  Drift events per week per 100 resources: >20
  Tension between manual changes and IaC apply
  Often resolved by re-running terraform (fighting with cloud)
  Engineer trust in IaC eroding

TRENDING:
  Drift increasing weekly = signal of process breakdown
  Drift decreasing weekly = signal of healthy discipline

GITHUB ACTIONS (Terraform daily check):

name: Drift Detection
on:
  schedule:
    - cron: '0 6 * * *'  # 6 AM daily

jobs:
  drift_check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: terraform init
        run: terraform init

      - name: terraform plan (drift detection)
        run: |
          terraform plan -detailed-exitcode -out=tfplan
          # Exit code 2 means changes detected (drift)
          if [ $? -eq 2 ]; then
            echo "DRIFT DETECTED"
            terraform show tfplan > drift-report.txt
          fi

      - name: Notify on drift
        if: success() && env.DRIFT_DETECTED == 'true'
        uses: 8398a7/action-slack@v3
        with:
          status: custom
          custom_payload: |
            {
              "text": "Drift detected in production IaC",
              "attachments": [{
                "text": "${{ env.DRIFT_DETAILS }}"
              }]
            }

INFRACOST INTEGRATION:
  Pre-merge: shows cost impact of IaC change

  Example PR comment:
    "Cost change: +$400/month
     New EC2 instance (m5.2xlarge): +$280/mo
     New EBS volume (1TB gp3): +$80/mo
     New NAT Gateway: +$40/mo"

  CAPTURES COST DECISIONS at PR review time

REVIEWER CAN:
  Question expensive additions
  Suggest right-sizing
  Approve if justified
  Block if over-provisioned

BAD COST DECISIONS caught before merge
GOOD COST DECISIONS documented in PR

IMPORT: bring existing cloud resource under IaC
  $ terraform import aws_instance.web i-1234567890
  Risk: LOW (no state changes; just adds to state file)
  Use: when adopting IaC for existing resources

MOVE: move resource between IaC sources / states
  $ terraform state mv aws_instance.web module.app.aws_instance.web
  Risk: MODERATE (state surgery; verify no unintended changes)
  Use: refactoring IaC structure

DESTROY: delete resource via IaC
  $ terraform destroy -target=aws_instance.web
  Risk: HIGH (may cause cascading data loss)
  Use: deliberate decommissioning
  Always: document, require approval, double-check

PERIODIC AUDIT pattern:
  Show drift events per IaC source/team
  Flag stuck drifts (>7 days unresolved)
  Track drift count weekly over time
  Alert on drift count spikes

HEALTHY DASHBOARD:
  Drift count flat or trending down
  Stuck drifts: 0 (resolved within 1 week)
  Per-team breakdown shows none over threshold

WHEN STUCK:
  Engineer assigned to resolve
  After 7 days: escalate to platform lead
  After 14 days: emergency review

TEAM: 100-engineer mid-size org, ~500 Terraform-managed resources

WEEKLY AUDIT (Monday morning):

T+0      Run scheduled drift detection
         terraform plan -detailed-exitcode (across all stacks)

T+10min  Results aggregated:
         47 modified resources detected across 12 stacks

T+15min  Categorize via investigation:

         INTENTIONAL (12 resources):
           Engineer changes for incident response last week
           Manual hotfixes during demo
           ACTION: PRs to update IaC to match cloud-side

         UNINTENTIONAL (30 resources):
           Cloud-side automation modifications
           Auto-scaling tag additions
           Managed service auto-updates
           ACTION: PRs to update IaC (cloud automation legit)

         UNAUTHORIZED (5 resources):
           Console changes by 2 engineers (unclear intent)
           Will investigate via Slack DM
           ACTION: revert via terraform apply; investigate source

T+1hr    Action items distributed:
         3 PRs to update IaC for intentional + unintentional drift
         2 Slack threads investigating unauthorized
         1 emergency revert (clear unauthorized change)

T+1 week (follow-up audit):
         Previous drifts resolved
         Drift count this week: 12 (down from 47)
         Trend: healthy

         Unauthorized investigation results:
           Engineer 1: experimenting; education provided
           Engineer 2: emergency fix; team updated process

LONG-TERM TREND:
  Q1: avg 40 drifts/week
  Q2: avg 25 drifts/week (after process improvements)
  Q3: avg 12 drifts/week (steady-state)

  Process improvements:
    1. IaC checklist for incident response
    2. Console access reviewed quarterly
    3. Auto-tagger reconciles tag drift automatically
    4. Cloud-native policy blocks some console changes

□ STEP 1: Pick a Terraform stack
  Stack path: __________
  Resource count: _____

□ STEP 2: Run drift detection
  $ terraform plan -detailed-exitcode

  Exit code: _____
  Resources with changes: _____

□ STEP 3: Categorize first 5 drifts
  Resource 1: __________
    Source: □ Intentional □ Unintentional □ Unauthorized
    Action: __________

  Resource 2-5: same template

□ STEP 4: Estimate drift rate
  Drifts per 100 resources per week: _____
  Health: □ Healthy (<5) □ Moderate (5-20) □ Unhealthy (>20)

□ STEP 5: Set up scheduled detection
  □ Daily/weekly GitHub Action
  □ Notification to Slack
  □ Owner: __________