When to write a watch policy

BUILT-IN RULES                  WATCH POLICIES
─────────────────────────────────────────────────────────────
Go rules under internal/rules   A policy you author in
                                Settings > Policy > Watch

Shipped by ZopNight             Authored by you, stored in
                                the config policy store
                                (domain=watch)

Write to `recommendations`      Write to
                                `custom_recommendation`

Recommendations tab             User-Defined tab

SCOPE      which resources this applies to (a selector)
SIGNALS    one or more: "avg cpu < 5% over 14 days"
DECISION   combine the signals: all-of, or any-of
OUTCOME    idle / schedule / rightsize / custom
           + a severity you declare

YES: the threshold is ORG-SPECIFIC
  "Our batch nodes should never sit under 15% CPU for a
   fortnight, because our jobs are sized to fill them."
  No shipped rule knows your sizing convention.

YES: the resource type matters to you and not generally
  A queue depth that means something in your architecture
  and nothing in anyone else's.

YES: you want a TIGHTER threshold than the shipped rule
  The built-in idle rule fires at its own threshold. You
  want to know earlier.

YES: the combination is what matters
  Low CPU alone is normal here. Low CPU AND near-zero
  network is not.

NO: a built-in rule already covers it
  Check first. A duplicate policy produces two findings on
  the same resource with two savings figures, and the
  cross-rec lever exclusivity that de-duplicates built-in
  rules does not span engines.

NO: you want an ALERT
  A watch policy produces a recommendation, which is a
  durable finding you triage. If you want to be paged when
  something crosses a line right now, that is an alerting
  policy or a budget threshold, not this.

NO: the signal is not a metric
  Policies read metrics. "Resources with no owner tag" is
  a tagging-policy question. "Resources violating a naming
  convention" is not expressible here at all.

NO: you want it to open a ticket
  Notify and Jira dispatch from a watch policy are
  deferred. Today it produces a recommendation and stops.

1. THE WIZARD GREYS OUT what your scope cannot collect.
   It reads the aggregator's metric catalog, so a metric
   no resource in scope emits is not offered. This
   prevents the most common failure: a policy that can
   never fire because nothing produces its signal.

2. THE SERVER VALIDATES before save.
   The spec goes through a validation endpoint, so a
   policy that is structurally incapable of firing is
   rejected at the write boundary rather than saved and
   silently doing nothing.

POLICY A: "batch nodes under-filled"  -> KEPT

  WHY NO BUILT-IN RULE COVERS IT
    The shipped idle rules fire at a low threshold, which
    is right for general estates. This team sizes batch
    nodes to run at 60-80% by design, so anything under
    ~15% sustained means a job stopped feeding them, and
    that is invisible to a rule tuned for idleness.

  SCOPE     resource_group = batch-compute
  SIGNAL    avg cpu lt 15% over 14 days
  DECISION  all-of  (single signal)
  OUTCOME   rightsize, severity medium

  RESULT    9 findings in the first pass. 6 were real:
            a decommissioned upstream feed nobody had
            noticed. ~$3,100/mo.

POLICY B: "unattached volumes"  -> DELETED WITHIN A WEEK

  WHAT WENT WRONG
    RC-002 already does this, and better: it carries the
    Safety Gate's orphan-evidence logic and knows about
    detached_since dwell.

    The watch policy produced a SECOND finding on the same
    volumes with its own savings number. Cross-rec lever
    exclusivity de-duplicates overlapping BUILT-IN rules;
    it does not span the two engines, so both figures were
    reported and the estate's total was overstated.

  THE LESSON
    Search the built-in catalogue before authoring. The
    duplicate did not just add noise, it corrupted the
    savings total.

POLICY C: "page us when spend spikes"  -> NEVER BUILT

  The team wanted an alert. A watch policy produces a
  durable finding you triage, not a page. This belongs in
  anomaly detection subscriptions or a budget threshold.

1. Name one waste pattern in your estate that you are
   confident no shipped rule catches:
   ______________________________________________

2. Search the built-in catalogue for it. Be honest:
   is there a rule that already covers it?      Y / N
   If Y, stop. Tune that rule's severity instead.

3. Is your pattern expressible as a METRIC threshold?
                                                Y / N
   If N (tags, naming, configuration), it is not a watch
   policy. Where does it belong?
   ______________________________________________

4. Do the resources in your intended scope actually emit
   the metric? Open the wizard and check whether it is
   greyed out.                     available: Y / N

5. Decide the outcome type before writing anything:
   [ ] idle   [ ] schedule   [ ] rightsize   [ ] custom
   L3 explains why this choice determines whether your
   finding can carry a dollar figure at all.

BEFORE  A ZopNight account with one cloud connected. Resources with metrics flowing: EC2, Compute Engine or Azure Virtual Machines discovered at least a fortnight ago.
ASK     "Write a policy that flags any instance under 5% CPU for 14 days, and check it before saving."
CHECK   that validation passed first. A policy that never evaluates is worse than none, because it looks like cover.