How the recommender thinks: Evaluate, MetricsAware, PricingAware

Outcome

By the end of this lesson, you will be able to explain how rules consume data and produce recommendations, identify which interfaces a rule implements based on its evidence, and reason about why the architecture is designed this way.


Prerequisites	M2.1.L1 · M2.1.L2
Time	10 minutes

1. Concept

Every rule has to answer a fixed set of questions, and which questions it answers decides what it can do.

In code those question sets are called interfaces. Knowing them explains three things that otherwise look arbitrary: how a recommendation gets made at all, why some rules refuse to run without particular data, and why the evidence shown for one recommendation looks nothing like the evidence for another.

INTERFACE             PURPOSE                        EXAMPLE RULES
─────────────────────────────────────────────────────────────────────
Rule (base)           Core logic; always required    All 684 rules
MetricsAware          Needs metrics (CPU, etc.)      RC-003, RC-169, RC-093
PricingAware          Needs per-resource pricing      Most rules with $ savings

The interfaces are composable: a rule can be just Rule (state-based), or Rule + MetricsAware, or all three.

A note on current code: MetricsAware and PricingAware are now marked Deprecated and slated for removal. They are legacy injection shims; rules that have migrated implement EvalContextRule instead and read metrics, pricing and history from a single evaluation context (evalCtx.Metrics, evalCtx.Pricing) passed to Evaluate, which also takes a *gofr.Context. The questions a rule answers (does it need metrics? does it price a saving?) are the same either way, so this lesson keeps the older names.

The Rule interface: every rule implements this

type Rule interface {
    ID() string                  // RC-001
    Name() string                // "Idle EC2 Instance"
    Provider() string            // "aws"
    ResourceType() string        // "*" (any) or specific type
    Severity() string            // "critical"|"high"|"medium"|"low"|"info"
    Category() string             // one of the 11 categories (M2.1.L1)
    Evaluate(ctx, resource, history) *Recommendation
}

Evaluate is where the rule’s logic lives. Given a resource and its state history, it returns:

*Recommendation: the rule fired (finding generated)
nil: no finding (resource is healthy)

The contract is simple; the logic per rule varies enormously.

The MetricsAware interface: rules needing metrics

Rules that need cloud monitoring data implement an additional interface:

type MetricsAware interface {
    RequiredMetrics() []model.MetricSpec
    SetMetrics(metrics map[string]model.ResourceMetric)
}

RequiredMetrics: declares what metrics the rule needs
                  Example: "CPUUtilization, 30-day lookback, 1h aggregation"

SetMetrics:      called by the recommender to inject pre-fetched data
                 Recommender batches the metric fetches; rule consumes

Examples of MetricsAware rules:

RC-003 (EC2 rightsizing):       CPUUtilization + memory
RC-093 (EC2 schedule heatmap):   CPUUtilization over a weekly grid
RC-169 (idle RDS):               DatabaseConnections (zero = idle)
K8s rules (RC-17xx, 18xx, 19xx): HPA ScalingLimited, pod CPU, memory

Non-MetricsAware rules fire on resource state alone:

RC-002 (orphan EBS): just checks attachedTo=null
RC-005 (RDS Multi-AZ): just checks multiAz=false

State-based rules are simpler; metrics-based rules need infrastructure to fetch metrics.

The PricingAware interface: rules that compute $ savings

Rules that compute savings using per-resource pricing implement:

type PricingAware interface {
    SetPricing(pricing map[string]float64)
}

The recommender fetches per-resource monthly cost from the Aggregator
and injects it via SetPricing.

Rules then use the injected pricing:
  Idle rules:        savings = what keeps billing (for a stopped EC2,
                     only its attached EBS and Elastic IP)
  Right-size rules:  savings = current SKU rate minus target SKU rate;
                     the rule abstains if either rate is unknown
  Compliance rules:  often $0 savings (governance-only)

PricingAware is what gives the recommendation card its dollar figure.

Data flow: end to end

1. DISCOVERER publishes "resource.discovered" for each new resource
   (plus "resource.changed" and "resource.deleted") to Redis Streams
   (Discoverer is the cron that polls cloud APIs for current state)

2. RECOMMENDER receives the event

3. RECOMMENDER fetches required data:
   - Resources (from Discoverer)
   - Metrics (from the Aggregator, which fills them on its metrics-sync cron)
   - State history (from the Config service)
   - Pricing (per-resource monthly cost from the Aggregator)

4. FOR EACH resource:
   - Registry.ForResource(provider, type) returns applicable rules
     (O(1) lookup; pre-indexed)
   - For MetricsAware: inject metrics
   - For PricingAware: inject per-resource cost
   - rule.Evaluate(ctx, resource, history) returns *Recommendation or nil

5. RESULTS batched:
   - *Recommendation → batched UpsertRecommendations to DB
   - nil → batched MarkOptimisedBatch (marks resource as healthy)

6. FLUSH every 50 resources (batch size for DB efficiency)

The flow is event-driven: discovery events trigger evaluation, and an admin can trigger a refresh. The recommender has no generation cron of its own, and the result is not real-time. This trade-off is intentional.

Thread safety + statelessness

THREAD SAFETY:
  A fresh Registry is created per evaluation
  Avoids races on shared rule state

RULE STATELESSNESS:
  Rules themselves are stateless except for injected per-resource data
  Each Evaluate call is self-contained

INJECTED DATA:
  Per-resource metrics: short-lived (only during this Evaluate call)
  Per-resource pricing: same

RESULT:
  Rules can be evaluated in parallel safely
  No shared mutable state across resources

The architecture is concurrency-friendly by design.

Why this design: three properties

1. FAST READ PATH (the UX win)
   Recommendations are pre-computed; the UI just reads from DB
   No live evaluation on user requests
   User clicks Recommendations → page loads in <500ms

   Without pre-computation:
     Live evaluation per request would be 10-30 seconds
     Unscalable; bad UX

   With pre-computation:
     Discovery runs every 6h; its events trigger evaluation
     UI reads static DB rows
     Scales to large estates (10K+ resources)

2. EXTENSIBLE (the developer win)
   Adding a rule:
     1 Go struct + 1 Evaluate function + 1 register call
     ~20-50 lines of code typical

   No SQL schema changes; no UI changes
   Rule appears in recommendations automatically

3. TESTABLE (the quality win)
   Each rule has tabular test cases:
     (resource state + metrics + history) → expected recommendation

   Add edge case to test table; rule is verified
   Refactor without fear; tests catch regressions

The architecture is the long-game investment: faster UX, faster development, fewer bugs.

Performance characteristics

TYPICAL TIMING (mid-size estate):

  Discovery cron: every 6h
    Reads 10K resources from cloud APIs (parallel)
    Publishes events to Redis
    Duration: 5-10 min

  Recommender: triggered by discovery events
    Processes resources in parallel
    Per-resource: ~50ms (Evaluate + DB upsert)
    Total: 10-15 min for 10K resources

  Read path (UI):
    Recommendations list query: <500ms
    Per-resource detail: <100ms

ESTATE SIZE SCALING:
  1K resources:    few minutes per cycle
  10K resources:    10-15 min per cycle
  100K resources:   30-60 min per cycle

The discovery cadence (every 6h) is the throttle; recommender adapts.

The performance is good enough for batch; not designed for real-time.

Adding a new rule: process

ADDING A RULE (engineering team's perspective):

1. Define the rule:
   - Category, severity, action default
   - Required metrics (if any)
   - Pricing usage (if any)
   - Evaluate logic

2. Implement the Go struct:
   - Implement the Rule methods (ID, Name, Provider,
     ResourceType, Severity, Category, Evaluate)
   - Read metrics and pricing from the evaluation
     context (if needed)

3. Write tabular tests:
   - Test resource configurations
   - Expected recommendations or nils

4. Register the rule:
   - Add to registry initialization
   - Specify provider + resource type

5. Document in user-facing docs

TYPICAL EFFORT: 1-3 days per rule (simple)
                1-2 weeks per rule (complex; novel metrics)

The simplicity of adding rules is why the library has grown to 684.

2. Demo

A typical Evaluate call for RC-001 (Idle EC2):

// Inside the IdleEC2 struct's Evaluate function
func (r *IdleEC2) Evaluate(ctx context.Context, resource *Resource, history []StateChange) *Recommendation {
    // STATE check: is the instance stopped?
    if resource.Status != "stopped" {
        return nil  // running = not idle by this rule
    }

    // TIME check: how long has it been stopped?
    daysStopped := daysSince(latestStop(history))
    if daysStopped < r.idleThresholdDays {  // 30 by default
        return nil  // recently stopped; not yet idle
    }

    // PRICING: a stopped box bills no compute, so only the
    // attachments that keep billing count (EBS volumes + Elastic IP)
    residual := attachedEBSCost(resource) + attachedEIPCost(resource)
    if residual <= 0 {
        return nil  // nothing still billing; abstain
    }

    // BUILD recommendation
    return &Recommendation{
        RuleID:            "RC-001",
        ResourceUID:        resource.UID,
        Severity:           "medium",
        Category:           "idle",
        CurrentMonthlyCost: residual,
        OptimizedMonthlyCost: 0,  // delete = residual goes to 0
        PotentialSavings:   residual,  // the residual, not the compute rate
        Evidence: map[string]any{
            "daysStopped": daysStopped,
            "status":       "stopped",
            "lastStop":     latestStop(history),
        },
        Lever: "delete",
    }
}

Notice:

No metrics needed (state-based rule; not MetricsAware)
Pricing injected via PricingAware
Returns nil if conditions aren’t met
Evidence map is what shows in the UI

This is the simplest rule type. More complex rules add metrics + multi-condition logic.

Compare to a MetricsAware rule (RC-003 EC2 right-size)

func (r *EC2RightSize) Evaluate(ctx, resource, history) *Recommendation {
    if resource.Status != "running" {
        return nil  // only evaluate running instances
    }

    // METRICS (injected via MetricsAware):
    cpu := r.metrics["CPUUtilization"][resource.UID]
    if cpu.Avg30d >= 5.0 {  // 30-day average CPU must be under 5%
        return nil  // using too much; can't downsize
    }
    if cpu.Max30d >= 80.0 {  // peak guard: a spike near saturation
        return nil
    }
    if memoryTooHigh(resource) {  // memory veto (60% for a 1-tier step)
        return nil
    }

    // FIND smaller instance type
    currentType := resource.InstanceType  // m5.2xlarge
    targetType := smallerInstance(currentType)  // m5.xlarge

    // PRICING (injected via PricingAware):
    currentMonthly := r.pricing[resource.UID]
    targetMonthly := r.pricingByType[targetType]
    savings := currentMonthly - targetMonthly

    return &Recommendation{
        RuleID:            "RC-003",
        ResourceUID:        resource.UID,
        Severity:           "medium",
        Category:           "rightsizing",
        CurrentMonthlyCost: currentMonthly,
        OptimizedMonthlyCost: targetMonthly,
        PotentialSavings:   savings,
        Evidence: map[string]any{
            "cpuAvg30d":   cpu.Avg30d,
            "lookback":     "30 days",
            "currentType":  currentType,
            "targetType":  targetType,
        },
        ActionType: "right-size",
    }
}

Same shape; more inputs; richer evidence.

3. Hands-on (6 min)

Inspect a recommendation in the UI:

□ STEP 1: Open Recommendations
  Pick one with $ savings
  Click into detail view

□ STEP 2: Inspect the evidence panel
  Was metrics data shown?
    CPU / memory / connections → MetricsAware rule
  Was pricing involved?
    Specific $ figure → PricingAware rule
  What's the lookback window mentioned?
    "30 days" or similar → from metrics

□ STEP 3: Classify the rule
  □ Just Rule (state-based; no metrics)
  □ Rule + MetricsAware (uses metrics)
  □ Rule + PricingAware (computes $ from injected pricing)
  □ All three (MetricsAware + PricingAware)

□ STEP 4: Find a non-PricingAware rule
  Look for a recommendation with $0 savings
  Examples: K8s reliability rules, governance rules
  These are quality categories (no direct savings)

The hands-on shows the interface boundary in action.

4. Knowledge check

Q1

A rule needs CPU utilization to fire, and reports no dollar figure. It implements:

A. Rule + MetricsAware
B. Just Rule, with no capability interfaces at all
C. Rule + PricingAware, since CPU is priced per core
D. Rule + MetricsAware + PricingAware, always both

Show answer

Correct: A. MetricsAware is what supplies the CPU data. PricingAware is a separate interface and is only implemented where the rule computes a dollar saving, which this one does not: a rule that reports no figure has no reason to carry it. Most cost-recovery rules do implement both, which is why D is the tempting answer, but the interfaces are composable rather than a fixed pair.

Q2

The recommender stores pre-computed results, evaluated when discovery events arrive, rather than evaluating on every read. The main reason:

A. Cost, since it uses fewer compute cycles
B. Read path is instant (no live evaluation)
C. Required by AWS
D. Backward compatibility

Show answer

Correct: B. Pre-computed results mean instant reads and scale to large estates. The trade-off is freshness: results follow the 6-hourly discovery, so a just-changed resource can lag until the next pass.

Q3

What does Registry.ForResource(provider, type) return?

A. The list of resources
B. A SQL query
C. The list of rules indexed for that provider+type combination: O(1) lookup
D. The cloud provider’s own rule catalog for that particular resource type

Show answer

Correct: C. The registry is the index of rules by provider+type for fast lookup during evaluation. Pre-indexed at startup; O(1) per query.

5. Apply

Rule definitions live in the recommender’s rule registry. Customers don’t modify rule code; they act on its output by applying findings or dismissing them, for a set period or forever.

For your team: when a recommendation surprises you, the evidence panel + interface knowledge explains how it fired.

Outcome

1. Concept

The Rule interface: every rule implements this

The MetricsAware interface: rules needing metrics

The PricingAware interface: rules that compute $ savings

Data flow: end to end

Thread safety + statelessness

Why this design: three properties

Performance characteristics

Adding a new rule: process

2. Demo

Compare to a MetricsAware rule (RC-003 EC2 right-size)

3. Hands-on (6 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Start with the bill.