Model selection trade-offs

MODEL COST + CAPABILITY MATRIX:

MODEL            COST/M IN+OUT     CAPABILITY              FIT
─────────────────────────────────────────────────────────────────────
Claude Opus       $15 + $75         Best reasoning          Complex analysis
Claude Sonnet     $3 + $15          Strong general          Most production
Claude Haiku      $0.80 + $4        Fast, capable           Simple Q&A, routing
Titan Express     $0.30 + $0.40     Basic                   Simple text gen
Llama 3.1 8B      $0.30 + $0.60     Basic                   Cost-extreme
Mistral models    varies            Varies                  Various

QUESTION TO ASK                          MODEL HINT
─────────────────────────────────────────────────────────
"Is precision critical?"                 Opus / Sonnet
"Is reasoning multi-step?"               Opus / Sonnet
"Is workload high-volume + simple?"      Haiku / Titan
"Does cost matter most?"                 Haiku / Titan
"Can response be approximate?"           Haiku / smaller model
"Need consistent style/voice?"           Higher-tier (Sonnet+)
"Latency-critical (real-time)?"          Haiku (faster)
"Long context needed?"                   Sonnet+ (better long-context)

INCOMING QUERY:
  Step 1: Classify with Haiku (~$0.001 per classification)
  Step 2: Route based on complexity:
    Haiku for simple Q&A (60% of volume)
    Sonnet for medium complexity (35%)
    Opus for complex reasoning (5%)

VS ALL-OPUS:
  Every query goes to Opus regardless of complexity
  Cost: ~5x average

VS ALL-SONNET:
  Acceptable for many workloads
  But overspends on the simple 60%
  Cost: ~2x average

ROUTING WINS because most production workloads have
natural complexity distribution

QUERY COMPLEXITY DISTRIBUTION (typical):
  60-70%: simple (status, lookup, classification)
  20-30%: medium (synthesis, summarization)
  5-10%:  complex (multi-step reasoning, analysis)

USING ALL OPUS:
  60-70% of queries overpay ~18.75x
  Wasted capability on simple queries

USING ROUTING:
  Each query pays only what's needed
  Total cost: 10-25% of all-Opus
  Quality: usually maintained (simple queries don't need premium)

SCENARIO                                     STAY ON OPUS / SONNET
─────────────────────────────────────────────────────────────────
Complex multi-step reasoning
  (research, analysis, architecture)

Critical decision-making
  (medical, legal, financial guidance)

Outputs feed into other automated systems
  (chained AI workflows)

Output quality is the differentiator
  (creative writing, brand voice)

Cost is not the bottleneck
  (low-volume but high-value)

Long context needed
  (smaller models may not handle long input well)

ROUTING ARCHITECTURE:

  Incoming query → Classifier → Route to correct model

CLASSIFIER OPTIONS:
  Use Haiku for classification (cheap; accurate enough)
  Use a fine-tuned smaller model for classification
  Use rule-based heuristics (token count, keywords)

CLASSIFICATION SIGNAL EXAMPLES:
  Token count: short = simple
  Keywords: "explain in detail" → complex
  History: similar queries handled successfully by N model
  Confidence: classifier returns confidence; route accordingly

ROUTING DECISIONS:
  Always route by complexity (most common)
  Sometimes route by user tier (paid → higher model)
  Sometimes route by time (real-time → faster model)

WHEN ROUTING TO CHEAPER MODELS:

  RISK: quality regression
  MITIGATION: monitor metrics

METRICS TO TRACK:
  Customer satisfaction (CSAT) per model
  Re-escalation rate (when did simple → complex?)
  Error rate
  Latency p95
  User feedback ratings

PROTOCOL:
  Start with high-confidence routes
  Monitor 1-2 weeks
  Expand if quality holds
  Iterate the classifier if quality drops
  Maintain higher-tier fallback for uncertain queries

PATTERN A: COMPLEXITY-BASED (most common)
  Classify by query complexity
  Route to matching model
  Best for: most production chatbots

PATTERN B: USER-TIER BASED
  Free tier → Haiku
  Paid tier → Sonnet
  Premium tier → Opus
  Best for: tiered SaaS products

PATTERN C: TASK-BASED
  Summarization → Haiku
  Q&A → Sonnet
  Code generation → Opus
  Best for: multi-purpose apps

PATTERN D: CONFIDENCE-BASED
  Classifier returns confidence score
  Low confidence → escalate to higher model
  Best for: maintaining quality bar

PATTERN E: HYBRID
  Multiple patterns combined
  E.g., user-tier + complexity
  Best for: mature ML products

WORKLOAD: customer support chatbot
  Currently: Claude Opus for every interaction
  Volume: 30K interactions/day
  Avg input: 1,200 tokens
  Avg output: 500 tokens

CURRENT COST:
  Per interaction:
    Input: 1,200 / 1M × $15 = $0.018
    Output: 500 / 1M × $75 = $0.0375
    Total: $0.0555 per interaction
  Monthly: 30K × 30 × $0.0555 = $50,000

REVIEW:
  - 70% are simple "where is my order" / "what's my balance"
    Could route to Haiku
  - 25% are complex billing issues
    Sonnet is sufficient
  - 5% are escalations / complex
    Opus appropriate

REVISED ROUTING:
  Haiku 70% (simple queries):
    Per interaction: $0.00096 + $0.002 = $0.003

  Sonnet 25% (medium):
    Per interaction: $0.0036 + $0.0075 = $0.011

  Opus 5% (complex):
    Per interaction: $0.0555 (unchanged)

WEIGHTED AVERAGE per interaction:
  (0.70 × $0.003) + (0.25 × $0.011) + (0.05 × $0.0555)
  = $0.0021 + $0.00275 + $0.002775
  = $0.0078 per interaction

MONTHLY COST: 30K × 30 × $0.0078 = $7,000

SAVINGS:
  $50,000 - $7,000 = $43,000/month (86% reduction)
  Annual: $516,000

IMPLEMENTATION (4 weeks):
  Week 1: build query classifier
  Week 2: shadow-mode testing (classify but still send to Opus)
  Week 3: route 25% to Haiku; monitor quality
  Week 4: full rollout

OUTCOMES (post-implementation):
  Cost: $7,200/mo (close to projection)
  Quality: CSAT unchanged
  Latency: improved (Haiku is faster)
  Customer satisfaction: high

□ STEP 1: Pick a Bedrock workload
  Workload: __________
  Current model: __________
  Volume per day: __________

□ STEP 2: Estimate complexity distribution
  Simple queries: ___%
  Medium queries: ___%
  Complex queries: ___%

□ STEP 3: Calculate current cost
  Per-query cost on current model: $_____
  Monthly: $_____

□ STEP 4: Calculate routed cost
  Weighted average per-query: $_____
  Monthly: $_____

□ STEP 5: Decide
  Savings: $_____/mo
  Effort: ___ weeks
  ROI: ___x