SentinelSREAutonomous Incident Engineering
All engineering posts
04 · FinOps & Telemetry5 min read·SentinelSRE Architecture Notes

Cost-Aware Model Routing: Cutting Investigation Spend by 65% Without Losing Quality

Parallel agents multiply token volume. Here is how our Cost-Aware Router and Postgres llm_traces table keep every incident investigation under $0.03.

Running four specialist agents in parallel, a synthesizer, and an isolated verifier means a single incident investigation executes seven or more LLM calls. If you blindly send every step to a frontier reasoning model like claude-3-7-sonnet or gpt-4o, token costs climb fast—even though most of those tokens are spent on simple log filtering and structured extraction.

In SentinelSRE, we built a dedicated Cost-Aware Model Router (backend/app/agents/router.py) paired with per-step telemetry in Postgres so every node uses only the model tier its task actually requires.

Per-Incident Cost Comparison (Incident #992)
Actual Routed Spend: $0.0242 (65% cheaper than unrouted)
Triage & Routerclaude-3-5-haiku
$0.0008 · 680ms
Regex/JSON extraction of file, line & exception
Log Investigatorclaude-3-5-haiku
$0.0024 · 1420ms
Filter noisy stack frames & isolate JSON payload
Git Historian (MCP)claude-3-5-haiku
$0.0025 · 1790ms
Inspect recent commits & fetch target file
Runbook & Memoryclaude-3-5-haiku
$0.0020 · 1310ms
Summarize top-k pgvector chunks & SQL history
Security & Rulesclaude-3-5-haiku
$0.0014 · 1120ms
Match target path against protected_paths
Synthesizerclaude-3-7-sonnet
$0.0098 · 4120ms
Cross-source RCA reasoning & exact code patch
L4 Verifierclaude-3-7-sonnet
$0.0053 · 3210ms
Adversarial fresh-context audit before HITL
Interactive FinOps Comparison — Toggle between Cost-Aware Routed and 100% Frontier Baseline

Task-to-Tier Routing: Fast Extraction vs. Deep Reasoning

Instead of hardcoding one model across the entire graph, every node requests either tier="fast" or tier="reasoning":

  • Fast Tier (claude-3-5-haiku / gpt-4o-mini): Handles the Triage Router and all four parallel specialists (log_agent, git_agent, runbook_agent, security_agent). These tasks have narrow, well-defined schemas: extract a file path and line number, filter noisy stack frames, inspect a commit diff, or match a path against protected_paths.
  • Reasoning Tier (claude-3-7-sonnet / gpt-4o): Reserved strictly for the Incident Commander Synthesizer (reconciling four independent evidence streams into an exact code patch) and the L4 Fresh-Context Verifier (adversarial correctness auditing).

Auditing Real Savings in Postgres (llm_traces)

Claiming "we save money with routing" means nothing without numbers. Every call through invoke_routed_llm() measures wall-clock latency with time.perf_counter(), reads exact input_tokens and output_tokens from usage_metadata, and writes two cost figures into the llm_traces table:

  • cost_usd: The actual dollar cost of the routed model used for that node.
  • baseline_frontier_cost_usd: What the exact same token count would have cost if routed to the frontier reasoning tier.

On our benchmark Incident #992 (500 KeyError: 'discount_code' on POST /api/checkout), total routed spend across all 7 nodes is $0.0242 versus $0.0695 unrouted—a ~65% cost reduction while keeping frontier reasoning intact where code correctness is decided.

Production Cost Guardrails

Beyond model routing, two additional safeguards protect production budgets: deterministic Python ast.parse() checks reject malformed diffs at $0 before calling the reasoning verifier, and Upstash Redis rate-limiting caps public demo investigations per user/IP so visitors can explore the live War Room without draining API credits.

Good FinOps in AI engineering isn't about downgrading quality—it's about refusing to pay frontier-reasoning prices for regex-level extraction.