Running four specialist agents in parallel, a synthesizer, and an isolated verifier means a single incident investigation executes seven or more LLM calls. If you blindly send every step to a frontier reasoning model like claude-3-7-sonnet or gpt-4o, token costs climb fast—even though most of those tokens are spent on simple log filtering and structured extraction.
In SentinelSRE, we built a dedicated Cost-Aware Model Router (backend/app/agents/router.py) paired with per-step telemetry in Postgres so every node uses only the model tier its task actually requires.
Task-to-Tier Routing: Fast Extraction vs. Deep Reasoning
Instead of hardcoding one model across the entire graph, every node requests either tier="fast" or tier="reasoning":
- Fast Tier (
claude-3-5-haiku/gpt-4o-mini): Handles the Triage Router and all four parallel specialists (log_agent,git_agent,runbook_agent,security_agent). These tasks have narrow, well-defined schemas: extract a file path and line number, filter noisy stack frames, inspect a commit diff, or match a path againstprotected_paths. - Reasoning Tier (
claude-3-7-sonnet/gpt-4o): Reserved strictly for the Incident Commander Synthesizer (reconciling four independent evidence streams into an exact code patch) and the L4 Fresh-Context Verifier (adversarial correctness auditing).
Auditing Real Savings in Postgres (llm_traces)
Claiming "we save money with routing" means nothing without numbers. Every call through invoke_routed_llm() measures wall-clock latency with time.perf_counter(), reads exact input_tokens and output_tokens from usage_metadata, and writes two cost figures into the llm_traces table:
cost_usd: The actual dollar cost of the routed model used for that node.baseline_frontier_cost_usd: What the exact same token count would have cost if routed to the frontier reasoning tier.
On our benchmark Incident #992 (500 KeyError: 'discount_code' on POST /api/checkout), total routed spend across all 7 nodes is $0.0242 versus $0.0695 unrouted—a ~65% cost reduction while keeping frontier reasoning intact where code correctness is decided.
Production Cost Guardrails
Beyond model routing, two additional safeguards protect production budgets: deterministic Python ast.parse() checks reject malformed diffs at $0 before calling the reasoning verifier, and Upstash Redis rate-limiting caps public demo investigations per user/IP so visitors can explore the live War Room without draining API credits.
Good FinOps in AI engineering isn't about downgrading quality—it's about refusing to pay frontier-reasoning prices for regex-level extraction.