When a production endpoint crashes, a human engineer does not investigate one clue, wait ten minutes, and only then look at the next clue. You open logs, Git blame, architecture runbooks, and security boundaries at the same time. Serializing those lookups inside one giant LLM prompt is both slow and cognitively flawed.
Why 4 Independent Specialists Instead of One Monolithic Agent
A single agent looping through 15 tool calls suffers from two failure modes:
- Wall-clock latency adds up linearly: Waiting for log parsing, then GitHub API calls, then vector search, then rule checks turns a 3-second investigation into a 15-second wait.
- Confirmation bias ("anchoring"): If a single agent misreads a noisy log line on step 1, that wrong hypothesis pollutes its GitHub search queries on step 2 and its runbook lookup on step 3.
In SentinelSRE, the Triage & Router Node extracts only the core error signature (for example, KeyError: 'discount_code' at api/checkout.py:84), loads the workspace's Procedural Rules, and fans out four independent specialists concurrently via LangGraph's Send() API. None of the four agents can see what the others are thinking.
Isolated 142 HTTP 500 crashes on POST /api/checkout; payload sends 'promo_code' while line 84 reads 'discount_code'.
Commit #9f3a1c2 (22m ago) renamed discount_code → promo_code in models/order.py but missed api/checkout.py.
Checkout Runbook PDF §4.2 (0.91 cosine match): keep 30-day fallback for legacy 'discount_code' keys.
Flagged api/checkout.py against protected path rule (/payments & checkout routes require HITL approval).
agent_started → agent_completed → hitl_pausedWhat Each Specialist Actually Does
- 2A. Log & StackTrace Specialist: Reads the pasted stack trace or pulls the raw
.logdump from Cloudflare R2, strips thousands of lines of healthcheck noise, and isolates the exact failing request payload (e.g.{"cart_id": "c_8821", "promo_code": "SAVE20"}). - 2B. Git & Codebase Specialist (GitHub MCP): Connects via our Python
mcpSDK client to the GitHub MCP Server, callinglist_commitsandget_file_contents. In our checkout scenario, it discovers Commit#9f3a1c2renameddiscount_codetopromo_codeinmodels/order.py22 minutes ago without updatingapi/checkout.py:84. - 2C. Runbook & History Specialist (Semantic + Episodic Memory): Queries
semantic_memory_chunksin Neonpgvectorand finds Page 4, Section 4.2 of the uploaded Checkout Runbook PDF: payload readers must accept bothpromo_codeand legacydiscount_codefor 30 days. It also queries past resolved/rejected rows inincidents. - 2D. Security & Blast-Radius Specialist: Cross-checks
api/checkout.pyagainstprocedural_rules.protected_paths(auth/,payments/,migrations/,checkout.py) and flags mandatory human review.
Why 3-Tier Memory Lives in Postgres
We intentionally modeled three distinct memory types in Postgres rather than throwing everything into one vector bucket:
- Semantic Memory (
semantic_memory_chunks): Static domain knowledge—PDF runbooks and post-mortems chunked into 600-token blocks with page metadata. - Episodic Memory (
incidents): Time-ordered history of past outages, including whether a human engineer approved or rejected the AI's previous patch and why. - Procedural Memory (
procedural_rules): Deterministic team guardrails—confidence thresholds and protected file paths—that cannot be overridden by prompt persuasion.
Streaming Live Progress Over Authenticated SSE
An investigation dashboard shouldn't feel like a black-box spinner. FastAPI exposes GET /api/incidents/{id}/stream using StreamingResponse(..., media_type="text/event-stream") over LangGraph's astream_events. On the Next.js frontend, we use @microsoft/fetch-event-source instead of the browser's native EventSource specifically because native EventSource cannot send Authorization: Bearer <clerk_jwt> headers.
As each specialist finishes, the Left Panel of the Split-Screen War Room updates in real time with the exact model badge, millisecond latency, and USD cost—before the Incident Commander Synthesizer reconciles all four structured JSON reports into a unified root-cause narrative and patch.
Parallelism compresses investigation time to the slowest single branch (~1.8s), while structured isolation gives the Synthesizer four uncorrelated sources of truth.