SentinelSREAutonomous Incident Engineering
All engineering posts
02 · Multi-Agent & Memory6 min read·SentinelSRE Architecture Notes

The Parallel War Room: 4 Specialists, 3-Tier Memory, and Live SSE

Inside Phase B: how LangGraph Send() eliminates confirmation bias, how Semantic, Episodic, and Procedural memory work together, and how the Split-Screen War Room streams live.

When a production endpoint crashes, a human engineer does not investigate one clue, wait ten minutes, and only then look at the next clue. You open logs, Git blame, architecture runbooks, and security boundaries at the same time. Serializing those lookups inside one giant LLM prompt is both slow and cognitively flawed.

Why 4 Independent Specialists Instead of One Monolithic Agent

A single agent looping through 15 tool calls suffers from two failure modes:

  • Wall-clock latency adds up linearly: Waiting for log parsing, then GitHub API calls, then vector search, then rule checks turns a 3-second investigation into a 15-second wait.
  • Confirmation bias ("anchoring"): If a single agent misreads a noisy log line on step 1, that wrong hypothesis pollutes its GitHub search queries on step 2 and its runbook lookup on step 3.

In SentinelSRE, the Triage & Router Node extracts only the core error signature (for example, KeyError: 'discount_code' at api/checkout.py:84), loads the workspace's Procedural Rules, and fans out four independent specialists concurrently via LangGraph's Send() API. None of the four agents can see what the others are thinking.

Parallel LangGraph Fan-Out vs Serial Execution
Wall-clock time: 1.79s parallel (slowest branch) vs 5.64s serial
Log Investigator1420ms · $0.0024
Source: R2 .log / Stack Trace

Isolated 142 HTTP 500 crashes on POST /api/checkout; payload sends 'promo_code' while line 84 reads 'discount_code'.

Git Historian1790ms · $0.0025
Source: GitHub MCP (commits + file)

Commit #9f3a1c2 (22m ago) renamed discount_code → promo_code in models/order.py but missed api/checkout.py.

Runbook & Episodic Agent1310ms · $0.0020
Source: pgvector + incidents SQL

Checkout Runbook PDF §4.2 (0.91 cosine match): keep 30-day fallback for legacy 'discount_code' keys.

Security & Rules Agent1120ms · $0.0014
Source: procedural_rules JSONB

Flagged api/checkout.py against protected path rule (/payments & checkout routes require HITL approval).

SSE Stream (`GET /api/incidents/{id}/stream`)Events: agent_started → agent_completed → hitl_paused
Interactive Parallel War Room — Click Simulate Parallel SSE to watch concurrent execution

What Each Specialist Actually Does

  • 2A. Log & StackTrace Specialist: Reads the pasted stack trace or pulls the raw .log dump from Cloudflare R2, strips thousands of lines of healthcheck noise, and isolates the exact failing request payload (e.g. {"cart_id": "c_8821", "promo_code": "SAVE20"}).
  • 2B. Git & Codebase Specialist (GitHub MCP): Connects via our Python mcp SDK client to the GitHub MCP Server, calling list_commits and get_file_contents. In our checkout scenario, it discovers Commit #9f3a1c2 renamed discount_code to promo_code in models/order.py 22 minutes ago without updating api/checkout.py:84.
  • 2C. Runbook & History Specialist (Semantic + Episodic Memory): Queries semantic_memory_chunks in Neon pgvector and finds Page 4, Section 4.2 of the uploaded Checkout Runbook PDF: payload readers must accept both promo_code and legacy discount_code for 30 days. It also queries past resolved/rejected rows in incidents.
  • 2D. Security & Blast-Radius Specialist: Cross-checks api/checkout.py against procedural_rules.protected_paths (auth/, payments/, migrations/, checkout.py) and flags mandatory human review.

Why 3-Tier Memory Lives in Postgres

We intentionally modeled three distinct memory types in Postgres rather than throwing everything into one vector bucket:

  • Semantic Memory (semantic_memory_chunks): Static domain knowledge—PDF runbooks and post-mortems chunked into 600-token blocks with page metadata.
  • Episodic Memory (incidents): Time-ordered history of past outages, including whether a human engineer approved or rejected the AI's previous patch and why.
  • Procedural Memory (procedural_rules): Deterministic team guardrails—confidence thresholds and protected file paths—that cannot be overridden by prompt persuasion.

Streaming Live Progress Over Authenticated SSE

An investigation dashboard shouldn't feel like a black-box spinner. FastAPI exposes GET /api/incidents/{id}/stream using StreamingResponse(..., media_type="text/event-stream") over LangGraph's astream_events. On the Next.js frontend, we use @microsoft/fetch-event-source instead of the browser's native EventSource specifically because native EventSource cannot send Authorization: Bearer <clerk_jwt> headers.

As each specialist finishes, the Left Panel of the Split-Screen War Room updates in real time with the exact model badge, millisecond latency, and USD cost—before the Incident Commander Synthesizer reconciles all four structured JSON reports into a unified root-cause narrative and patch.

Parallelism compresses investigation time to the slowest single branch (~1.8s), while structured isolation gives the Synthesizer four uncorrelated sources of truth.