SentinelSREAutonomous Incident Engineering
All engineering posts
03 · Reliability & Safety6 min read·SentinelSRE Architecture Notes

Eliminating Hallucinations: Zero-History L4 Verification & Postgres HITL Gates

Why multi-agent chains get tunnel vision, how we strip agent chatter before auditing a patch, and why LangGraph checkpoints every sensitive fix in Neon Postgres.

The hardest problem in autonomous code remediation is not generating a plausible-looking diff—it is preventing the AI from rationalizing a broken fix. After four specialists and a synthesizer exchange thousands of tokens, the conversation history itself becomes a source of bias.

Why Agents Hallucinate in Multi-Step Pipelines

If you ask the same LLM session that diagnosed a bug to also grade its own patch, it already believes its own narrative. It overlooks an undefined variable, misses a syntax error, or ignores a constraint from the runbook because its context window is saturated with justifications.

Our solution is a two-layer defense: a deterministic code validator paired with an L4 Fresh-Context Verifier, followed by a durable Human-in-the-Loop (HITL) escalation gate.

Interactive Verification & Escalation Simulator
1. Isolated Inputs Only
Raw stack trace snippet
Raw GitHub file (`api/checkout.py`)
Proposed Unified Diff + Rules
STRIPPED: specialist_findings, root_cause_report
2. AST + Fresh-Context Audit
AST Parse CheckPASS
Verifier Confidence94% / 85% min
3. Policy Gate Outcome
interrupt() → PostgresSaver

Even at 94% confidence, api/checkout.py matches protected_paths. Graph freezes until human clicks Approve.

Interactive Verification & Escalation Simulator — Toggle scenarios to see how the gate behaves

Layer 1: Deterministic Checks at $0 LLM Cost

Before invoking any reasoning model to audit a proposed fix, verifier_node runs fast deterministic checks in Python:

  • Non-empty Unified Diff: Computes difflib.unified_diff(original_lines, patched_lines) and verifies the patch actually modifies the target file.
  • Python AST Syntax Validation: For .py files, runs ast.parse(patched_file_content). If a syntax error exists, the verifier immediately returns is_valid=False and confidence_score=0 with the exact SyntaxError line—routing back to the Synthesizer without burning frontier tokens.

Layer 2: The Zero-History L4 Fresh-Context Verifier

Once deterministic checks pass, SentinelSRE spawns an isolated verifier prompt. Crucially, we strip specialist_findings and root_cause_report completely. The verifier sees only four raw artifacts:

  • The raw runtime stack trace from Sentry/Railway/Vercel.
  • The raw target file contents fetched directly from GitHub.
  • The proposed unified Git diff.
  • The workspace's procedural_rules.

Acting like a strict senior reviewer with zero prior bias, it outputs a structured VerifierOutput(is_valid, confidence_score, critique). If is_valid is false and retry_count < 2, LangGraph loops back to the Synthesizer with the verifier's exact critique so it can self-correct before a human ever sees it.

Layer 3: Durable HITL Checkpointing with PostgresSaver & interrupt()

Verification proves a patch is syntactically sound and addresses the stack trace—but verification is not authorization. Whenever confidence_score < min_confidence_threshold (default 85%) or the target file matches protected_paths (such as api/checkout.py, payments/, auth/, or migrations/), the escalation_gate_node calls LangGraph's interrupt().

Because the graph is compiled with AsyncPostgresSaver backed by Neon Postgres, pausing at interrupt() freezes the entire IncidentState snapshot under thread_id = incident.id. Even if the FastAPI container restarts or scales down, the paused investigation survives intact.

In the Split-Screen War Room, the engineer inspects the side-by-side diff, the citations, and the verifier's confidence meter. Clicking Approve & Open Hotfix Draft PR calls POST /api/incidents/{id}/decision, resumes the exact Postgres checkpoint via Command(resume=...), invokes the GitHub MCP and Slack MCP servers to open a draft PR (never mutating main directly), and saves the decision into Episodic Memory.

Software verifies correctness; humans authorize production impact. And if an engineer rejects a patch with feedback, Episodic Memory ensures SentinelSRE doesn't make the same mistake twice.