SentinelSREAutonomous Incident Engineering
All engineering posts
01 · System Design7 min read·SentinelSRE Architecture Notes

Why We Built SentinelSRE: Turning a Stack Trace into a Verified Draft PR

How we connected runtime production errors, Cloudflare R2 object storage, Neon pgvector memory, parallel LangGraph agents, and MCP into a single auditable pipeline.

Imagine an e-commerce team deploys a routine refactor on Friday afternoon. Minutes later, POST /api/checkout starts failing with 500 KeyError: 'discount_code'. In most engineering orgs, a senior engineer has to wake up, pull server logs from Railway or Vercel, inspect recent GitHub commits, search PDF architecture runbooks, check past post-mortems, write a patch, open a PR, and update Slack. That takes ~2 hours of context switching.

SentinelSRE compresses that investigation to roughly 20 seconds: it investigates logs, Git history, runbooks, and safety rules in parallel, audits its own fix in an isolated verifier, pauses for human approval, and opens a draft pull request automatically.

Why Incident Response (and Not Another AI PR Reviewer)

When we designed this project, the easiest trap to fall into was building another generic "AI PR Review Agent." Every tutorial covers that, and half the components in a modern full-stack AI architecture—like Cloud Object Storage and multi-tier operational memory—feel bolted on for a simple diff reviewer.

By applying the same multi-agent systems architecture to autonomous production incident engineering, every layer becomes genuinely necessary:

  • Cloudflare R2 Presigned URLs are required because production .log dumps and architecture PDF runbooks can easily hit tens of megabytes.
  • 3-Tier Postgres + pgvector Memory is required because incident triage depends on architecture docs (Semantic), past outages (Episodic), and deployment safety rules (Procedural).
  • Human-in-the-Loop (HITL) Checkpointing is required because no sane engineering team lets an LLM silently push unreviewed patches to /payments or /auth.
Interactive System Map — Click any stage to inspect implementation detailsPhase A + Phase B
4 Parallel Specialist AgentsLangGraph Send() Fan-Out
4 concurrent branches · Structured Pydantic outputs

Log Investigator, Git Historian (MCP), Runbook & Episodic Memory Agent (pgvector + SQL), and Security Agent run simultaneously without seeing each other's reasoning—cutting wall-clock latency and preventing confirmation bias.

Interactive System Map — Click any stage in SentinelSRE to inspect how it works

The Practical Runtime Reality: GitHub Alone Doesn't Know Production Broke

One of the most important design decisions we made early was acknowledging a simple truth: if you push code and production crashes, GitHub cannot tell you it broke. GitHub knows about your source code; your runtime platform (Railway, Render, Vercel, Docker) or error tracker (Sentry) knows about the exception.

Instead of pretending there is a magical "get all production logs" button, we designed the MVP around an explicit, testable workflow:

  • An engineer copies a sanitized stack trace from Sentry, Railway, or Vercel (stripping secrets and PII), selects the GitHub repo and deployed commit SHA, and optionally attaches a .log file.
  • SentinelSRE compares the runtime error against the exact deployed code version, runs the investigation, and pauses at an approval gate before opening a draft pull request.
  • Automated Sentry webhooks sit cleanly on top of this exact pipeline once the core manual loop is verified end-to-end.

Two Separate Flows: Phase A vs. Phase B

To keep the codebase clean and predictable, SentinelSRE separates setup and ingestion from live incident investigation:

Phase A: Auth, Direct R2 Uploads, and 3-Tier Memory

Authentication runs through Clerk on Next.js 16. Every request to our FastAPI backend carries a short-lived RS256 JWT (Authorization: Bearer ...), which FastAPI validates directly against Clerk's JWKS endpoint to extract clerk_user_id and enforce tenant isolation across all queries.

When uploading a 50MB .log dump or an architecture PDF runbook, streaming the file through Next.js → FastAPI → Object Storage would spike backend RAM and risk OOM crashes. Instead, FastAPI generates a 60-second Cloudflare R2 Presigned PUT URL via boto3 (S3 v4 signature), and the browser uploads bytes directly to R2. Only after the PUT succeeds does FastAPI kick off background chunking (600-token windows via pymupdf) and VECTOR(1536) embedding into Neon Postgres.

One Postgres Database, Six Canonical Tables

Rather than splitting state across separate vector databases, SQL stores, and checkpoint services, we consolidated everything into a single Neon Postgres instance with pgvector:

  • workspaces — Multi-tenant workspace mapping synced with clerk_user_id, repo URLs, and Slack channel IDs.
  • knowledge_files — Tracks uploaded R2 keys and ingestion status (pending → processing → indexed).
  • semantic_memory_chunks — Tier 1 Semantic Memory storing VECTOR(1536) embeddings and page/section JSONB metadata.
  • procedural_rules — Tier 2 Procedural Memory storing min_confidence_threshold (default 85%) and protected_paths.
  • incidents — Tier 3 Episodic Memory (also doubling as the LangGraph thread_id) storing root-cause reports, diffs, human feedback, and PR URLs.
  • llm_traces — FinOps telemetry logging model, tokens, latency, and USD cost per agent node.
Every architectural choice in SentinelSRE—presigned uploads, isolated Postgres tenants, parallel agents, and draft-only PRs—exists to make autonomous debugging safe enough for real production teams.