Imagine an e-commerce team deploys a routine refactor on Friday afternoon. Minutes later, POST /api/checkout starts failing with 500 KeyError: 'discount_code'. In most engineering orgs, a senior engineer has to wake up, pull server logs from Railway or Vercel, inspect recent GitHub commits, search PDF architecture runbooks, check past post-mortems, write a patch, open a PR, and update Slack. That takes ~2 hours of context switching.
SentinelSRE compresses that investigation to roughly 20 seconds: it investigates logs, Git history, runbooks, and safety rules in parallel, audits its own fix in an isolated verifier, pauses for human approval, and opens a draft pull request automatically.
Why Incident Response (and Not Another AI PR Reviewer)
When we designed this project, the easiest trap to fall into was building another generic "AI PR Review Agent." Every tutorial covers that, and half the components in a modern full-stack AI architecture—like Cloud Object Storage and multi-tier operational memory—feel bolted on for a simple diff reviewer.
By applying the same multi-agent systems architecture to autonomous production incident engineering, every layer becomes genuinely necessary:
- Cloudflare R2 Presigned URLs are required because production
.logdumps and architecture PDF runbooks can easily hit tens of megabytes. - 3-Tier Postgres + pgvector Memory is required because incident triage depends on architecture docs (Semantic), past outages (Episodic), and deployment safety rules (Procedural).
- Human-in-the-Loop (HITL) Checkpointing is required because no sane engineering team lets an LLM silently push unreviewed patches to
/paymentsor/auth.
4 concurrent branches · Structured Pydantic outputsLog Investigator, Git Historian (MCP), Runbook & Episodic Memory Agent (pgvector + SQL), and Security Agent run simultaneously without seeing each other's reasoning—cutting wall-clock latency and preventing confirmation bias.
The Practical Runtime Reality: GitHub Alone Doesn't Know Production Broke
One of the most important design decisions we made early was acknowledging a simple truth: if you push code and production crashes, GitHub cannot tell you it broke. GitHub knows about your source code; your runtime platform (Railway, Render, Vercel, Docker) or error tracker (Sentry) knows about the exception.
Instead of pretending there is a magical "get all production logs" button, we designed the MVP around an explicit, testable workflow:
- An engineer copies a sanitized stack trace from Sentry, Railway, or Vercel (stripping secrets and PII), selects the GitHub repo and deployed commit SHA, and optionally attaches a
.logfile. - SentinelSRE compares the runtime error against the exact deployed code version, runs the investigation, and pauses at an approval gate before opening a draft pull request.
- Automated Sentry webhooks sit cleanly on top of this exact pipeline once the core manual loop is verified end-to-end.
Two Separate Flows: Phase A vs. Phase B
To keep the codebase clean and predictable, SentinelSRE separates setup and ingestion from live incident investigation:
Phase A: Auth, Direct R2 Uploads, and 3-Tier Memory
Authentication runs through Clerk on Next.js 16. Every request to our FastAPI backend carries a short-lived RS256 JWT (Authorization: Bearer ...), which FastAPI validates directly against Clerk's JWKS endpoint to extract clerk_user_id and enforce tenant isolation across all queries.
When uploading a 50MB .log dump or an architecture PDF runbook, streaming the file through Next.js → FastAPI → Object Storage would spike backend RAM and risk OOM crashes. Instead, FastAPI generates a 60-second Cloudflare R2 Presigned PUT URL via boto3 (S3 v4 signature), and the browser uploads bytes directly to R2. Only after the PUT succeeds does FastAPI kick off background chunking (600-token windows via pymupdf) and VECTOR(1536) embedding into Neon Postgres.
One Postgres Database, Six Canonical Tables
Rather than splitting state across separate vector databases, SQL stores, and checkpoint services, we consolidated everything into a single Neon Postgres instance with pgvector:
workspaces— Multi-tenant workspace mapping synced withclerk_user_id, repo URLs, and Slack channel IDs.knowledge_files— Tracks uploaded R2 keys and ingestion status (pending→processing→indexed).semantic_memory_chunks— Tier 1 Semantic Memory storingVECTOR(1536)embeddings and page/section JSONB metadata.procedural_rules— Tier 2 Procedural Memory storingmin_confidence_threshold(default 85%) andprotected_paths.incidents— Tier 3 Episodic Memory (also doubling as the LangGraphthread_id) storing root-cause reports, diffs, human feedback, and PR URLs.llm_traces— FinOps telemetry logging model, tokens, latency, and USD cost per agent node.
Every architectural choice in SentinelSRE—presigned uploads, isolated Postgres tenants, parallel agents, and draft-only PRs—exists to make autonomous debugging safe enough for real production teams.