Aakash Devkar
← All projects

Multi-agent · RAG · Jul 2026 – Sep 2026

Autonomous Code Review Agent

Four LLM agents review every pull request and block merges on critical bugs.

merge-gate block recall
17% → 83%
merge-gate block recall
false merge blocks
0
false merge blocks
LLM cost per reviewed file
$0.0017
LLM cost per reviewed file

01

The problem

Code review is a bottleneck, and LLM review bots fail in three ways. They see only the diff, so they miss who calls the changed function. They rate real vulnerabilities too low to act on: in the first eval the agents found 5 of the 6 blocking issues but rated them HIGH. And a diff is attacker-controlled text, so a planted comment can talk the model out of reporting a bug.

02

Architecture: review pipeline

  1. 1

    PR webhook

    GitHub App webhook verified with HMAC-SHA256; job stored in SQLite with a duplicate-delivery guard and a stale-commit check

  2. 2

    Filter & screen

    Skips lockfiles, generated and oversized patches; Bedrock Guardrails screens each patch for prompt injection and quarantines flagged files

  3. 3

    Context retrieval

    Callers, callees and imports from an AST call graph, plus similar code from Qdrant (Titan embeddings), capped at 8,000 chars

  4. 4

    4 agents in parallel

    Security, architecture, test coverage and documentation on Bedrock gpt-oss-120b / 20b; 45 s timeout each; untrusted text in randomly named tags

  5. 5

    Post-processing

    Re-anchors line numbers to the quoted snippets, applies the deterministic severity floor and ceiling, de-duplicates and ranks

  6. 6

    Synthesizer

    gpt-oss-120b writes the PR comment, with a deterministic fallback

  7. 7

    Merge gate

    Any CRITICAL finding fails the ai-code-review status until a reviewer with write access approves or blocks

A React dashboard streams each agent's progress over WebSocket. Groq is the fallback LLM provider.

03

Design decisions

Plain asyncio orchestration

Instead of
LangGraph or CrewAI
Why
The four agents are independent, so they only need gather, a timeout and failure isolation, and I can see exactly what runs.

Severity decided in code (a deterministic floor and ceiling)

Instead of
Trusting the LLM's severity, or prompt tuning
Why
The agents detected the issues but under-rated them. The floor and ceiling took block recall from 17% to 83% with 0 false blocks.

Bedrock gpt-oss as primary, Groq as fallback

Instead of
Groq only, or Bedrock Nova 2 Lite
Why
Same quality as Groq at p50 5.5 s vs 19.2 s per file, with no daily token cap. Nova followed injected instructions in evals.

The injection guardrail fails open

Instead of
Failing closed
Why
Failing closed would block every PR during an AWS incident; the code-level severity floor still applies.

04

What I built

  • FastAPI service installed as a GitHub App: HMAC-verified webhooks (constant-time compare), duplicate-delivery and stale-commit guards, background reviews and graceful shutdown.
  • Codebase-aware RAG: AST chunking per function and class, Titan V2 or local granite embeddings in Qdrant (one collection per model, filtered by repo), and an AST call graph rebuilt on every push to the default branch.
  • Four specialist agents run in parallel with asyncio, each with its own prompt, JSON schema, parse-retry and timeout; one agent's failure never stops the others.
  • Prompt-injection defence in layers: a Bedrock Guardrails screen, random per-call prompt delimiters, and a deterministic severity floor and ceiling in code.
  • LLM routing with Bedrock as primary and Groq as fallback, per-model client-side rate limiting, and fail-fast on daily quotas.
  • Human-in-the-loop merge gate through GitHub commit statuses, with approve, block and clear overrides.
  • React 19 dashboard with GitHub OAuth, single-use WebSocket tickets, live agent progress, a Monaco diff viewer and a React Flow call graph.
  • MCP server with 7 read-only tools, so a developer can review a patch from Claude Code or Cursor before pushing.
  • A 45-case labelled eval set with a strict scorer, and a CI eval gate on five thresholds.
  • The whole AWS stack in Terraform, with CloudWatch metrics, 7 alarms and LangSmith tracing.

05

Hard problems

Real vulnerabilities were found but didn't block merges

Problem
In the first eval, merge-gate block recall was 17% (1 of 6). The agents detected 5 of the 6 blocking issues but rated them HIGH, and only CRITICAL blocks a merge.
Fix
A deterministic floor raises 5 canonical classes (SQL injection, command injection, hard-coded secrets, unsafe deserialization, missing auth) to CRITICAL only when the wording and a regex on the actual code agree. A ceiling caps context-dependent classes such as SSRF at HIGH.
Verified
Re-scoring the saved runs on the 33 original cases: block recall 83% (5 of 6), block precision 100%, 0 false blocks. Unit tests cover the floor.

Prompt injection through the diff

Problem
On Nova 2 Lite, the security agent returned no findings for an f-string SQL injection because an adjacent comment told it to. A fake closing tag could also break out of the fenced diff.
Fix
Four layers, because none is enough alone: a Bedrock Guardrails screen that quarantines flagged files, random per-call tag names the diff can't guess, a production model chosen with injection evals, and the code-level severity floor.
Verified
The guardrail caught 5 of 8 injection cases with 0 of 37 false positives, and gpt-oss ignored the injection in all 3 misses. The CI eval gate passes all 5 rules.

LLM line numbers pointed at the wrong code

Problem
In the first eval, 37 of 157 findings sat more than 3 lines from their own quoted snippet, so PR comments pointed at the wrong lines.
Fix
Each finding is re-anchored to where its quoted snippet appears in the patch's new-file lines (whitespace-normalised), choosing the match nearest the reported line.
Verified
In later runs, strict line scoring and snippet-anchored scoring agree exactly. Unit tests cover re-anchoring.

06

Results

Merge-gate block recall (33 cases)17% → 83%, block precision 100%, 0 false blocks
Recall, all categories81% (25 / 32)
Exact-severity match46% → 73%
Latency per file (Bedrock)p50 5.5 s, p95 10.9 s
LLM cost$0.0017 per reviewed file
Injection guardrail5 / 8 caught, 0 / 37 false positives
Tests597 passed with the network blocked, 84% app line coverage

Every eval number comes from a single run, so differences of 1–2 issues are noise. The call graph parses Python only.

07

Deployment

ProviderAWS ap-south-1 (Mumbai), defined in Terraform (50 resource blocks)
EdgeCloudFront serves the dashboard from private S3 and forwards API, WebSocket and webhook paths to an ALB that only accepts CloudFront
ComputeECS Fargate, 1 task (0.5 vCPU / 1 GB); SQLite and call-graph JSON on EFS
AI servicesAmazon Bedrock (gpt-oss, Titan embeddings, Guardrails), Qdrant Cloud, Groq as fallback
Secrets & monitoringSecrets Manager; CloudWatch logs, EMF metrics and 7 alarms → SNS email
CI/CD5 required checks (tests, dashboard lint + build, Docker build + Trivy, LLM eval gate, the bot's own review); deploys via GitHub OIDC after manual approval, with ECS circuit-breaker rollback
CostAbout $1.5 a day while running

08

Stack

PythonFastAPIAmazon BedrockQdrantGroqLangSmithMCPSQLiteReactMonacoReact FlowTerraformAWS ECS FargateDockerGitHub Actions