RepoSentinel

An AI security reviewer for pull requests: an LLM audits the whole change, a separate verifier call double-checks every finding, and a GitHub Action posts the confirmed ones as inline comments on the exact line.

  • Python
  • FastAPI
  • OpenRouter / Groq
  • Semgrep
  • tree-sitter
  • Qdrant
  • jina-embeddings-v2-base-code
  • SQLAlchemy
  • GitHub Actions

Problem

A pull request is a proposed change that teammates review before it is merged. Reviewers are good at spotting broken features, but security mistakes are easy to miss: a deleted permission check, or user input passed straight into a shell command or a database query. RepoSentinel reviews every pull request on GitHub for these mistakes and leaves a comment on the exact line when it finds one.

The hard part turned out to be measurement. A function and its fixed version embed almost identically, so similarity search alone can't tell a bug from its fix, and it's easy to tune a system like this against a handful of examples and fool yourself. So I measure each stage on code from real CVE fix commits and report what each one can and can't do.

How it works

  • Read the change. tree-sitter splits the changed files into functions and keeps those that overlap changed lines, with a whole-file fallback for changes outside any function. The review gets the unified diff and each changed function before and after the change, plus direct callers and callees from a symbol index of the PR's files when they fit.
  • Gather evidence. Cheap, deterministic checks point at suspicious spots, as leads rather than findings: Semgrep rules, a guard-removal check that compares old and new code for deleted sanitisers, auth checks or safe-API calls, and sensitive sinks on added lines. Optional reference material comes from two Qdrant collections: known vulnerable code from public CVEs (2,890 vulnerable/fixed pairs, 25 handwritten and the rest mined from real fix commits of PyPI and npm advisories) and the team's own past review comments, each stored with its diff hunk.
  • Audit the whole PR with an LLM. The model is asked what the change newly makes exploitable, following data from attacker-controlled sources to sinks. Each finding gives a file, line, CWE, severity, exploit scenario, verbatim quote and a 1–10 confidence, and a finding whose quote isn't in the new code is dropped. The model may ask for more code from the PR before it answers (the option exists, but it never used it in the 160-PR eval run). The PR title and description are never in the prompt.
  • Double-check every finding. Each candidate goes to a separate verifier call with fresh context, which has to establish the source, the missing control, the sink and a reachable path. Only findings confirmed with a confidence of at least 7 out of 10 are reported. A candidate the verifier can't confirm, at a spot where the change provably removed a security control, is listed separately as "worth a look" and never fails the check.
  • Report. The Markdown report is rendered in code, with every LLM-written field escaped. A GitHub Action posts inline comments on the quoted lines, updates them in place on later pushes (deduplicated with hidden markers) and adds a summary comment. It can fail the check on a severe finding, and by default also when the review was partial or failed, because an incomplete review is never reported as clean.
  • Delivery. POST /api/v1/analyze/ returns 202 with a job id and runs the review as a background task, and clients poll for the result. The GitHub Action is the main way to use it. There is also a small vanilla-JS dashboard for pasting code by hand.

Architecture

RepoSentinel: pull request review data flow Pull request Action posts changed files 202 + job id; Action polls Read the change diff + changed functions, before and after (tree-sitter) Leads Semgrep, sinks, removed guards Reference CVE code, team comments (Qdrant) LLM audit of the whole PR what does it newly make exploitable? each finding quotes the code Verifier, one call per finding reported if confirmed at 7/10 or more non-blocking “worth a look” tier Review output Markdown rendered in code, escaped inline PR comments, severity gate
Files mode, as used by the GitHub Action. The highlighted audit and verifier make the decision; the dashed reference stage is optional.

In words: the GitHub Action sends a pull request's changed files to the API, which returns 202 and a job id. A background job collects the diff and each changed function before and after the change. Deterministic leads (Semgrep hits, sensitive sinks and removed guards) and, optionally, reference material retrieved from Qdrant (known-CVE code and past review comments) feed an LLM audit of the whole change, which ignores the PR title and description and must quote the code for every finding. A separate verifier call re-checks each candidate. Only findings confirmed with a confidence of 7 or more are reported, and unconfirmed candidates at a spot where the change removed a security control are listed as non-blocking "worth a look" items. The report is rendered to Markdown in code with all LLM text escaped, and the Action polls for it and posts inline comments behind a severity gate.

Stack

  • LLMs: free OpenRouter models by default (Qwen 3.8 27B, then Gemma 4 31B), falling back to Groq (gpt-oss-120b, then Qwen 3.8 27B), through OpenAI-compatible endpoints; Gemini is also supported. An answer that fails its format check falls through to the next model.
  • Static analysis: Semgrep CE with a vendored, permissively licensed rule set (GitLab sast-rules)
  • Code parsing: tree-sitter, for planning, the symbol index and the guard-removal check
  • Embeddings: jinaai/jina-embeddings-v2-base-code for the optional reference retrieval, which replaced UniXcoder (see below)
  • Vector store: Qdrant, as a local file store in development and a networked server in production
  • Reranker (optional, off by default): BAAI/bge-reranker-v2-m3
  • API and storage: FastAPI, SQLAlchemy on SQLite in development and PostgreSQL in production
  • Delivery: a GitHub Action and a vanilla-JS dashboard that sanitises rendered Markdown with DOMPurify
  • Prompts: the audit and verifier prompts are adapted from anthropics/claude-code-security-review (MIT) and openai/codex-security (Apache-2.0)
  • Testing: a fast pytest suite with no model loads, a mocked LLM and a fresh temporary SQLite database per run, plus ruff

Results and evaluation

Every stage is measured on code from real CVE fix commits, mined from OSV advisories for PyPI and npm and split by advisory, so nothing evaluated shares an advisory with the corpus being searched. For the PR-level review, each fix commit reversed gives a bug-introducing PR, and the real fix gives a PR that should pass.

What each stage can do on its own
What was measured Result Sample
Similarity search alone Can't tell a bug from its fix: precision 0.5 (a coin flip) at every threshold. A function and its fixed version embed at a cosine similarity of 0.96–0.998. 1,054 eval functions
Old design: an LLM reviews each changed function on its own Caught 1 of 8 real bugs on the right lines (95% CI 2–47%) 8 bug/fix pairs + 13 ordinary functions
Static checks only (Semgrep + guard-removal check), best case Reach 23% of bugs at a 4% false-alarm rate on benign changes 502 held-out bug/fix pairs
Guard-removal "alert" tier alone Fires on 3% of bugs, with 0 false alarms measured 506 pairs + 994 benign edits

The per-function result comes from a different, much smaller test than the PR-level numbers below, so the two aren't directly comparable. Static checks are precise but see less than a quarter of the bugs, so they work as leads for the LLM, not as the verdict.

PR-level review (preliminary)

The current design is measured on a dev sample of 200 PRs: 60 bug-introducing PRs (reversed fixes), their 60 real fixes and 80 benign PRs. Not every PR finished, so the numbers below are over the 160 that were scored. Reported findings are what the review posts after the verifier; audit candidates are everything the audit raised before verification. A bug-introducing PR counts as caught when a finding lands in the function the bug is in.

PR-level review on the dev sample (160 of 200 PRs scored, previous configuration)
Measure Reported Audit candidates
Bug-introducing PRs caught within the changed function 16/55 35/55
Fix PRs wrongly flagged 2/57 4/57
Benign PRs wrongly flagged 0/48 2/48

On the exact lines, the reported findings catch 12/55. Leaving out denial-of-service and timing bugs, which the prompt doesn't target, they catch 16/41 within the function. The verifier trades recall for quiet: within the changed function it keeps 16 of the 35 catches the audit made, and removes both benign false alarms and 2 of the 4 fix-PR alarms. Tuning that trade-off is the next step.

These numbers are preliminary. They come from a partial run using the free Qwen 3.8 27B model through OpenRouter, with temperature 0, the old verifier prompt and a confidence cutoff of 8. The current defaults differ (Qwen's recommended sampling, a rewritten verifier prompt and a cutoff of 7), and a rerun with them is pending, as is the held-out test split. The benign PRs are files that changed alongside security fixes in the same commits, not ordinary PRs, and only 48 of the 80 were scored: if all 32 unscored ones were flagged, the false-alarm rate would be 32/80. The 0/48 also rests on the cutoff of 8. One benign PR was confirmed at confidence 7, so the same verdicts at today's cutoff would give 1/48.

Retrieval

Similarity search ranks known-CVE code by how alike it looks, and that isn't enough to call something vulnerable. Its category hit rate (the right bug class in the top match) is about 0.43 overall, but a catch-all "other" class inflates it: on named categories it is 0.279, and always guessing "other" would score 0.447. In a three-arm test on one model (8 bug/fix pairs and 13 ordinary functions), the per-function review with retrieved CVEs flagged exactly the same items as the same prompt with none, for 77% more tokens. So retrieval is only context, and the LLM review with verification makes the decision.

Reranker comparison on 150 held-out OSV items (75 vulnerable)
Configuration Category hit Correct Seconds per item
No reranker 0.373 28/75 0.05
bge-reranker-v2-m3, 1,024 tokens 0.387 29/75 48.6
bge-reranker-v2-m3, 512 tokens 0.413 31/75 25.0
bge-reranker-base, 512 tokens 0.360 27/75 7.2

All four land within 27–31 correct out of 75, which is within noise, and the reranker's probability didn't separate vulnerable from fixed code at any threshold.

What I don't claim

  • Final PR-level numbers. They are preliminary: a partial run on a dev sample with 40 of 200 PRs unscored, measured with the previous configuration. The rerun with the current defaults and the held-out test split are both pending.
  • A realistic false-alarm rate. The benign PRs are bystander files from security-fix commits, 32 of 80 were not scored (worst case 32/80), and the 0/48 would be 1/48 at today's cutoff.
  • That the per-function and PR-level results can be compared. They come from different tests, and the per-function one has only 8 bug/fix pairs.
  • That retrieval detects vulnerabilities. Its precision is 0.5 at every threshold, and on named categories it picks the right bug class less often than always guessing "other" would.
  • Any gain from reranking. The differences in the comparison above are within noise.

Decisions and trade-offs

  • The LLM review decides; retrieval is context. No similarity threshold separates a vulnerable function from its fix, and in the three-arm test retrieved CVEs changed no verdict. So retrieval only supplies optional reference material, and in the PR review the retrieved fix examples are off by default.
  • Review the PR as a whole. The earlier per-function review caught 1 of 8 real bugs on the right lines. Tools that produce useful PR security reviews share a PR-level framing, a separate verification pass and static tools as leads, so files mode now audits the change as a whole and verifies each candidate. The per-function review is kept (REVIEW_MODE=units) for comparison.
  • Static checks are leads, not verdicts. They are precise but reach about 23% of bugs at best. A guard-removal alert is still reported as its own deterministic finding, but by default it fails the check only when the LLM review or Semgrep flags the same spot.
  • The PR description is ignored. A description that says "harmless refactor" can't talk the review out of a finding. The eval checks this offline: for all 60 bug-introducing PRs in the dev sample, a misleading title and body left the prompts byte-identical.
  • It never fakes a clean result. If part of the review failed or ran out of budget, the result says partial or failed, lists what wasn't reviewed and, by default, fails the check. If every LLM call fails, the scan still completes with its Semgrep and guard-removal results.
  • Anti-hallucination in code. Every finding must quote the offending code, a finding whose quote isn't there is dropped server-side, and the quote anchors the PR comment. A cited CVE or team-PR id outside the retrieved set is removed from the finding. Every LLM-written field is escaped when rendered, so a finding can't inject links, HTML, @-mentions or the Action's hidden markers.
  • Prompt-injection hardening. PR code, file names, corpus code, advisory text and team comments go into tagged blocks with a per-prompt random nonce, and the system prompt treats them as data. An adversarial review found that stripping HTML comments could hide code from the model (# <!-- … # --> hid an os.system call), so comment openers are now defused in place and nothing in the reviewed code is deleted.
  • Reranking ships off. The original MS-MARCO MiniLM cross-encoder scored about zero on every code pair. Its replacement, bge-reranker-v2-m3, showed no measurable gain over similarity order at 7–49 seconds per item and about 3 GB of RAM, so it is off by default and never loaded unless enabled.
  • Hybrid retrieval and the twin-margin gate exist but are off. Dense-plus-BM25 fusion measured an F1 gain below the adoption bar. Scoring each CVE hit against its stored fix (the twin margin) separated vulnerable from fixed code only slightly better than chance (AUC 0.534 on 506 held-out pairs), too weak to gate on.
  • Embedder swap. jina-embeddings-v2-base-code replaced UniXcoder: a real embedding model rather than self-pooled hidden states, with an 8,192-token context instead of truncation at 512 tokens, in the same size class. It was never expected to solve the safe-versus-vulnerable problem.
  • Free models, bounded cost. The default model chain is free. Groq's free tier only serves prompts up to about 7.5K tokens, so the larger PR-review prompts run on OpenRouter and Groq backs up the smaller calls. A typical PR review costs about 1–6 LLM calls (roughly 5K–40K tokens), at most 12.
  • Operational limits. At most 2 scans run at once, and queued scans are re-run after a restart. Retrieval results are committed before the slow LLM call so the database write lock isn't held across it. Local Qdrant is single-process, so the API, ingestion and eval can't run at the same time.