RepoSentinel

Most AI code reviewers only know about public vulnerabilities. RepoSentinel adds a second memory: the review comments a team has already written. When a new pull request repeats a mistake a reviewer flagged before, the earlier discussion comes back with it, alongside any known-CVE code the change resembles.

The hard part turned out to be measurement. Safe code and its vulnerable twin embed almost identically, so similarity search alone can't tell them apart, and it's easy to tune a system like this against a handful of examples and fool yourself. I built the evaluation harness before tuning anything.

• Two collections in Qdrant. Ghost Hunter holds known-CVE vulnerable code: 25 handwritten entries plus function pairs mined from real fix commits in OSV advisories for PyPI and npm, each stored with its patched version where one is known. Team Memory holds closed-PR review comments crawled from GitHub, one memory per comment together with its diff hunk.
• Diff-aware planning. tree-sitter splits changed files into functions (Python, JavaScript, TypeScript through the JavaScript grammar, Go and Java). Only functions that overlap changed lines are analysed, and a whole-file fallback catches changes outside any function, so module-level code is never silently skipped.
• Retrieval. Each function is embedded with jinaai/jina-embeddings-v2-base-code (mean-pooled, 768 dimensions, cached) and searched against both collections. Candidates pass a deliberately low similarity gate of 0.25. Team matches are weighted by recency (a 180-day half-life) and by the reviewer's association with the repository.
• Feedback. Every finding ties back to a vector id. Thumbs-up and thumbs-down votes are aggregated across scans: a memory at a net −2 or below is suppressed, and negative votes downweight it.
• The LLM report. The LLM (Groq, falling back to Gemini) returns structured JSON findings. The prompt lists the exact CVE and PR ids that were retrieved, and any finding that cites an id outside that list is dropped. For CVE matches with a known fix, the prompt includes the vulnerable-to-fixed diff and asks whether the code looks pre-fix or post-fix. The final Markdown is rendered in code, not by the model.
• Delivery. POST /api/v1/analyze/ returns 202 with a job id and runs the scan as a background task, and clients poll for the result. A GitHub Action posts inline review comments anchored to changed lines, deduplicated across pushes with hidden markers, plus a summary comment and a configurable severity gate. There is also a small vanilla-JS dashboard.

In words: the GitHub Action sends a pull request's changed files to the API, which returns 202 and a job id. A background scan uses tree-sitter to keep only the functions that overlap changed lines and embeds them with jina-embeddings-v2-base-code. Each function is searched against two Qdrant collections, Ghost Hunter (known-CVE code) and Team Memory (past review comments). Matches go through the 0.25 similarity gate, team weighting and feedback votes; the cross-encoder rerank stage exists but is off by default. An LLM (Groq, falling back to Gemini) writes structured findings, which are checked against the retrieved ids and rendered to Markdown in code. The Action polls for the result and posts inline comments behind a severity gate.

• Embeddings: jinaai/jina-embeddings-v2-base-code, which replaced UniXcoder (see below)
• Vector store: Qdrant, as a local file store in development and a networked server in production
• Reranker (optional, off by default): BAAI/bge-reranker-v2-m3
• Code parsing: tree-sitter
• LLMs: Groq with Gemini as fallback, through OpenAI-compatible endpoints; transient failures are retried on the same provider before falling back
• API and storage: FastAPI, SQLAlchemy on SQLite in development and PostgreSQL in production
• Delivery: a GitHub Action and a vanilla-JS dashboard that sanitises rendered Markdown with DOMPurify
• Testing: a fast test suite that runs without loading any ML model

The detection harness measures the Ghost Hunter retrieval stage on labelled items, half vulnerable and half safe. An item counts as flagged when a candidate passes the similarity gate. It reports precision, recall and F1 of that flag, and the category hit rate: of the vulnerable items correctly flagged, the share whose top match has the right vulnerability category. The baseline is committed, and a regression test fails if F1 or the category hit rate drops more than 0.02 below it.

The OSV sets are split by advisory, never by individual function, so nothing in them shares an advisory with the corpus being searched.

Precision stays at about 0.5 at every threshold because a safe snippet and its vulnerable twin embed alike; a function and its own fix embed at a cosine similarity of 0.96–0.998. No similarity threshold separates them, which is why retrieval is tuned for recall and the LLM is meant to be the precision filter. The category hit rate fell from 0.96 on the handwritten set to 0.431 once real fix commits were included, so the handwritten numbers don't carry over to real code.

All four land within 27–31 correct out of 75, which is within noise, and the reranker's probability didn't separate vulnerable from fixed code at any threshold.

Team Memory

The first version stored one blob of prose per pull request. Switching to one memory per review comment plus its diff hunk turned matching into code-against-code: in an end-to-end check against the repository's offline demo history, a snippet with a bare except matched a review comment about the same pattern at a similarity of 0.573, up from about 0.11 with the old ingestion. That is one example on demo data, measured with the earlier UniXcoder embedder, not an aggregate metric.

• Retrieval tuned for recall, the LLM as the precision filter. Since no similarity threshold separates a vulnerable function from its fix, the gate sits low at 0.25 and the LLM decides. The GitHub Action gates and comments only on findings the LLM validated, not on the raw retrieval matches.
• Reranking ships off. The original MS-MARCO MiniLM cross-encoder scored close to zero on code pairs, so I stopped gating on its probability. Its replacement, bge-reranker-v2-m3, showed no measurable gain over similarity order at 7–49 seconds per item and about 3 GB of RAM, so it is off by default and never loaded unless enabled.
• Hybrid retrieval ships disabled. I measured a dense-plus-sparse (BM25-style) variant fused with Reciprocal Rank Fusion against a 2-point F1 adoption bar. On the 50-item set it reached F1 0.676 against 0.667 for dense, a 0.9-point gain, so it stays behind a flag.
• Embedder swap. UniXcoder was a self-pooled encoder truncated at 512 tokens. jina-embeddings-v2-base-code is trained for code retrieval with an 8,192-token context in the same size class. Swapped in together with bge-reranker-v2-m3, it moved recall on the handwritten set from 0.96 to 1.0 and F1 from 0.658 to 0.667, with precision unchanged.
• A twin-margin gate exists but is off. Each CVE hit is also scored against its patched twin, and an optional gate can drop matches that look at least as much like the fix as the bug. With functions and fixes embedding at 0.96–0.998, the margin stays within ±0.05 and gating costs recall without gaining precision.
• Anti-hallucination in code. Checking cited ids against the retrieved set and rendering Markdown deterministically keeps the model from inventing CVE or PR references or formatting its own output.
• Hardening. I ran two adversarial review rounds on the code. The second round fixed 13 of its 14 findings, including a stored-XSS path in the dashboard (report Markdown is now sanitised before it reaches the page); the remaining one, a performance refactor, was done afterwards by batching feedback lookups into one query per scan.
• Operational limits. At most 2 scans run model inference at once, and queued scans are re-run after a restart. Retrieval results are committed before the slow LLM call so the database write lock isn't held across it. Local Qdrant is single-process, so the API, ingestion and eval can't run at the same time.

• Code on GitHub (RepoSentinel)
• Current eval baseline
• Project roadmap, including the reranker measurements