Semantic Search with RAG Engine

Retrieval decides what a RAG system's language model gets to read. It's also easy to keep adding stages to it, such as a second retriever or a reranker, on the assumption that each one helps. I built this search engine so that every stage has to show its value on gold relevance labels, with a paired significance test, before it becomes the default.

The same applies to the grounded answers. A model that is told to abstain when the evidence is missing still has to be checked: does it abstain when it should, and does it answer when it has nothing to go on?

The corpus is BEIR SciFact: 5,183 scientific abstracts, with 300 test claims and gold relevance labels (qrels).

• Two first-stage retrievers. BM25 through bm25s with English stemming, and dense retrieval with BAAI/bge-small-en-v1.5 embeddings stored in Qdrant. Each returns its top 100 candidates.
• Fusion. Reciprocal Rank Fusion (k = 60) merges the two lists by rank position, so BM25 scores and cosine similarities never need to be calibrated against each other.
• Optional reranking. A cross-encoder can rescore the top 32 fused candidates when a request asks for mode=hybrid_rerank. The rest keep their fused order.
• Search. GET /search returns the ranked results (8 by default), and the same app serves a small self-contained search UI. Modes are bm25, dense, hybrid (the default) and hybrid_rerank.
• Grounded answers. GET /answer passes the top 5 results to an LLM through an OpenAI-compatible client. The prompt asks it to answer only from those passages, cite them as [n] and say so when they don't contain enough information. The citations are mapped back to document ids. For a claim rather than a question, the model ends with a verdict line: supported, refuted or not enough evidence.

In words: a query goes to BM25 and to dense retrieval on Qdrant, 100 candidates each. Reciprocal Rank Fusion merges the two lists. In hybrid_rerank mode only, a cross-encoder then reranks the top 32. /search returns the ranked list. /answer sends the top 5 passages to an LLM, which answers with [n] citations or says the context doesn't contain the evidence.

• Embeddings: BAAI/bge-small-en-v1.5, with its query instruction prefix on queries only
• Lexical: bm25s with PyStemmer (English)
• Vector store: Qdrant, embedded local mode
• Fusion: Reciprocal Rank Fusion, hand-rolled
• Rerankers evaluated: cross-encoder/ms-marco-MiniLM-L-6-v2 and BAAI/bge-reranker-base
• API: FastAPI, with slowapi rate limiting
• LLMs: free OpenRouter models by default: Ling 3.0 Flash Sante as the generator and Nemotron 3 Ultra as the judge, from different model families so the generator never grades its own output. openai/gpt-oss-120b is an optional paid generator, and Groq, Ollama or any OpenAI-compatible endpoint can be configured instead.
• Evaluation: ranx on the BEIR SciFact qrels, with paired significance tests
• Packaging: uv lockfile, CPU-only PyTorch, a digest-pinned Docker base image, and CI that runs ruff and pytest

All retrieval numbers are on the 300 BEIR SciFact test queries, scored against the gold qrels. Every comparison uses a paired two-sided t-test over the 300 per-query scores, reported with the difference (Δ), the p-value and the per-query win/tie/loss count.

Reranking only reorders the top 32 of the 100 fused candidates, so it can't change Recall@100.

What the numbers say

• Fusion's gain is recall, not ranking. Hybrid beats BM25 on nDCG@10, but its +0.011 over dense alone is not significant (p = 0.26), so I don't claim it. Where hybrid does separate from dense is Recall@100: 0.942 to 0.965 (p = 0.035), decided by 11 queries out of 300.
• That recall gain sits entirely on NEI claims. Split by SciFact's claim labels, all 11 of those queries are "not enough information" claims, where annotators found no rationale in any abstract but the qrels still mark the cited one relevant. On the 188 claims that do have evidence, hybrid and dense have identical Recall@100 on every query. So fusion gives a fuller candidate pool, but on this corpus it doesn't retrieve more supporting or refuting evidence than dense alone.
• Neither reranker paid off. MS-MARCO MiniLM, trained on short web queries, cost 0.027 nDCG@10 against plain hybrid (p = 0.056, so not significant either). The domain-appropriate bge-reranker-base beat MiniLM by the same amount (p = 0.038) and landed on top of doing nothing: Δ = +0.0001, p = 0.996. The choice of reranker matters; reranking itself didn't help here.
• RRF's k doesn't matter much on this data. Replaying the cached top-100 lists with k from 1 to 100 moved nDCG@10 by at most 0.0046 (all p ≥ 0.19), and Recall@100 stayed at 0.965. Weighting dense at 0.3 instead of equally cost Recall@100 (0.928, p = 0.002).

Measured in-process on an Intel i5-10210U with PyTorch on 4 threads, after 3 warm-up queries, excluding HTTP. The rerank rows use a fixed 40-query sample. bge-reranker-base costs roughly 188 times the latency of plain hybrid for a statistical tie.

Grounded answers

The answer eval scores each reply's verdict and its answer-or-abstain decision against SciFact's own labels, without a judge. An LLM judge scores only faithfulness and context relevance. Abstention is checked against a rationale oracle: the context "has evidence" when a document the annotators cited with rationale sentences is in the top 5. The sample is 50 random claims (seed 13), generated with Ling 3.0 Flash Sante and judged by Nemotron 3 Ultra.

An early version of the project's README read an answer rate of 0.50 on 10 claims as the system correctly abstaining. That was never measured: an abstention rate on its own can't tell a justified refusal from an over-cautious one. Crossing each decision with whether evidence was actually retrieved is what made it measurable, and the choice of oracle matters too. Scored against BEIR's qrels, the same answers give an abstention precision of 0.29. Against the rationale documents it is 0.82, because 9 of the 12 "false" abstentions were NEI claims where refusing was the right call.

• Hybrid is the default, for recall. I kept fusion for its fuller candidate pool, not for top-10 ordering, and I report that the extra recall sits on NEI claims.
• Reranking ships off. It stays available as mode=hybrid_rerank. The comparison also meant rewriting the project's own README, which had claimed that reranking "only pays off with a domain-appropriate model" before that had ever been run.
• The rerank slice is capped at 32. A cross-encoder runs a full forward pass per candidate, and retrieval is serialised behind a lock, so an unbounded slice would let one client at the rate limit hold the service for minutes. The cap is a denial-of-service guard first and a latency setting second.
• Rate limiting is keyed carefully. The API is unauthenticated, so per-IP limits (30 searches and 10 answers a minute) are its only guard. Behind a proxy it counts in from the right of X-Forwarded-For by the configured number of hops, which closed a hole where a client could choose its own rate-limit key.
• Prompt injection is handled structurally. The question is collapsed to a single line with no repeated quote runs before it goes into the prompt, and tests check that a crafted query can't forge a new question or answer turn.
• Free models by default. A full answer eval costs $0 on the default models. The trade-off is that free models can disappear: the previous judge's free variant stopped being served, which forced a judge change and a re-run.
• Reproducibility. The eval harness caches each configuration by a signature of everything that affects its numbers and checkpoints as it goes. PyTorch is the CPU-only build (the virtual environment went from 5.4 GB to 1.6 GB). The tests run on fakes, with no models, index or API key.

• Code on GitHub (Semantic Search with RAG Engine)
• Retrieval results and significance tests
• Label-stratified analysis and RRF sensitivity
• Latency measurements
• Answer-quality results