Scientific Claim Checker

For a scientific or health claim, hybrid BM25 and bge-small search over 5,183 SciFact abstracts finds the relevant papers, and a free LLM says whether they support the claim, refute it or don't contain enough evidence, citing the papers it used.

  • Python
  • FastAPI
  • Qdrant
  • bm25s
  • bge-small
  • ranx
  • Docker

Problem

Retrieval decides what a RAG system's language model gets to read. It's also easy to keep adding stages to it, such as a second retriever or a reranker, on the assumption that each one helps. I built this search engine so that every stage has to show its value on gold relevance labels, with a paired significance test, before it becomes the default.

The same applies to the grounded answers. A model that is told to abstain when the evidence is missing still has to be checked: does it abstain when it should, and does it answer when it has nothing to go on?

How it works

You give it a scientific or health claim, such as "Vitamin D deficiency increases the risk of multiple sclerosis". It finds the most relevant papers, reads them and says whether they support the claim, refute it or don't contain enough evidence, citing the papers it relied on. The corpus is BEIR SciFact: 5,183 biomedical paper abstracts, with 300 test claims, gold relevance labels (qrels) and each claim's expert label.

  • Two first-stage retrievers. BM25 through bm25s with English stemming, and dense retrieval with BAAI/bge-small-en-v1.5 embeddings stored in Qdrant. Each returns its top 100 candidates.
  • Fusion. Reciprocal Rank Fusion (k = 60) merges the two lists by rank position, so BM25 scores and cosine similarities never need to be calibrated against each other.
  • Optional reranking. A cross-encoder can rescore the top 32 fused candidates when a request asks for mode=hybrid_rerank. The rest keep their fused order.
  • Search. GET /search returns the ranked results (8 by default), and the same app serves a small self-contained search UI. Modes are bm25, dense, hybrid (the default) and hybrid_rerank, plus the optional networked web and hybrid_web.
  • Grounded answers. GET /answer passes the top 5 results to an LLM through an OpenAI-compatible client (by default the free Ling 3.0 Flash Sante on OpenRouter). The prompt asks it to answer only from those passages, cite them as [n] and say so when they don't contain enough information. The citations are mapped back to document ids. For a claim rather than a question, the model ends with a verdict line: supported, refuted or not enough evidence. If that reply has no verdict the parser can read (cut off at the token budget, or prose without a verdict line), the generator makes one verdict-only follow-up call.
  • Optional web search. Off the default path. A deterministic pipeline with no LLM pools Semantic Scholar keyword search (on a rule-based keyword rewrite of the claim), Semantic Scholar snippet search and PubMed Best Match, then re-ranks the pool locally by RRF of bge-small similarity and BM25. Snippets that quote the claim verbatim or mention SciFact are dropped, because NLP papers reprint SciFact claims together with their labels. hybrid_web fuses the web list with the local one.
The search UI with the claim "Vitamin D deficiency is associated with increased risk of multiple sclerosis" entered in hybrid (RRF) mode. It shows 8 results; the first three are the abstracts "Vitamin D and latitude as environmental factors in multiple sclerosis", "Past exposure to sun, skin phenotype, and risk of multiple sclerosis: case-control study" and "Vitamin D: The ‘sunshine’ vitamin", each with its document id and fused score.
The search UI served by the same FastAPI app, in the default hybrid (RRF) mode, searching the 5,183 SciFact abstracts for one claim.

Architecture

Scientific Claim Checker: query data flow Query search UI or API BM25 (bm25s) top 100 Dense: bge-small Qdrant, top 100 Reciprocal Rank Fusion rank-based, k = 60 Cross-encoder rerank, top 32 only in hybrid_rerank mode GET /search ranked results (8 by default) GET /answer top 5 to an LLM cites [n] or abstains
Default path in solid outline; the dashed stage runs only when requested.

In words: a query goes to BM25 and to dense retrieval on Qdrant, 100 candidates each. Reciprocal Rank Fusion merges the two lists. In hybrid_rerank mode only, a cross-encoder then reranks the top 32. /search returns the ranked list. /answer sends the top 5 passages to an LLM, which answers with [n] citations or says the context doesn't contain the evidence.

Stack

  • Embeddings: BAAI/bge-small-en-v1.5, with its query instruction prefix on queries only
  • Lexical: bm25s with PyStemmer (English)
  • Vector store: Qdrant, embedded local mode
  • Fusion: Reciprocal Rank Fusion, hand-rolled
  • Rerankers evaluated: cross-encoder/ms-marco-MiniLM-L-6-v2 and BAAI/bge-reranker-base
  • API: FastAPI, with slowapi rate limiting
  • LLMs: free OpenRouter models by default: Ling 3.0 Flash Sante as the generator and Nemotron 3 Ultra as the judge, from different model families so the generator never grades its own output. openai/gpt-oss-120b is an optional paid generator, and Groq, Ollama, any OpenAI-compatible endpoint or a price-capped paid OpenRouter model can be configured instead.
  • Web search (optional): Semantic Scholar search and snippets plus PubMed, rate-limited, with responses cached for the eval
  • Evaluation: ranx on the BEIR SciFact qrels with paired significance tests; answer verdicts scored against SciFact's claim labels, with exact McNemar tests between answer runs
  • Packaging: uv lockfile, CPU-only PyTorch, a digest-pinned Docker base image, and CI that runs ruff and pytest

Results and evaluation

All retrieval numbers are on the 300 BEIR SciFact test queries, scored against the gold qrels. Comparisons use a paired two-sided t-test over the 300 per-query scores, reported with the difference (Δ), the p-value and the per-query win/tie/loss count. Per-query Recall@100 is almost binary, so for it the exact sign test on the wins and losses is the fairer test.

Retrieval quality per stage (BEIR SciFact, 300 queries)
Configuration nDCG@10 Recall@100 MRR@10 MAP@100
BM25 0.6863 0.9127 0.6492 0.6439
Dense (bge-small) 0.7127 0.9417 0.6822 0.6736
Hybrid (RRF), the default 0.7241 0.9650 0.6886 0.6816
Hybrid + MS-MARCO MiniLM rerank 0.6975 0.9650 0.6632 0.6558
Hybrid + bge-reranker-base 0.7242 0.9650 0.6901 0.6834

Reranking only reorders the top 32 of the 100 fused candidates, so it can't change Recall@100. Hybrid's Recall@10 is 0.855 (measured in the web-search comparison run).

Paired significance tests (300 queries, p < 0.05, uncorrected)
Comparison Metric Δ p W/T/L Significant
Hybrid vs BM25 nDCG@10 +0.0378 0.0007 72/202/26 Yes
Hybrid vs dense nDCG@10 +0.0114 0.2578 53/213/34 No
Hybrid vs dense Recall@100 +0.0233 0.0654 9/289/2 No (exact sign test; the t-test gives p = 0.0346)
MiniLM rerank vs hybrid nDCG@10 −0.0266 0.0555 45/192/63 No
bge rerank vs hybrid nDCG@10 +0.0001 0.9964 48/196/56 No
bge rerank vs MiniLM rerank nDCG@10 +0.0266 0.0376 65/190/45 Yes

What the numbers say

  • Fusion's case is recall, not ranking, and it is only suggestive. Hybrid beats BM25 on nDCG@10 (p = 0.0007), but its +0.011 over dense alone is not significant (p = 0.26), so I don't claim it. Against dense, hybrid lifts Recall@100 from 0.942 to 0.965, but only 11 of 300 queries differ (9 better, 2 worse), and the exact sign test gives p = 0.065: suggestive, not significant.
  • Whatever recall it adds sits on NEI claims. Split by SciFact's claim labels, all 11 of those queries are "not enough information" claims, where annotators found no rationale in any abstract but the qrels still mark the cited one relevant. On the 188 claims that do have evidence, hybrid and dense have identical Recall@100 on every query. So fusion may give a fuller candidate pool, but on this corpus it doesn't retrieve more supporting or refuting evidence than dense alone.
  • Neither reranker paid off. MS-MARCO MiniLM, trained on short web queries, cost 0.027 nDCG@10 against plain hybrid (p = 0.056, so not significant either). The domain-appropriate bge-reranker-base beat MiniLM by the same amount (p = 0.038) and landed on top of doing nothing: Δ = +0.0001, p = 0.996. The choice of reranker matters; reranking itself didn't help here.
  • RRF's k doesn't matter much on this data. Replaying the cached top-100 lists with k from 1 to 100 moved nDCG@10 by at most 0.0046 (all p ≥ 0.19), and Recall@100 stayed at 0.965. Weighting dense at 0.3 instead of equally cost Recall@100 (0.928, p = 0.002).
Retrieval latency per query, 4-core laptop CPU
Configuration Mean (s) p95 (s) Queries
Hybrid (RRF) 0.124 0.159 300
Hybrid + MS-MARCO MiniLM rerank 3.94 5.63 40
Hybrid + bge-reranker-base 23.4 29.3 40

Measured in-process on an Intel i5-10210U with PyTorch on 4 threads, after 3 warm-up queries, excluding HTTP. The rerank rows use a fixed 40-query sample. bge-reranker-base costs roughly 188 times the latency of plain hybrid for a statistical tie.

Web search

The optional web modes are measured on the same 300 claims by whether they find the gold paper. The rewrite rules, sources and settings were chosen on 100 train claims and frozen before the test run.

Web search against local search (300 SciFact test claims)
Configuration Recall@5 Recall@100 nDCG@10
Semantic Scholar, raw claim as query 0.040 0.056 0.035
Semantic Scholar, keyword rewrite + local re-rank 0.185 0.244 0.159
Pooled: rewrite + snippets + PubMed, re-rank of top 50 (web default) 0.228 0.384 0.203
Local hybrid (RRF, 5,183 abstracts), for scale 0.766 0.965 0.724

Pooling raised Recall@100 over rewrite + re-rank on 46 claims and lowered it on none (p < 0.0001). Web search still trails local search on every metric (p < 0.0001), which is expected: Semantic Scholar searches about 200M papers against these 5,183, so this asks whether open-web search can find the gold paper at all. Not every gold paper maps to a Semantic Scholar record, so even a perfect web search would score a Recall@100 of about 0.90 here. These rows use the evaluation's web settings. The API embeds at most 30 candidates instead of 50, drops sources still pending after 20 s and drops dataset-dump snippets that overlap a SciFact claim, so its results can differ from the table.

Grounded answers

The answer eval scores each reply's verdict and its answer-or-abstain decision against SciFact's own labels, without a judge. An LLM judge scores only faithfulness and context relevance. Abstention is checked against a rationale oracle: the context "has evidence" when a document the annotators cited with rationale sentences is in the top 5, so abstaining on a claim with no such document is the right call. The run covers all 300 test claims, generated with Ling 3.0 Flash Sante and judged by Nemotron 3 Ultra. Both are free, so a full run costs $0.

Answer quality on all 300 SciFact test claims
Metric Score
3-class verdict accuracy (vs gold label, no judge; 240 of 300) 0.80
95% Wilson interval on that accuracy 0.751–0.841
Replies citing at least one paper (282 of 300) 0.94
Abstention precision (abstained and had no evidence) 0.83
Abstention recall (had no evidence and abstained) 0.70
False abstention (had evidence, abstained anyway) 0.10
Answered without evidence, as a share of answers 0.19
Faithfulness over answered claims (LLM judge, 192 answers) 0.98
Context relevance (LLM judge) 0.72

By label, verdict accuracy is 0.815 on SUPPORT (124 claims), 0.891 on CONTRADICT (64) and 0.732 on NEI (112). Almost all of the 60 wrong verdicts are evidence-judgement errors, not format ones: 30 of the 112 NEI claims were answered anyway, and 23 SUPPORT or CONTRADICT claims were abstained on (for 14 of the 17 SUPPORT ones, the rationale document was in the context).

Post-processing steps on the same answers, from 0.73 with a strict verdict-line parser (300 claims, paired exact McNemar against the previous step)
Step Verdict accuracy Fixed Broken p
+ verdict-recovery parser (verdict restated in prose, or a stance rule) 0.7767 14 0 0.0001
+ verdict re-ask (one verdict-only call when a claim's reply has none) 0.8000 7 0 0.016

The re-ask fired on 15 of 300 claims and produced a verdict for 14 (12 correct) without breaking any. Both steps were chosen with the test claims in view: the parser's patterns came from an error analysis of test failures, and the re-ask was adopted after three candidate fixes were scored on test. The parser's gain survives any correction. The re-ask's becomes borderline once the other variants scored on the same claims are counted (Holm-adjusted p from 0.047 to 0.078).

Models and labels

  • Paid models were not better. Paid GPT-6 Luna got fewer verdicts right than the free generator (0.71 vs 0.78, both without the re-ask; p = 0.012). Paid gpt-oss-120b was indistinguishable on accuracy over the 50 claims tested, too few to rule a difference in or out, and cited a passage far less often (0.42 of replies vs 0.94, p < 0.0001).
  • Tried, measured, not adopted. An evidence-first prompt, two verifier models (an off-the-shelf NLI model and a fine-tuned PubMedBERT) and a "second look" step. None beat the current pipeline.
  • Some of the dataset's labels are wrong. A published 2026 audit (Sylvestre, BioNLP 2026) corrects 11 of the 300 test labels (about 4%) and calls 8 more debatable. Scored against the corrections, the same answers reach 0.837. The audit skipped the NEI claims and all 11 changes happened to favour the model, so this is a one-sided check, and the headline keeps the original labels.

An early version of this eval reported an answer rate of 0.50 from the first 10 claims in dataset order, a head slice rather than a sample, and an abstention rate on its own can't tell a justified refusal from an over-cautious one. Crossing each decision with whether evidence was actually retrieved is what made it measurable, and the choice of oracle matters too. Scored against BEIR's qrels, which mark a cited abstract relevant even for NEI claims, the same 300 answers give an abstention precision of 0.41 with 62 "false" abstentions. Against the rationale documents, precision is 0.83 and only 18 are false.

What I don't claim

  • An nDCG@10 improvement of hybrid over dense retrieval. The difference is not significant.
  • That fusion finds more evidence. The Recall@100 gain over dense is only suggestive (sign test p = 0.065) and sits on NEI claims only.
  • Small effects either way. With 300 queries, the minimum detectable effect at 80% power is about 0.028 nDCG@10 and 0.031 Recall@100, so the null results rule out large effects, not small ones.
  • Corrected significance. The retrieval tests above are uncorrected. A Bonferroni bar at α = 0.05 would be p ≈ 0.008, which only hybrid vs BM25 clears, so I treat bge vs MiniLM (p = 0.038) as suggestive.
  • A clean held-out 0.80. The fixes behind it were chosen partly by looking at the same 300 test claims, and 8 variants have been scored on them. The generator isn't deterministic either: between two runs on the same 50 claims, correctness flipped on 7 of them, which the Wilson interval doesn't capture. Confirming 0.80 needs a fresh held-out set.
  • 0.837 as the real accuracy. The label audit is a one-sided sensitivity check from a single annotator, with no NEI claims audited.
  • A checked faithfulness score. It comes from one judge model with no second model or human labels to check it against. The judge's repeat consistency is measured, not its correctness, and that study used an earlier run's answers, not these 300.
  • That web search can replace the local index. Pooled web search reaches Recall@5 0.228 against 0.766 locally, takes many seconds per query, and moves with Semantic Scholar's and PubMed's indexes.

Decisions and trade-offs

  • Hybrid is the default, for recall. I kept fusion for its possibly fuller candidate pool, not for top-10 ordering. On this evidence that is a judgement call, not a settled result, and I report that the extra recall sits on NEI claims.
  • Reranking ships off. It stays available as mode=hybrid_rerank. The comparison also meant rewriting the project's own README, which had claimed that reranking "only pays off with a domain-appropriate model" before that had ever been run.
  • The rerank slice is capped at 32. A cross-encoder runs a full forward pass per candidate, and retrieval is serialised behind a lock, so an unbounded slice would let one client at the rate limit hold the service for minutes. The cap is a denial-of-service guard first and a latency setting second.
  • Rate limiting is keyed carefully. The API is unauthenticated, so per-IP limits (30 searches and 10 answers a minute) are its only guard. Behind a proxy it counts in from the right of X-Forwarded-For by the configured number of hops, which closed a hole where a client could choose its own rate-limit key. The web modes, where one search fans out to several Semantic Scholar and PubMed calls, add a stricter shared limit (6 a minute by default) and at most 2 concurrent web searches per process.
  • Prompt injection is handled structurally. The question is collapsed to a single line with no repeated quote runs before it goes into the prompt, and tests check that a crafted query can't forge a new question or answer turn. Web titles and abstracts are untrusted third-party text and are flattened the same way.
  • Free models by default. A full answer eval costs $0 on the default models, and the paid generators I tried were not better. The trade-off is that free models can disappear: the previous judge's free variant stopped being served, which forced a judge change and a re-run.
  • Web search stays off the default path. It is optional and slow, and even pooled it finds the gold paper far less often than the local index, so local hybrid search remains the default and the measured headline.
  • Reproducibility. The eval harness caches each configuration by a signature of everything that affects its numbers and checkpoints as it goes. PyTorch is the CPU-only build. The tests run on fakes, with no models, index or API key. An answer run that hits the free daily quota stops, and re-running the same command resumes where it left off.