Why Different AI Benchmarks Report Different Hallucination Rates

Why reported hallucination rates can span single digits to over half

The data suggests reported hallucination rates vary wildly across published benchmark results and vendor reports. In practice you will see numbers that range from low single digits up to 50% or more, depending on the test and how hallucination is defined. Closed-book factual QA benchmarks commonly report mid-teens rates under one set of scoring rules, while retrieval-augmented pipelines can show both much lower or much higher rates depending on retrieval quality and annotation policy.

Why so wide a spread? Analysis reveals multiple layers of methodological choice that move a measured rate dramatically: task framing, the gold standard used, automated metrics versus human raters, the version of the model and its decoding settings, the retrieval stack in RAG tests, and even how strict annotators are about partial credit.

Which concrete numbers should you trust? Ask: what exact model (for example, GPT-4 evaluated in mid-2024 or Llama 2 70B evaluated in 2023), what test date, what prompt templates, what retrieval components were used, and how was 'hallucination' operationalized. Without that metadata, a headline percentage is close to meaningless.

6 Critical Factors Behind Conflicting Hallucination Rates in AI Benchmarks

Below are the main drivers that cause two benchmark reports to disagree even when they test the same base model.

https://fire2020.org/why-the-facts-benchmark-rated-gemini-3-pro-at-68-8-for-factuality/ Definition of hallucination - Is hallucination any unsupported assertion, or only positively false statements? Some benchmarks label an answer as hallucinated if it lacks a cited source (strict), others allow unsupported but plausibly true answers (lenient). Dataset and task design - Open-domain QA, fact verification, summarization, and instruction-following exercises exercise different failure modes. Closed-book tasks stress internal model knowledge; RAG tasks stress retrieval alignment. Gold evidence quality and granularity - Is the gold evidence a single sentence, a document, or a curated multi-document proof? The more complete the gold set, the easier it is for an answer to be labeled "supported." Annotation policy and human rater variability - Inter-annotator agreement for factuality is often low. Are annotators allowed to give partial credit for partially-correct answers? What constitutes an "error" at the entity/date/number level? Evaluation method and metrics - Automatic metrics like ROUGE, BLEU, BERTScore, or entailment classifiers generate different judgments than crowdsourced human checks. Which threshold and which classifier model are used for automated scoring matters. Pipeline and model versioning - Small changes in temperature, top-p, decoding strategy, or a minor model update (for example, a June 2024 checkpoint vs a March 2024 checkpoint) can change hallucination counts. RAG stacks add more variability: embedding model, vector index freshness, chunking strategy, reranker, and passage selection rules.

Why FACTS and Vectara produce different hallucination profiles for the same model

Which differences are most actionable when you compare a public benchmark like FACTS with a vendor or product benchmark such as Vectara's internal tests? Evidence indicates three categories dominate the divergence.

1. Task framing and scope

FACTS-type benchmarks often focus on claim-level fact checking: a short claim is paired with targeted evidence and a verdict (supported/refuted/insufficient). Vectara-style tests that measure a retrieval-backed assistant may evaluate longer multi-sentence answers where grounding and fluency are also scored. Which is harder? It depends. Claim-level checks are strict and binary, so they can inflate hallucination rates when models make minor factual slips. Conversational RAG tests can underreport hallucination if graders prioritize helpfulness over factual strictness.

2. Retrieval architecture and index freshness

In Vectara-style experiments, the vector search, embedding model, and reranker determine what evidence is available to the model. If the index returns irrelevant snippets, the model often hallucinates by projecting plausible but unsupported facts. Conversely, FACTS tests typically assume curated evidence exists in the benchmark. Analysis reveals retrieval failure is a major source of "apparent hallucination" in production-like tests, not fundamental model invention.

3. Annotation and scoring policies

FACTS often uses a strict ground-truth oriented rubric where only exact support counts. Vectara reports may use hybrid scoring that accounts for partial support, citation presence, and user-facing appropriateness. Evidence indicates stricter scorings report higher hallucination rates; looser scorings lower them. That explains many headline disagreements between benchmarks.

Examples and concrete contrasts

When FACTS asks: "Is the claim X true?" and supplies a ground-truth passage, a model that omits a subtle date will be marked wrong. Vectara may rate the same output as acceptable if the answer cites a plausible source and the user intent is satisfied. When RAG is measured with fresh web indexes, some models show a dramatic drop in hallucination for time-sensitive topics. If a benchmark uses a stale index, hallucination rises sharply.

The data suggests differences in pipeline, not just model capability, explain a large share of variance between benchmark reports.

How specific methodological choices flip measured outcomes - evidence and examples

Let's dig into measurable cases. Which methodological switches move a hallucination rate from 12% to 28%? Below are repeatable examples with expected direction of change.

Scoring threshold - strict vs partial credit

Strict exact-match or strict evidence requirement will raise measured hallucination. Allowing partial credit for correct entity but wrong qualifier lowers it. Evidence indicates switching to partial-credit scoring can reduce the reported rate by half on many benchmarks.

Automated entailment classifier vs human raters

Using an NLI model to decide support will produce systematic biases linked to that model's training. For example, FactCC-style classifiers tend to penalize paraphrases that humans accept. When teams swap human checks for automated models to scale evaluation, reported hallucination changes non-randomly.

Retrieval depth and reranking

RAG experiments that retrieve top-1 passage versus top-10 with reranking show different hallucination profiles. Better rerankers reduce hallucination by improving grounding options; shallow retrieval increases it.

Model temperature and decoding

Higher temperature increases factual invention. Deterministic decoding (beam search, greedy) reduces novel but sometimes leads to overly safe answers. The same model checkpoint can thus report different rates simply by changing decoding.

Sampling variability and confidence intervals

Small test sets produce unstable estimates. Bootstrapped 95% confidence intervals often span several percentage points; ignoring them makes comparisons misleading. Analysis reveals many reports publish point estimates without uncertainty, which exaggerates the appearance of differences.

Questions to ask before trusting a headline: Was the evaluation automated or human? Were annotator guidelines published? Did they report confidence intervals and sample sizes?

What system builders know about interpreting conflicting hallucination numbers

What should you conclude when vendor A reports 4% hallucination and benchmark B reports 30% on the same model? The short answer: look for methodological gaps rather than assuming one number is the absolute truth.

Key synthesis points:

The data suggests many divergences are driven by the evaluation pipeline - not purely by model capability. Comparisons are only meaningful when the same test set, the same evidence pool, identical prompt templates, and identical hyperparameters are used. Vendor reports often measure production configurations (retrieval, UI constraints, answer pruning) while academic benchmarks isolate intrinsic model factuality. Both views are valid, but they answer different operational questions.

Which is more relevant for production? If you deploy an assistant with retrieval and a custom UI, vendor-style RAG evaluations are closer to reality. If you aim to measure core model knowledge, closed-book benchmarks like FACTS-type tests matter grok AI hallucination rate more. Can you have both? Yes - report both intrinsic and extrinsic factuality with consistent documentation.

7 Proven, measurable steps to tighten hallucination measurement in your stack

Below are concrete steps you can implement immediately to make benchmark comparisons meaningful and repeatable. Each step includes measurable outputs you should publish.

Publish exact artifacts - Release the test set, prompt templates, model version hash, decoding settings (temperature, top-p, max tokens), and the date of the run. Measurable output: full run manifest (JSON) for each experiment. Standardize the hallucination taxonomy - Use categories (fabricated entity, incorrect number/date, unsupported inference) and require annotators to tag error types. Measurable output: error-type distribution per run. Dual scoring: automated plus human spot-check - Run an NLI-based entailment check and then human-verify a statistically significant sample (e.g., 1,000 items). Measurable output: automated score, human-corrected score, and discrepancy rate. Fix the retrieval stack in RAG tests - Publish embedding model, index snapshot, chunking rules, and reranker config. Measurable output: retrieval precision@k and MRR reported alongside hallucination. Report uncertainty - Use bootstrap sampling to report 95% confidence intervals for hallucination rates. Measurable output: point estimate plus CI. Include adversarial and temporal slices - Run adversarial queries and time-sensitive prompts to surface worst-case behaviors. Measurable output: hallucination rate on adversarial slice and on time-sensitive slice. Automate provenance and citation checks - Require the system to return source spans and a provenance score; measure the link between provenance score and factuality. Measurable output: correlation coefficient between provenance and human-verified factuality.

Advanced techniques you can add: constrained decoding for entity slots, contrastive decoding to reduce unsupported assertions, post-hoc fact verification using a separate verifier model, and knowledge editing for persistent factual corrections.

Comprehensive summary and a final checklist

Summary of the main insights and a short checklist to run a defensible comparison.

Core insights

The data suggests differences in reported hallucination rates are often methodological, not purely model-level. Major sources of disagreement: hallucination definition, dataset design, evidence availability, annotation policy, evaluation metric, retrieval pipeline, and model-versioning. FACTS-style and Vectara-style evaluations answer different operational questions: one isolates intrinsic knowledge, the other tests production grounding and retrieval. Comparisons without exhaustive metadata, confidence intervals, and identical pipelines are unreliable.

Quick checklist for running trustworthy comparisons

Publish exact model checkpoint, date, and decoding settings. Release prompts, rubric, and gold evidence or index snapshot. Report point estimates with 95% CIs using bootstrap resampling. Tag error types and produce per-type rates (entity, date, inference, omission). Include both automated scores and human-verification subsample results. For RAG, publish retrieval precision@k, MRR, and index freshness. Run adversarial and time-sensitive slices and disclose those results.

Final question: do you want a one-page template to capture every variable that must be fixed to compare two runs? I can produce a machine-readable manifest spec (JSON schema) and a scoring rubric you can use to align internal and vendor benchmarks. That step will stop most of the noise and let you focus on actual model differences, not evaluation artifacts.

Edit

Pub: 05 Mar 2026 08:08 UTC

Views: 4