Refact Eval

---
name: refact-eval
description: This benchmark evaluates large language models' ability to detect, localize, and correct scientific confabulations in generated answers. It probes fine-grained factuality awareness, span-level error identification, and factual restoration capabilities under domain-specific scrutiny. Use when the user wants to benchmark on ReFACT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25868
  bibtex_key: wang2025refact
  confidence: high
---

# refact-eval

> ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations — Wang et al. (2025) (arXiv:2509.25868, 2025)

## What this evaluates

This benchmark evaluates large language models' ability to detect, localize, and correct scientific confabulations in generated answers. It probes fine-grained factuality awareness, span-level error identification, and factual restoration capabilities under domain-specific scrutiny.

## Datasets

- **ReFACT** — total 1001; splits: test (1001); repo https://github.com/ddz5431/ReFACT

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions across judgment, localization, and correction tasks. Reported separately for factual and confabulated instances to capture asymmetric error patterns.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for confabulation detection, computed separately for factual and confabulated classes.
- `IoU` — range: [0, 1]
  - Intersection over Union between predicted confabulated spans and gold annotations, measuring boundary accuracy and coverage completeness for multi-span localization.
- `Exact Match (EM)` — range: [0, 1]
  - Binary metric indicating whether the model's predicted correction perfectly matches the original factual entity span.

## Input / output format

**Input**: Per instance: a question paired with one or two answer versions (factual and confabulated). For localization/correction tasks, the model receives the question and the confabulated/transformed answer containing altered entities or negations. All tasks use zero-shot prompting with task-specific templates.

**Output**: Per instance: a binary classification (confabulated vs. factual), a selection of the confabulated answer, a list of identified spans/sentences containing errors, or the corrected factual entity span.

## Scoring recipe

```python
def score_judgment(pred_label, gold_label):
    return int(pred_label == gold_label)

def score_localization(pred_spans, gold_spans):
    intersection = len(set(pred_spans) & set(gold_spans))
    union = len(set(pred_spans) | set(gold_spans))
    iou = intersection / union if union > 0 else 0.0
    acc = 1.0 if intersection > 0 else 0.0
    return acc, iou

def score_correction(pred_entity, gold_entity):
    return int(pred_entity == gold_entity)
```

## Common pitfalls

- Using BERTScore for span localization or correction yields deceptively high similarity scores (often >85%) because factual and confabulated entities are semantically/syntactically similar, making it unsuitable for fine-grained factuality checks.
- Evaluating localization or correction only after successful judgment causes compounding errors; the protocol requires independent evaluation of each stage to accurately attribute failure modes.
- Assuming comparative judgment is easier than independent judgment; models actually perform worse on comparative tasks due to shallow heuristics when both answers appear plausible.

## Evidence (verbatim from paper)

> For localization, we report the Intersection over Union (IoU) between the predicted spans and the gold annotations. Unlike single-entity classification tasks, our setting often involves multiple or distributed spans, making token-level precision and recall insufficient. We do not report BERTScore, as the original and confabulated entities are often semantically and syntactically similar which leads to deceptively high similarity scores (87%) even in clearly incorrect outputs. This makes BERTScore ill-suited for evaluating fine-grained confabulation localization.

## Citation

```bibtex
@misc{wang2025refact,
  title={ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.25868}
}
```

- arXiv: 2509.25868