Pubmedfact1k Eval

---
name: pubmedfact1k-eval
description: This evaluation probes a model's ability to verify scientific claims in a three-way classification setting that includes uncertainty abstention. It measures performance on Supported, Refuted, and NEI (Not Enough Information) labels, testing the model's capacity to avoid overconfident predictions when evidence is insufficient or conflicting. Use when the user wants to benchmark on PubMedFact1k, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11036
  bibtex_key: [REDACTED]
  confidence: high
---

# pubmedfact1k-eval

> Uncertainty-Aware Web-Conditioned Scientific Fact-Checking — Vinod et al. (2026) (arXiv:2604.11036, 2026)

## What this evaluates

This evaluation probes a model's ability to verify scientific claims in a three-way classification setting that includes uncertainty abstention. It measures performance on Supported, Refuted, and NEI (Not Enough Information) labels, testing the model's capacity to avoid overconfident predictions when evidence is insufficient or conflicting.

## Datasets

- **PubMedFact1k** — total ?; splits: test (-1)

## Metrics

- `Macro F1` **(primary)** — range: percent
  - Unweighted mean of F1 scores computed independently for each class (Supported, Refuted, NEI).

## Input / output format

**Input**: Scientific claim paired with context (local evidence snippets or retrieved web documents).

**Output**: A single verification label: 'Supported', 'Refuted', or 'NEI'.

## Scoring recipe

```python
def compute_pubmedfact_metrics(preds, golds):
    classes = ['Supported', 'Refuted', 'NEI']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(preds, golds) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(preds, golds) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Confusing the 2-way (Supported/Refuted) and 3-way (Supported/Refuted/NEI) evaluation regimes, as different datasets use different label sets.
- Reporting standard accuracy instead of Balanced Accuracy, which fails to account for class imbalance in datasets like CLIMATE-FEVER.
- Aggregating per-atom predictions using simple majority vote instead of the calibrated judge, which causes a ~14.6 F1 drop on BIONLI-300.

## Evidence (verbatim from paper)

> Table 2. Main results. Balanced Accuracy, Recall, and F1 on BIONLI-300; Macro-F1 on PubMedFact1k; and Balanced Accuracy, Recall (Supports), and F1 (Supports) on CLIMATE-FEVER, evaluated only on the Supported/Refuted subsets.

## Citation

```bibtex
@misc{vinod2026uncertainty,
  title={Uncertainty-Aware Web-Conditioned Scientific Fact-Checking},
  author={Vinod et al. (2026)},
  year={2026},
  note={arXiv:2604.11036}
}
```

- arXiv: 2604.11036