Bionli 300 Eval

---
name: bionli-300-eval
description: This evaluation probes a model's ability to verify scientific claims against provided or retrieved evidence in a binary classification setting. It measures performance on Supported vs. Refuted labels, testing factual grounding, uncertainty calibration, and the impact of atomic decomposition and web corroboration. Use when the user wants to benchmark on BIONLI-300, or asks about evaluating this task. Reports Balanced Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11036
  bibtex_key: [REDACTED]
  confidence: high
---

# bionli-300-eval

> Uncertainty-Aware Web-Conditioned Scientific Fact-Checking — Vinod et al. (2026) (arXiv:2604.11036, 2026)

## What this evaluates

This evaluation probes a model's ability to verify scientific claims against provided or retrieved evidence in a binary classification setting. It measures performance on Supported vs. Refuted labels, testing factual grounding, uncertainty calibration, and the impact of atomic decomposition and web corroboration.

## Datasets

- **BIONLI-300** — total ?; splits: test (-1)

## Metrics

- `Balanced Accuracy` **(primary)** — range: percent
  - Average of recall obtained on each class: (TPR + TNR) / 2. Computed for the Supported and Refuted classes.
- `F1` — range: percent
  - Harmonic mean of precision and recall for the Supported class: 2 * (Precision * Recall) / (Precision + Recall).
- `Recall` — range: percent
  - True positive rate for the Supported class: TP / (TP + FN).

## Input / output format

**Input**: Scientific claim paired with context (local evidence snippets or retrieved web documents).

**Output**: A single verification label: 'Supported' or 'Refuted'.

## Scoring recipe

```python
def compute_bionli_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Supported')
    fn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Supported')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Refuted')
    tn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Refuted')
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    bal_acc = (tpr + tnr) / 2
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * (prec * tpr) / (prec + tpr) if (prec + tpr) > 0 else 0
    return bal_acc, f1, tpr
```

## Common pitfalls

- Confusing the 2-way (Supported/Refuted) and 3-way (Supported/Refuted/NEI) evaluation regimes, as different datasets use different label sets.
- Reporting standard accuracy instead of Balanced Accuracy, which fails to account for class imbalance in datasets like CLIMATE-FEVER.
- Aggregating per-atom predictions using simple majority vote instead of the calibrated judge, which causes a ~14.6 F1 drop on BIONLI-300.

## Evidence (verbatim from paper)

> Table 2. Main results. Balanced Accuracy, Recall, and F1 on BIONLI-300; Macro-F1 on PubMedFact1k; and Balanced Accuracy, Recall (Supports), and F1 (Supports) on CLIMATE-FEVER, evaluated only on the Supported/Refuted subsets.

## Citation

```bibtex
@misc{vinod2026uncertainty,
  title={Uncertainty-Aware Web-Conditioned Scientific Fact-Checking},
  author={Vinod et al. (2026)},
  year={2026},
  note={arXiv:2604.11036}
}
```

- arXiv: 2604.11036