Finerumfact Eval

---
name: finerumfact-eval
description: Evaluates a model's ability to perform sentence-level fact verification on generated summaries and localize specific factuality error types. It measures how well the model's judgments align with human annotations across sentence, summary, and system levels. Use when the user wants to benchmark on FineSumFact, or asks about evaluating this task. Reports balanced accuracy (bAcc).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.10689
  bibtex_key: oh2024learnstoverify
  confidence: high
---

# finerumfact-eval

> Learning to Verify Summary Facts with Fine-Grained LLM Feedback — Oh et al. (2024) (arXiv:2412.10689, 2024)

## What this evaluates

Evaluates a model's ability to perform sentence-level fact verification on generated summaries and localize specific factuality error types. It measures how well the model's judgments align with human annotations across sentence, summary, and system levels.

## Datasets

- **FineSumFact** — total 26353; splits: train (25660), test (693); repo https://github.com/DISL-Lab/FineSumFact

## Metrics

- `balanced accuracy (bAcc)` **(primary)** — range: [0, 1]
  - Standard balanced accuracy for binary classification: (TP/(TP+FN) + TN/(TN+FP)) / 2. Measures sentence-level verification accuracy.
- `summary-level correlation (Pearson)` — range: [-1, 1]
  - Pearson correlation coefficient between the model's predicted summary-level factuality scores and human-provided summary-level scores.
- `system-level correlation (Rank/Spearman)` — range: [-1, 1]
  - Spearman rank correlation between the model's ranking of summarizers and the human ranking of summarizers.
- `error localization accuracy` — range: [0, 1]
  - Accuracy of predicting the correct factuality error category among seven predefined types for misclassified sentences.

## Input / output format

**Input**: A generated summary and its corresponding reference text.

**Output**: Per sentence: a binary label (factually correct/incorrect), optionally accompanied by reasoning and an error type/category.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # Binary sentence-level
    tp = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 0 and g == 0)
    bAcc = (tp / (tp + fn) + tn / (tn + fp)) / 2

    # Correlations
    pearson_r = pearsonr(preds['summ_score'], gold['summ_score'])
    spearman_r = spearmanr(preds['sys_rank'], gold['sys_rank'])

    # Error localization (only on incorrect sentences)
    err_preds = [p for p, g in zip(preds['err_type'], gold['err_type']) if g == 0]
    err_gold = [g for p, g in zip(preds['err_type'], gold['err_type']) if g == 0]
    loc_acc = sum(1 for p, g in zip(err_preds, err_gold) if p == g) / len(err_gold)

    return {'bAcc': bAcc, 'pearson': pearson_r, 'spearman': spearman_r, 'loc_acc': loc_acc}
```

## Common pitfalls

- Confusing sentence-level binary accuracy with summary-level or system-level correlation metrics, which require different aggregation strategies.
- Assuming human-annotated fine-grained error types are more reliable than LLM-generated ones; the paper notes human inter-annotator Kappa is often < 0.5 for fine-grained tasks.
- Evaluating error localization on correctly classified sentences; it should only be computed on sentences where the binary judgment is incorrect.

## Evidence (verbatim from paper)

> Metrics. We follow the widely used metrics in recent works*Song et al. ([2024]); Liu et al. ([2023])*, verifying the agreement with human in three different levels: *balanced accuracy* (bAcc), an indicator of sentence-level verification accuracy; *summary-level* correlation, an indicator of agreement with humans’ summary-level scores; *system-level* correlation, an indicator of agreement with humans’ ranking across different summarizers.

## Citation

```bibtex
@misc{oh2024learnstoverify,
  title={Learning to Verify Summary Facts with Fine-Grained LLM Feedback},
  author={Oh et al. (2024)},
  year={2024},
  note={arXiv:2412.10689}
}
```

- arXiv: 2412.10689