Coft Eval

---
name: coft-eval
description: Evaluates the ability of retrieval-augmented language models to mitigate knowledge hallucination and maintain robustness in reading comprehension and question-answering tasks when processing long, noisy contexts with selective highlighting. Use when the user wants to benchmark on FELM, RACE-H, RACE-M, Natural Questions, TriviaQA, WebQ, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15116
  bibtex_key: lv2024coarse
  confidence: high
---

# coft-eval

> Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models — Lv et al. (2024) (arXiv:2410.15116, 2024)

## What this evaluates

Evaluates the ability of retrieval-augmented language models to mitigate knowledge hallucination and maintain robustness in reading comprehension and question-answering tasks when processing long, noisy contexts with selective highlighting.

## Datasets

- **FELM** — total ?; splits: test (-1)
- **RACE-H** — total ?; splits: test (-1)
- **RACE-M** — total ?; splits: test (-1)
- **Natural Questions** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **WebQ** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `precision` — range: [0, 1]
  - Ratio of correctly predicted positive tokens to the total predicted tokens.
- `recall` — range: [0, 1]
  - Ratio of correctly predicted positive tokens to the total actual positive tokens.
- `EM` — range: [0, 1]
  - Exact-match accuracy: binary score of 1 if the model's prediction exactly matches the ground truth string, 0 otherwise.

## Input / output format

**Input**: Context passages (optionally pre-processed with COFT highlighting), questions, and retrieved documents. All backbone models use greedy decoding with temperature=0.

**Output**: Free-text answers or responses generated by the backbone LLM.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    f1s, ems, precisions, recalls = [], [], [], []
    for p, g in zip(preds, golds):
        p_tok, g_tok = set(normalize(p)), set(normalize(g))
        if not p_tok:
            precisions.append(0.0); recalls.append(0.0); f1s.append(0.0); ems.append(0.0); continue
        tp = len(p_tok & g_tok)
        prec = tp / len(p_tok)
        rec = tp / len(g_tok)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        ems.append(1.0 if p == g else 0.0)
        precisions.append(prec); recalls.append(rec); f1s.append(f1)
    return {'f1': mean(f1s), 'precision': mean(precisions), 'recall': mean(recalls), 'em': mean(ems)}
```

## Common pitfalls

- Temperature is strictly fixed to 0 for all experiments to guarantee stable and reproducible results.
- In QA tasks, noise ratio is controlled by randomly concatenating relevant and noisy documents to mitigate position bias.
- FELM evaluation is reported separately across three specific domains: WK, Sci/Tech, and Wri/Rec.

## Evidence (verbatim from paper)

> For knowledge hallucination, we use FELM (Chen et al., [2023c]) as the benchmark with precision, recall, and F1 score as evaluation metrics (Chen et al., [2023c]). For reading comprehension, we use RACE-H (high school level reading comprehension) and RACE-M (middle school level reading comprehension) (Lai et al., [2017]) as benchmarks with precision as the metric (Bi et al., [2024]; Rae et al., [2021]). For question answering, we use Natural Question (Kwiatkowski et al., [2019]), TriviaQA (Joshi et al., [2017]), and WebQ (Berant et al., [2013]) as benchmarks with EM and F1 score as metrics (Chen et al., [2017]; Zhu et al., [2021]).

## Citation

```bibtex
@misc{lv2024coarse,
  title={Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Models},
  author={Lv et al. (2024)},
  year={2024},
  note={arXiv:2410.15116}
}
```

- arXiv: 2410.15116