Coqa Eval

---
name: coqa-eval
description: Evaluates a model's ability to answer free-form questions in a multi-turn conversational setting. It probes coreference resolution, pragmatic reasoning, and the capacity to maintain and leverage dialogue history over a given context passage. Use when the user wants to benchmark on CoQA, or asks about evaluating this task. Reports macro-average F1 score of word overlap.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.07042
  bibtex_key: reddy2018coqa
  confidence: high
---

# coqa-eval

> CoQA: A Conversational Question Answering Challenge — Reddy et al. (2018) (arXiv:1808.07042, 2018)

## What this evaluates

Evaluates a model's ability to answer free-form questions in a multi-turn conversational setting. It probes coreference resolution, pragmatic reasoning, and the capacity to maintain and leverage dialogue history over a given context passage.

## Datasets

- **CoQA** — total ?; splits: dev (-1), test (-1)

## Metrics

- `macro-average F1 score of word overlap` **(primary)** — range: [0, 1]
  - Word overlap F1 between the predicted answer and each of the n=4 gold answers. Articles (a, an, the) and punctuation are stripped before comparison. For each question, the maximum F1 across the n gold answers is taken, and these maximums are averaged across all questions.

## Input / output format

**Input**: A context passage, the conversation history (previous question-answer pairs), and the current question.

**Output**: A free-form text answer (and optionally a rationale span from the passage).

## Scoring recipe

```python
def normalize(text):
    return re.sub(r'\b(a|an|the)\b', '', text).replace('.', '').replace(',', '')
def compute_metric(predictions, golds):
    f1_scores = []
    for pred, gold_list in zip(predictions, golds):
        pred_n = normalize(pred)
        max_f1 = 0
        for g in gold_list:
            max_f1 = max(max_f1, f1_score(pred_n, normalize(g)))
        f1_scores.append(max_f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Forgetting to strip articles ('a', 'an', 'the') and punctuation during answer normalization.
- Averaging F1 against only the first gold answer instead of taking the maximum F1 across all 4 gold answers per question.
- Treating the task as strict span extraction; the dataset requires free-form generation and the metric measures lexical overlap, not semantic equivalence.

## Evidence (verbatim from paper)

> Following SQuAD, we use macro-average F1 score of word overlap as our main evaluation metric. We use the gold answers of history to predict the next answer. In SQuAD, for computing a model's performance, each individual prediction is compared against $n$ human answers resulting in $n$ F1 scores, the maximum of which is chosen as the prediction's F1. For each question, we average out F1 across these $n$ sets, both for humans and models. In our final evaluation, we use $n = 4$ human answers for every question (the original answer and 3 additionally collected answers). The articles $a$, $an$ and the and punctuations are excluded in evaluation.

## Citation

```bibtex
@misc{reddy2018coqa,
  title={CoQA: A Conversational Question Answering Challenge},
  author={Reddy et al. (2018)},
  year={2018},
  note={arXiv:1808.07042}
}
```

- arXiv: 1808.07042