Concurrence Eval

---
name: concurrence-eval
description: Measures how consistently a modeling approach's performance ranking holds across different question answering benchmarks. It probes whether improvements in QA models generalize across datasets with varying data collection procedures, passage/question distributions, and targeted linguistic phenomena. Use when the user wants to benchmark on SQuAD, NewsQA, NaturalQuestions, DROP, HotpotQA, QAMR, or asks about evaluating this task. Reports concurrence (Spearman's τ).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.01065
  bibtex_key: liu2021concurrence
  confidence: high
---

# concurrence-eval

> Do Question Answering Modeling Improvements Hold Across Benchmarks? — Liu et al. (2021) (arXiv:2102.01065, 2021)

## What this evaluates

Measures how consistently a modeling approach's performance ranking holds across different question answering benchmarks. It probes whether improvements in QA models generalize across datasets with varying data collection procedures, passage/question distributions, and targeted linguistic phenomena.

## Datasets

- **SQuAD** — total ?; splits: test (-1)
- **NewsQA** — total ?; splits: test (-1)
- **NaturalQuestions** — total ?; splits: test (-1)
- **DROP** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **QAMR** — total ?; splits: test (-1)

## Metrics

- `concurrence (Spearman's τ)` **(primary)** — range: [-1, 1]
  - Spearman's rank correlation coefficient computed over the performance scores of a fixed set of models evaluated on two different benchmarks.

## Input / output format

**Input**: Performance scores of a fixed set of QA models on two distinct benchmarks.

**Output**: A single Spearman's rank correlation coefficient (τ) quantifying the agreement between the two benchmarks' model rankings.

## Scoring recipe

```python
def compute_concurrence(scores_A, scores_B):
    models = list(scores_A.keys())
    assert models == list(scores_B.keys())
    vals_A = [scores_A[m] for m in models]
    vals_B = [scores_B[m] for m in models]
    ranks_A = rankdata(vals_A)
    ranks_B = rankdata(vals_B)
    return spearmanr(ranks_A, ranks_B).correlation
```

## Common pitfalls

- Concurrence evaluates the correlation of model rankings, not absolute score differences or dataset overlap.
- Requires the exact same model suite to be evaluated on both benchmarks; missing models break the correlation calculation.
- The paper uses MRQA-formatted versions of NewsQA, NQ, DROP, and HotpotQA, which differ from their original releases (e.g., DROP subset excludes non-extractive answers).

## Evidence (verbatim from paper)

> We study the concurrence between six human-constructed benchmarks: SQuAD, NewsQA, NaturalQuestions, DROP (Dua et al., 2019), HotpotQA (Yang et al., 2018), and QAMR (Michael et al., 2018). Despite differences in benchmark crowdsourcing setups, passage and questions distributions, and even linguistic phenomena of interest, modeling improvements generally hold across human-constructed benchmarks (Table 1).

## Citation

```bibtex
@misc{liu2021concurrence,
  title={Do Question Answering Modeling Improvements Hold Across Benchmarks?},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2102.01065}
}
```

- arXiv: 2102.01065