Coherence Modeling Eval

---
name: coherence-modeling-eval
description: Evaluates whether neural coherence models can distinguish coherent text from artificially incoherent permutations and whether their scores correlate with human judgments on real-world downstream tasks like machine translation and summarization. Use when the user wants to benchmark on WSJ, WMT2017-2018, CNN/DM, DUC 2003, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.14626
  bibtex_key: mohiuddin2020rethinking
  confidence: high
---

# coherence-modeling-eval

> Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks — Mohiuddin et al. (2020) (arXiv:2004.14626, 2020)

## What this evaluates

Evaluates whether neural coherence models can distinguish coherent text from artificially incoherent permutations and whether their scores correlate with human judgments on real-world downstream tasks like machine translation and summarization.

## Datasets

- **WSJ** — total 2431; splits: train (1378), test (1053); repo https://github.com/taasnim/unified-coherence-model
- **WMT2017-2018** — total 20680; splits: test (20680)
- **CNN/DM** — total ?; splits: test (-1)
- **DUC 2003** — total 16; splits: test (16)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct binary classifications (coherent vs. incoherent) out of total test instances.
- `AC1 agreement` — range: [0, 1]
  - Pairwise ranking agreement between model-generated coherence scores and human annotator rankings, computed using the AC1 metric.

## Input / output format

**Input**: Document or summary text presented as pairs (original vs. permuted/incoherent) or sets (reference vs. multiple system outputs) for scoring or ranking.

**Output**: A continuous coherence score per document, or a binary label (coherent/incoherent), or a ranked list of candidates based on scores.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_ac1_agreement(model_scores, human_scores):
    model_ranks = rank_pairs(model_scores)
    human_ranks = rank_pairs(human_scores)
    return calculate_ac1(model_ranks, human_ranks)
```

## Common pitfalls

- High accuracy on synthetic permutation tasks does not transfer to downstream human judgment tasks.
- Models trained on global discrimination fail to capture local coherence changes, leading to poor performance on local discrimination.
- Agreement metrics (AC1) measure ranking correlation with humans, not direct classification accuracy.

## Evidence (verbatim from paper)

> We report the accuracy of the coherence models trained on the global discrimination task in distinguishing the more coherent reference text from the less coherent system translations in Table 5. We can see that most models perform worse than a random baseline of 50%, showing that their training on the global discrimination task is not helpful in detecting coherence quality in MT text.

## Citation

```bibtex
@misc{mohiuddin2020rethinking,
  title={Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks},
  author={Mohiuddin et al. (2020)},
  year={2020},
  note={arXiv:2004.14626}
}
```

- arXiv: 2004.14626