Oversmoothing Rate Eval

---
name: oversmoothing-rate-eval
description: This evaluation probes the tendency of autoregressive neural machine translation models to prematurely terminate sequences by assigning high probability to short prefixes. It measures how well a model balances sequence length distribution and translation quality under beam search decoding. Use when the user wants to benchmark on IWSLT'17, WMT'16 En->De, WMT'19, or asks about evaluating this task. Reports oversmoothing_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.08914
  bibtex_key: kulikov2021oversmoothing
  confidence: high
---

# oversmoothing-rate-eval

> Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling — Kulikov et al. (2021) (arXiv:2112.08914, 2021)

## What this evaluates

This evaluation probes the tendency of autoregressive neural machine translation models to prematurely terminate sequences by assigning high probability to short prefixes. It measures how well a model balances sequence length distribution and translation quality under beam search decoding.

## Datasets

- **IWSLT'17** — total ?; splits: train (-1), val (800), test (8000)
- **WMT'16 En->De** — total ?; splits: train (4500000), val (3000), test (3000)
- **WMT'19** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `oversmoothing_rate` **(primary)** — range: other
  - Averages the log-probability and normalized rank of the `<eos>` token across all positions $t$ in generated sequences, then averages across all translations. Higher values indicate a stronger tendency to prematurely end sequences.
- `BLEU` — range: percent
  - Standard n-gram overlap metric computed via sacreBLEU on detokenized predictions and references.

## Input / output format

**Input**: Source sentence in the source language (e.g., German, French, Chinese, Russian).

**Output**: Detokenized target language translation sequence generated via beam search decoding.

## Scoring recipe

```python
# Compute BLEU
bleu_score = sacrebleu.corpus_bleu(predictions, references)

# Compute oversmoothing_rate
eos_log_probs = []
for pred in predictions:
    for t in range(1, len(pred)):
        eos_log_probs.append(model.log_prob('<eos>', prefix=pred[:t]))
oversmoothing_rate = sum(eos_log_probs) / len(eos_log_probs)
```

## Common pitfalls

- The evaluation explicitly disables length normalization and length penalty during decoding to isolate oversmoothing effects.
- Results are highly sensitive to beam search size; metrics must be reported across multiple beam configurations.
- Early stopping is triggered by the training objective function on the validation set, not by translation quality metrics.

## Evidence (verbatim from paper)

> We compute and report BLEU scores using sacreBLEU on detokenized predictions. We vary beam sizes to study their effect in-depth. We set the lower- and upper-bound of a generated translation to be, respectively, 0 and $1.2\cdot l_{x}+10$, where $l_{x}$ is the length of the source $x$. We do not use either length normalization nor length penalty, in order to study the impact of oversmoothing on decoding faithfully.

## Citation

```bibtex
@misc{kulikov2021oversmoothing,
  title={Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling},
  author={Kulikov et al. (2021)},
  year={2021},
  note={arXiv:2112.08914}
}
```

- arXiv: 2112.08914