Punctuation Restoration Iwslt Eval

---
name: punctuation-restoration-iwslt-eval
description: Evaluates a model's ability to restore punctuation marks (commas, periods, question marks) in English text. It specifically probes robustness on both manually transcribed transcripts and ASR-generated transcripts, ignoring non-punctuation tokens during evaluation. Use when the user wants to benchmark on IWSLT2011, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.00248
  bibtex_key: yi2020adversarial
  confidence: high
---

# punctuation-restoration-iwslt-eval

> Adversarial Transfer Learning for Punctuation Restoration — Jiangyan Yi et al. (2020) (arXiv:2004.00248, 2020)

## What this evaluates

Evaluates a model's ability to restore punctuation marks (commas, periods, question marks) in English text. It specifically probes robustness on both manually transcribed transcripts and ASR-generated transcripts, ignoring non-punctuation tokens during evaluation.

## Datasets

- **IWSLT2011** — total ?; splits: train (-1), dev (-1), test_Ref (-1), test_ASR (-1)

## Metrics

- `F1-score` **(primary)** — range: percent
  - Standard F1 score calculated as 2 * (Precision * Recall) / (Precision + Recall). Precision and Recall are computed exclusively on punctuation tokens (COMMA, PERIOD, QUESTION), completely ignoring non-punctuation tokens (O).
- `Precision` — range: percent
  - Ratio of correctly predicted punctuation tokens to all predicted punctuation tokens. Computed only on COMMA, PERIOD, and QUESTION labels.
- `Recall` — range: percent
  - Ratio of correctly predicted punctuation tokens to all actual punctuation tokens. Computed only on COMMA, PERIOD, and QUESTION labels.

## Input / output format

**Input**: A sequence of English words/tokens from TED Talk transcripts.

**Output**: A sequence of token-level labels: O, COMMA, PERIOD, or QUESTION.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # Filter out non-punctuation tokens (O) as per protocol
    pred_punct = [p for p, g in zip(predictions, gold) if g != 'O']
    gold_punct = [g for g in gold if g != 'O']
    
    tp = sum(1 for p, g in zip(pred_punct, gold_punct) if p == g)
    fp = sum(1 for p, g in zip(pred_punct, gold_punct) if p != g)
    fn = sum(1 for p, g in zip(pred_punct, gold_punct) if p != g)
    
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1
```

## Common pitfalls

- Including non-punctuation tokens (label 'O') in the precision/recall/F1 calculation, which artificially inflates scores.
- Failing to distinguish between the two test sets (Ref. for manual transcripts vs. ASR for speech recognition outputs), leading to misreported results.
- Misinterpreting the label mapping: exclamation marks and semicolons are grouped under PERIOD, not treated as separate classes.

## Evidence (verbatim from paper)

> All models are evaluated in terms of precision (P), recall (R), F1-score (F1) in our experiments. We focus on the performance of the punctuation marks. So the correctly predicted non-punctuation marks O are ignored. We only evaluate the performance of COMMA, PERIOD and QUESTION on two test sets: Ref. and ASR, respectively.

## Citation

```bibtex
@misc{yi2020adversarial,
  title={Adversarial Transfer Learning for Punctuation Restoration},
  author={Jiangyan Yi et al. (2020)},
  year={2020},
  note={arXiv:2004.00248}
}
```

- arXiv: 2004.00248