Wmt22 Slt Eval

---
name: wmt22-slt-eval
description: Evaluates sign language translation from video to spoken text. It probes the model's ability to handle long videos, large vocabularies, and high singleton rates by leveraging full-body and lip-reading visual features. Use when the user wants to benchmark on WMT 2022 Shared Task, PHOENIX 2014T, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.13326
  bibtex_key: dey2022clean
  confidence: high
---

# wmt22-slt-eval

> Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation — Dey et al. (2022) (arXiv:2210.13326, 2022)

## What this evaluates

Evaluates sign language translation from video to spoken text. It probes the model's ability to handle long videos, large vocabularies, and high singleton rates by leveraging full-body and lip-reading visual features.

## Datasets

- **WMT 2022 Shared Task** — total ?; splits: dev (-1), test (-1)
- **PHOENIX 2014T** — total ?; splits: dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 1]
  - Standard n-gram precision with brevity penalty. The paper also reports 'reduced BLEU (RedB)', which filters out singleton words (appearing only once in training) to mitigate vocabulary sparsity.

## Input / output format

**Input**: Per-frame visual feature embeddings extracted from video frames (768-dim mouth patches via AV-HuBERT or full-body via I3D).

**Output**: Sequence of spoken German words.

## Scoring recipe

```python
def compute_bleu(preds, refs, filter_singletons=False):
    if filter_singletons:
        preds = [w for w in preds if w not in singletons]
        refs = [[w for w in r if w not in singletons] for r in refs]
    return standard_bleu_score(preds, refs)
```

## Common pitfalls

- Ignoring singleton words (appearing only once in training) severely degrades performance on this dataset.
- Utterance boundary markers (full stops) significantly impact BLEU scores (~1% relative difference).
- Lip-reading features improve rare word prediction but may hurt performance on common words without lexical data augmentation.

## Evidence (verbatim from paper)

> Configurations are evaluated on the WMT 2022 Dev dataset using the reduced BLEU (RedB) and standard BLEU (Stand.) score as metric.

## Citation

```bibtex
@misc{dey2022clean,
  title={Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation},
  author={Dey et al. (2022)},
  year={2022},
  note={arXiv:2210.13326}
}
```

- arXiv: 2210.13326