Wmt Slt 22 Eval

---
name: wmt-slt-22-eval
description: Evaluates the capability of sign language to text translation systems on low-resource datasets. It measures how accurately a model can convert 3D pose sequences of sign language into corresponding spoken language text. Use when the user wants to benchmark on FocusNews, SRF, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.01140
  bibtex_key: tarres2022tackling
  confidence: high
---

# wmt-slt-22-eval

> Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22 — Tarrés et al. (2022) (arXiv:2212.01140, 2022)

## What this evaluates

Evaluates the capability of sign language to text translation systems on low-resource datasets. It measures how accurately a model can convert 3D pose sequences of sign language into corresponding spoken language text.

## Datasets

- **FocusNews** — total ?; splits: dev (-1), test (-1)
- **SRF** — total ?; splits: dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 1]
  - Standard machine translation metric computing geometric mean of n-gram precisions (1-4) with sentence-level smoothing, normalized to [0, 1].

## Input / output format

**Input**: 3D MediaPipe keypoint sequences representing sign language videos, preprocessed to 25fps.

**Output**: Natural language text sentences corresponding to the signed utterance.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    scores = []
    for pred, refs in zip(predictions, references):
        score = sentence_bleu([refs], pred, smoothing_function=smoothing)
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Dev set performance does not correlate with official test set rankings (e.g., best dev model failed on test).
- BLEU optimization may not align with human perception; chrF++ correlates better with human rankings but yields lower scores.
- Low-resource nature leads to fundamentally poor BLEU scores despite architectural improvements.

## Evidence (verbatim from paper)

> We optimized our systems to obtain the best BLEU metric, without taking other metrics into consideration. However, organizers also compute chrF++ (Popovic, 2017) and BLEURT (Sellam et al., 2020) metrics (Müller et al., 2022). We find that the BLEURT score shows a similar performance than BLEU. However, for the chrF++ metric, which correlates better with respect to human relative rankings, our models score lower compared to other submissions.

## Citation

```bibtex
@misc{tarres2022tackling,
  title={Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22},
  author={Tarrés et al. (2022)},
  year={2022},
  note={arXiv:2212.01140}
}
```

- arXiv: 2212.01140