Quality Classification Eval

---
name: quality-classification-eval
description: Evaluates a model's ability to classify machine translation outputs as 'good' (zero HTER) or 'bad' (non-zero HTER) for practical post-editing filtering. It probes whether binary classification outperforms thresholded regression for identifying adequate translations in real-world deployment scenarios. Use when the user wants to benchmark on WMT17 QE/QC, or asks about evaluating this task. Reports R@P_t.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.03519
  bibtex_key: zhou2020practical
  confidence: high
---

# quality-classification-eval

> Practical Perspectives on Quality Estimation for Machine Translation — Zhou et al. (2020) (arXiv:2005.03519, 2020)

## What this evaluates

Evaluates a model's ability to classify machine translation outputs as 'good' (zero HTER) or 'bad' (non-zero HTER) for practical post-editing filtering. It probes whether binary classification outperforms thresholded regression for identifying adequate translations in real-world deployment scenarios.

## Datasets

- **WMT17 QE/QC** — total ?; splits: train (48000), dev (2000), test (4000)

## Metrics

- `R@P_t` **(primary)** — range: [0, 1]
  - Recall at precision above threshold t. Computes the recall achieved on the positive class when the model's precision on predicted positives is at least t. The paper evaluates at t=0.8 and t=0.9.

## Input / output format

**Input**: Source sentence and target sentence pair.

**Output**: Binary label: 'good' (positive) or 'bad' (negative).

## Scoring recipe

```python
def compute_R_at_Pt(preds, gold, t):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    if precision >= t:
        recall = tp / sum(gold) if sum(gold) > 0 else 0.0
        return recall
    return 0.0
```

## Common pitfalls

- Class imbalance is severe, as only sentences with exactly 0.0 HTER are labeled positive (9–44% positive rate depending on language pair and split).
- Thresholding a regression model's TER output to create a binary classifier yields poor precision/recall trade-offs compared to training a dedicated binary classifier from scratch.
- Performance varies significantly between language directions (En-De vs De-En) due to domain mismatch with the parallel training data (IT vs. Pharmaceutical).

## Evidence (verbatim from paper)

> QE datasets list source/target sentence pairs with HTER scores as labels; for QC we label samples with 0.0 HTER as 'good' (positive) while the rest get 'bad' (negative) labels. We have tuned hyperparameters for QC models according to the  $R@P_t$  on the development dataset by grid-search, and the final parameters we finally picked are shown in Table 2.

## Citation

```bibtex
@misc{zhou2020practical,
  title={Practical Perspectives on Quality Estimation for Machine Translation},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2005.03519}
}
```

- arXiv: 2005.03519