Comtail Translation Metric

---
name: comtail-translation-metric
description: Evaluates the ability of neural models to predict human-assigned translation quality scores for Indian language pairs, measuring alignment with crowd-sourced DA+SQM ratings. Use when the user has predictions and gold and needs to compute Pearson correlation, Spearman correlation.
metadata:
  skill_kind: metric
  source_arxiv: 2509.17667
  bibtex_key: ahsan2025comtail
  confidence: medium
---

# comtail-translation-metric

> Crosslingual Optimized Metric for Translation Assessment of Indian Languages — Arafat Ahsan et al. (2025) (arXiv:2509.17667, 2025)

## What this evaluates

Evaluates the ability of neural models to predict human-assigned translation quality scores for Indian language pairs, measuring alignment with crowd-sourced DA+SQM ratings.

## Datasets

- **COMTAIL** — total 221941; splits: train (199737), dev (9815), test (9887); repo https://github.com/aenaliph/COMTAIL

## Metrics

- `Pearson correlation, Spearman correlation` **(primary)** — range: [-1, 1]
  - Standard Pearson and Spearman rank correlation coefficients computed between model-predicted quality scores and human-annotated DA+SQM ratings. Ranges from -1 to 1, where 1 indicates perfect agreement.

## Input / output format

**Input**: Source sentence, target/hypothesis sentence, and optionally a reference sentence (for reference-based models).

**Output**: A single continuous quality score (normalized to [0, 1] or on a 0–100 scale depending on the model variant).

## Scoring recipe

```python
def compute_metrics(predictions, human_scores):
    pearson = pearsonr(predictions, human_scores)
    spearman = spearmanr(predictions, human_scores)
    return {'pearson': pearson, 'spearman': spearman}
```

## Common pitfalls

- Source sentences may overlap across train/dev/test splits, potentially causing data leakage.
- Low-resource languages (Kashmiri, Sindhi, Dogri) were excluded from the discernment rater filter due to minimal quality differences in raw data.
- Normalization differs between COMTAIL (custom min-max on z-scores) and WMT datasets (default scikit-learn MinMaxScaler).

## Evidence (verbatim from paper)

> We create train, dev, test splits on this data in ratio of 0.9, 0.05, 0.05. We group the data by source and target language pairs and randomly sample for each split. Since the data vary greatly across languages pairs this ensures proportional representation in each split. The entire data described in Table [5] are the training, development, and test sets used in building all subsequent models.

## Citation

```bibtex
@misc{ahsan2025comtail,
  title={Crosslingual Optimized Metric for Translation Assessment of Indian Languages},
  author={Arafat Ahsan et al. (2025)},
  year={2025},
  note={arXiv:2509.17667}
}
```

- arXiv: 2509.17667