Cti Plausibility Eval

---
name: cti-plausibility-eval
description: Evaluates whether neural machine translation models correctly rely on contextual cues when generating target tokens. It compares model-extracted cue-target pairs against human-annotated discourse-level expectations to measure the plausibility of context reliance. Use when the user wants to benchmark on SCAT+, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.01188
  bibtex_key: sarti2023quantifying
  confidence: medium
---

# cti-plausibility-eval

> Quantifying the Plausibility of Context Reliance in Neural Machine Translation — Sarti et al. (2023) (arXiv:2310.01188, 2023)

## What this evaluates

Evaluates whether neural machine translation models correctly rely on contextual cues when generating target tokens. It compares model-extracted cue-target pairs against human-annotated discourse-level expectations to measure the plausibility of context reliance.

## Datasets

- **SCAT+** — total ?; splits: test (-1)

## Metrics

- `Macro F1` **(primary)** — range: [0, 1]
  - Standard macro-averaged F1 score computed over binary predictions of context-sensitive tokens versus gold SCAT+ labels.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve, measuring the trade-off between precision and recall across different thresholds for identifying context-dependent tokens.

## Input / output format

**Input**: Source context (Cx), target context (Cy), source sentence (x), and target sentence (y) for machine translation instances.

**Output**: Binary predictions or scores indicating whether specific target tokens are context-sensitive, aligned with gold SCAT+ annotations.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return {"macro_f1": f1, "auprc": compute_auprc(gold_labels, predictions)}
```

## Common pitfalls

- SCAT+ annotations only cover gender disambiguation for anaphoric pronouns, missing other context dependencies like formality cohesion.
- The evaluation is explicitly noted as a lower bound because it is restricted to only two phenomena (anaphora resolution and lexical choice) available in the dataset.

## Evidence (verbatim from paper)

> CTI Results Figure 5 and Figure 6 present the CTI plausibility of all tested models for the Macro F1 and AUPRC metrics, similarly to Figure 3 in the main analysis. This suggests our evaluation of CTI metrics plausibility can be considered a lower bound, as it is restricted to the two phenomena available in the datasets we used (anaphora resolution and lexical choice).

## Citation

```bibtex
@misc{sarti2023quantifying,
  title={Quantifying the Plausibility of Context Reliance in Neural Machine Translation},
  author={Sarti et al. (2023)},
  year={2023},
  note={arXiv:2310.01188}
}
```

- arXiv: 2310.01188