Character Level F1

---
name: character-level-f1
description: This evaluation probes an LLM-based autorater's ability to predict fine-grained machine translation errors (spans, severities, categories) without using human references. It specifically tests how well the model can specialize to a given test set by leveraging in-context examples of human ratings from other systems on the same inputs. Use when the user has predictions and gold and needs to compute character-level F1.
metadata:
  skill_kind: metric
  source_arxiv: 2411.15387
  bibtex_key: finkelstein2024specializing
  confidence: high
---

# character-level-f1

> From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set — Finkelstein et al. (2024) (arXiv:2411.15387, 2024)

## What this evaluates

This evaluation probes an LLM-based autorater's ability to predict fine-grained machine translation errors (spans, severities, categories) without using human references. It specifically tests how well the model can specialize to a given test set by leveraging in-context examples of human ratings from other systems on the same inputs.

## Datasets

- **WMT'23 and WMT'24 MQM datasets** — total ?; splits: test (-1)

## Metrics

- `character-level F1` **(primary)** — range: [0, 1]
  - Standard character-level precision, recall, and F1 for error span prediction. Partial credit of 0.5 is awarded if a character is correctly identified as an error but the predicted severity is incorrect.
- `Acc23` — range: [0, 1]
  - Segment-level pairwise accuracy with tie calibration, rewarding correct ranking of translations and correct tie prediction.

## Input / output format

**Input**: Source sentence, hypothesis translation, and a set of in-context learning examples containing other translations from the same test set paired with their human MQM ratings (error spans, severity, category). No human reference is provided.

**Output**: JSON object containing a list of predicted errors, where each error includes "span" (character indices), "severity", and "category" fields.

## Scoring recipe

```python
def compute_char_f1(gold_spans, pred_spans):
    gold_chars = set()
    for s in gold_spans: gold_chars.update(range(s.start, s.end))
    pred_chars = set()
    for s in pred_spans: pred_chars.update(range(s.start, s.end))
    tp = len(gold_chars & pred_chars)
    fp = len(pred_chars - gold_chars)
    fn = len(gold_chars - pred_chars)
    sev_mismatch = sum(0.5 * (s.end - s.start) for g, p in zip(gold_spans, pred_spans) if g.severity != p.severity and g.start == p.start and g.end == p.end)
    prec = (tp + sev_mismatch) / (tp + fp + sev_mismatch) if (tp + fp + sev_mismatch) > 0 else 0
    rec = (tp + sev_mismatch) / (tp + fn + sev_mismatch) if (tp + fn + sev_mismatch) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Including the human reference in the input violates the reference-free (QE) constraint of the metric.
- Failing to use hold-one-out prompting when constructing ICL examples causes data leakage by including the target system's own ratings.
- Shuffling ICL examples across different source sentences or raters breaks the pseudo-SxS constraint, significantly degrading performance.

## Evidence (verbatim from paper)

> To meta-evaluate the quality of Specialist AutoMQM, we compute the character-level precision, recall, and F1 span tagging evaluation metrics (used by the WMT’23 QE Shared Task; Blain et al. (2023)). Given gold and predicted ratings, these metrics represent the precision, recall, and F1 of predicting whether a character in the hypothesis translation is included in an error span or not. Partial credit of 0.5 is given if the predicted rating marks a character as an error but predicts the incorrect severity.

## Citation

```bibtex
@misc{finkelstein2024specializing,
  title={From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set},
  author={Finkelstein et al. (2024)},
  year={2024},
  note={arXiv:2411.15387}
}
```

- arXiv: 2411.15387