Finsquad Eval

---
name: finsquad-eval
description: Evaluates the quality of a machine-translated extractive QA dataset (FinSQuAD) by training and testing QA models on it, comparing performance against other translated SQuAD datasets and the original English version. It also assesses translation fidelity through backtranslation and manual error analysis. Use when the user wants to benchmark on Finnish SQuAD2.0, SQuAD2.0, or asks about evaluating this task. Reports exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.05963
  bibtex_key: [REDACTED]
  confidence: high
---

# finsquad-eval

> Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations — Nuutinen et al. (2025) (arXiv:2501.05963, 2025)

## What this evaluates

Evaluates the quality of a machine-translated extractive QA dataset (FinSQuAD) by training and testing QA models on it, comparing performance against other translated SQuAD datasets and the original English version. It also assesses translation fidelity through backtranslation and manual error analysis.

## Datasets

- **Finnish SQuAD2.0** — total ?; splits: train (-1), val (-1), test (-1)
- **SQuAD2.0** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `exact match (EM)` **(primary)** — range: [0, 1]
  - The proportion of questions that receive the exactly correct answer span. A prediction matches the gold span exactly.
- `token F1` — range: [0, 1]
  - The F1 score of the precision and recall of tokens in the predicted answer span compared to the reference answer span. More tolerant to minor changes at the span boundaries.

## Input / output format

**Input**: A question and a context passage in Finnish (or English for baseline models).

**Output**: A predicted answer span (text string) extracted from the context.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    em_scores = []
    f1_scores = []
    for pred, ref in zip(predictions, references):
        pred_tokens = normalize(pred)
        ref_tokens = normalize(ref)
        em_scores.append(1.0 if pred_tokens == ref_tokens else 0.0)
        if not pred_tokens or not ref_tokens:
            f1_scores.append(0.0)
            continue
        common = Counter(pred_tokens) & Counter(ref_tokens)
        num_same = sum(common.values())
        precision = num_same / len(pred_tokens)
        recall = num_same / len(ref_tokens)
        f1 = (2 * precision * recall) / (precision + recall)
        f1_scores.append(f1)
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}
```

## Common pitfalls

- Assuming the performance drop on the translated dataset is solely due to translation noise; it also includes inherent language and model architecture differences.
- Interpreting backtranslation evaluation results as direct single-round translation error; backtranslation accumulates errors over two translation rounds, so the reported drop overestimates the actual impact on the target model.
- Confusing token F1 with exact match; F1 measures token overlap and is tolerant to boundary shifts, but does not guarantee exact span recovery.

## Evidence (verbatim from paper)

> We use as metrics the *exact match (EM)*, the proportion of questions that receive the exactly correct answer span, and *token F1*, the F1 score of the precision and recall of tokens in the predicted answer span, compared to the reference answer span. The latter metric is more tolerant to minor changes at the span boundaries.

## Citation

```bibtex
@misc{nuutinen2025finsquad,
  title={Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations},
  author={Nuutinen et al. (2025)},
  year={2025},
  note={arXiv:2501.05963}
}
```

- arXiv: 2501.05963