Germanquad Eval

---
name: germanquad-eval
description: Evaluates extractive question answering and dense passage retrieval capabilities in German. It probes a model's ability to locate precise answer spans within a given context and retrieve relevant passages from a large corpus. Use when the user wants to benchmark on GermanQuAD, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.12741
  bibtex_key: moeller2021germanquad
  confidence: high
---

# germanquad-eval

> GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval — Möller et al. (2021) (arXiv:2104.12741, 2021)

## What this evaluates

Evaluates extractive question answering and dense passage retrieval capabilities in German. It probes a model's ability to locate precise answer spans within a given context and retrieve relevant passages from a large corpus.

## Datasets

- **GermanQuAD** — total ?; splits: train (-1), test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - Binary score: 1 if the predicted answer span exactly matches the ground truth span, else 0.
- `F1` — range: [0, 1]
  - F1 score calculated as the ratio of overlapping words between the predicted answer span and the ground truth answer span.
- `Top-1-Accuracy` — range: [0, 1]
  - Binary hit-or-miss score: 1 if there is any positional overlap between the model's predicted answer span and the ground truth, else 0.
- `Recall@10` — range: percent
  - Percentage of queries for which the correct passage appears in the top 10 retrieved results.

## Input / output format

**Input**: Question and a context passage (for QA); Query and a set of candidate passages (for retrieval).

**Output**: Extractive answer span (start and end indices or text) for QA; Ranked list of passages for retrieval.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    f1 = [f1_overlap(p, g) for p, g in zip(predictions, golds)]
    top1 = [1.0 if has_positional_overlap(p, g) else 0.0 for p, g in zip(predictions, golds)]
    return {
        'EM': sum(em) / len(em),
        'F1': sum(f1) / len(f1),
        'Top-1-Acc': sum(top1) / len(top1)
    }
```

## Common pitfalls

- Exact Match and F1 penalize semantically valid but stylistically different answers (e.g., 'Lion' vs 'in star constellation Lion').
- Top-N-Accuracy is lenient and may mask failures in precise span extraction, making it unsuitable for datasets requiring strict annotation alignment.
- Machine-translated training data yields lower performance than hand-annotated in-language data, despite similar lexical overlap.

## Evidence (verbatim from paper)

> Next to exact match (EM) and the ratio of overlapping words (F1) metric we also report Top-N-Accuracy scores. Top-N-Accuracy is a binary hit or miss score for the first N model predictions. The value is one if there is any positional overlap between the ground-truth answer and the model prediction — otherwise, it is zero. Such a hit or miss score is less prone to details of how answers are annotated (e.g., Answer1: "19th century", Answer2: "middle of 19th century") and useful in settings without multi-way annotations or where answers can be more than numbers, single entities, or a few words.

## Citation

```bibtex
@misc{moeller2021germanquad,
  title={GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval},
  author={Möller et al. (2021)},
  year={2021},
  note={arXiv:2104.12741}
}
```

- arXiv: 2104.12741