Jaquad Eval

---
name: jaquad-eval
description: Extractive machine reading comprehension in Japanese. It probes a model's ability to locate exact answer spans in Japanese Wikipedia text given a question, evaluating performance across different answer types, question reasoning types, and answer lengths. Use when the user wants to benchmark on JaQuAD, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.01764
  bibtex_key: so2022jaquad
  confidence: high
---

# jaquad-eval

> JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension — So et al. (2022) (arXiv:2202.01764, 2022)

## What this evaluates

Extractive machine reading comprehension in Japanese. It probes a model's ability to locate exact answer spans in Japanese Wikipedia text given a question, evaluating performance across different answer types, question reasoning types, and answer lengths.

## Datasets

- **JaQuAD** — total 39696; splits: dev (-1), test (-1); repo https://github.com/SkelterLabsInc/JaQuAD

## Metrics

- `F1 score` **(primary)** — range: percent
  - Token-level F1 score measuring the overlap between predicted and gold answer spans. Calculated as 2 * (precision * recall) / (precision + recall), where precision and recall are based on token counts.
- `EM` — range: percent
  - Exact Match accuracy, where the predicted answer string must exactly equal the gold answer string to receive a score of 1, otherwise 0.

## Input / output format

**Input**: A Japanese question paired with a context paragraph from Japanese Wikipedia.

**Output**: An exact text span extracted from the context that answers the question.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(preds, golds):
        pred_tokens = pred.split()
        gold_tokens = gold.split()
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        if num_same == 0:
            f1_scores.append(0.0)
            em_scores.append(0.0)
            continue
        precision = num_same / len(pred_tokens)
        recall = num_same / len(gold_tokens)
        f1 = (2 * precision * recall) / (precision + recall)
        f1_scores.append(f1)
        em_scores.append(1.0 if pred == gold else 0.0)
    return sum(f1_scores) / len(f1_scores), sum(em_scores) / len(em_scores)
```

## Common pitfalls

- Context truncation to 384 tokens may cut off answer spans, artificially lowering scores.
- Rare answer/question types (e.g., Manner, Cause, Logical reasoning) comprise <1% of data, making aggregate metrics unrepresentative for those categories.
- Short (1-2 tokens) and long (9+ tokens) answers show systematically lower performance, so overall scores mask length-dependent difficulty.

## Evidence (verbatim from paper)

> The baseline achieves 78.92% for F1 score and 63.38% for EM on test set.

## Citation

```bibtex
@misc{so2022jaquad,
  title={JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension},
  author={So et al. (2022)},
  year={2022},
  note={arXiv:2202.01764}
}
```

- arXiv: 2202.01764