Drcd Eval

---
name: drcd-eval
description: Evaluates a model's ability to perform span-based machine reading comprehension in traditional Chinese. It probes factual retrieval and exact answer extraction from provided context paragraphs without requiring complex inference or multiple-choice reasoning. Use when the user wants to benchmark on DRCD, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.00920
  bibtex_key: shao2018drcd
  confidence: high
---

# drcd-eval

> DRCD: a Chinese Machine Reading Comprehension Dataset — Shao et al. (2018) (arXiv:1806.00920, 2018)

## What this evaluates

Evaluates a model's ability to perform span-based machine reading comprehension in traditional Chinese. It probes factual retrieval and exact answer extraction from provided context paragraphs without requiring complex inference or multiple-choice reasoning.

## Datasets

- **DRCD** — total 33941; splits: dev (-1), test (-1); repo https://github.com/DRCKnowledgeTeam/DRCD

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Character-level F1 score calculated as the intersection of predicted and ground-truth Chinese characters divided by the average of their lengths. Punctuation is ignored.
- `Exact Match` — range: [0, 1]
  - Binary metric that returns 1 if the predicted answer string exactly matches the ground-truth answer string (ignoring punctuation), else 0.

## Input / output format

**Input**: A context paragraph and a corresponding question in traditional Chinese.

**Output**: A text span extracted from the context paragraph that answers the question.

## Scoring recipe

```python
def compute_metrics(pred, gold):
    pred_clean = remove_punctuation(pred)
    gold_clean = remove_punctuation(gold)
    em = 1.0 if pred_clean == gold_clean else 0.0
    pred_chars = set(pred_clean)
    gold_chars = set(gold_clean)
    if not pred_chars or not gold_chars:
        f1 = 0.0
    else:
        precision = len(pred_chars & gold_chars) / len(pred_chars)
        recall = len(pred_chars & gold_chars) / len(gold_chars)
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1, em
```

## Common pitfalls

- Punctuation must be stripped before comparison, otherwise scores will be artificially low.
- F1 is computed at the character level (bag of characters), not word or token level, which is critical for Chinese text.
- Models must extract a contiguous span from the context; generated free-form answers are not evaluated correctly.

## Evidence (verbatim from paper)

> F1 score and exact match from Rajpurkar et al. (2016) are used as the evaluation metrics. Both metrics ignore punctuations. In F1 score metric, we consider predictions and ground truth as bag of Chinese character.

## Citation

```bibtex
@misc{shao2018drcd,
  title={DRCD: a Chinese Machine Reading Comprehension Dataset},
  author={Shao et al. (2018)},
  year={2018},
  note={arXiv:1806.00920}
}
```

- arXiv: 1806.00920