Exams Qa Eval

---
name: exams-qa-eval
description: Evaluates multilingual and cross-lingual question answering capabilities on high school-level exams across multiple subjects and languages. Probes domain-specific reasoning, knowledge retrieval, and zero-shot transfer between languages with varying linguistic and subject overlaps. Use when the user wants to benchmark on EXAMS, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.03080
  bibtex_key: hardalov2020exams
  confidence: high
---

# exams-qa-eval

> EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering — Hardalov et al. (2020) (arXiv:2011.03080, 2020)

## What this evaluates

Evaluates multilingual and cross-lingual question answering capabilities on high school-level exams across multiple subjects and languages. Probes domain-specific reasoning, knowledge retrieval, and zero-shot transfer between languages with varying linguistic and subject overlaps.

## Datasets

- **EXAMS** — total 24000; splits: test (-1); repo http://github.com/mhardalov/exams-qa

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: A multiple-choice question with four options (A-D), optionally accompanied by retrieved Wikipedia context paragraphs.

**Output**: A single predicted option label (A, B, C, or D) corresponding to the correct answer.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Assuming simple information retrieval (IR) over Wikipedia is sufficient; the dataset requires complex domain-specific reasoning beyond word matching.
- Overlooking subject-level performance variations; models struggle significantly with Chemistry and Physics compared to other subjects.
- Ignoring the impact of subject overlap and language family similarity on cross-lingual zero-shot transfer performance.

## Evidence (verbatim from paper)

> In Table 4, we show the overall per-language performance of the evaluated models. ... IR is better than random guessing, but it is clear that most questions require reasoning beyond simple word matching. ... We can see that the Natural Science questions are the most challenging ones, which is mostly due to Chemistry and Physics.

## Citation

```bibtex
@misc{hardalov2020exams,
  title={EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering},
  author={Hardalov et al. (2020)},
  year={2020},
  note={arXiv:2011.03080}
}
```

- arXiv: 2011.03080