Qg Bench Eval

---
name: qg-bench-eval
description: Evaluates the ability of generative language models to produce paragraph-level questions conditioned on a target answer and a context sentence. It probes domain adaptability and multilingual generalization across diverse extractive QA datasets. Use when the user wants to benchmark on SQuAD v1.1, SQuADShifts, SubjQA, Multilingual QA (JAQuAD, GerQuAD, SberQuAD, KorQuAD, FQuAD, Spanish SQuAD, Italian SQuAD), or asks about evaluating this task. Reports automatic evaluation metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.03992
  bibtex_key: ushio2022generative
  confidence: medium
---

# qg-bench-eval

> Generative Language Models for Paragraph-Level Question Generation — Ushio et al. (2022) (arXiv:2210.03992, 2022)

## What this evaluates

Evaluates the ability of generative language models to produce paragraph-level questions conditioned on a target answer and a context sentence. It probes domain adaptability and multilingual generalization across diverse extractive QA datasets.

## Datasets

- **SQuAD v1.1** — total 98169; splits: train (75722), val (10570), test (11877)
- **SQuADShifts** — total ?; splits: train (-1), val (-1), test (-1)
- **SubjQA** — total ?; splits: train (-1), val (-1), test (-1)
- **Multilingual QA (JAQuAD, GerQuAD, SberQuAD, KorQuAD, FQuAD, Spanish SQuAD, Italian SQuAD)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `automatic evaluation metrics` **(primary)** — range: [0, 1]
  - Standard sequence-to-sequence generation metrics (e.g., BLEU, ROUGE, BERTScore) are used to compare generated questions against gold questions. Specific formulas are not detailed in this section.
- `manual assessment for answerability and coherence` — range: percent
  - Human evaluators rate generated questions on answerability (whether the question can be answered by the context) and coherence (fluency and logical consistency).

## Input / output format

**Input**: paragraph, sentence, answer

**Output**: question (constrained to be a substring of the provided sentence)

## Scoring recipe

```python
# Pseudo-code for evaluation
# 1. Generate question from (paragraph, sentence, answer)
# 2. Compute automatic metrics (e.g., BLEU/ROUGE/BERTScore) against gold question
# 3. Conduct manual evaluation:
#    - Rate answerability: Can the question be answered by the paragraph?
#    - Rate coherence: Is the question fluent and logically sound?
# 4. Aggregate scores across domains and languages
```

## Common pitfalls

- Many multilingual datasets lack official test sets; test splits are sampled from training data with no paragraph overlap.
- Automatic metrics alone are insufficient; manual evaluation is required to assess answerability and coherence.
- Domain-specific and low-resource language settings show significant performance degradation compared to general English domains.

## Evidence (verbatim from paper)

> The benchmark enables fair comparison of generative language models through fine-tuning on diverse domains and languages, revealing limitations in automatic evaluation metrics and highlighting the importance of manual assessment for answerability and coherence.

## Citation

```bibtex
@misc{ushio2022generative,
  title={Generative Language Models for Paragraph-Level Question Generation},
  author={Ushio et al. (2022)},
  year={2022},
  note={arXiv:2210.03992}
}
```

- arXiv: 2210.03992