Mixqg Qg Eval

---
name: mixqg-qg-eval
description: Evaluates a neural question generation model's ability to produce fluent, relevant questions conditioned on a target answer and context. It probes both automatic n-gram/embedding-based similarity metrics and human-rated utility for educational quiz creation. Use when the user wants to benchmark on SQuAD, NQ, Quoref, DROP, or asks about evaluating this task. Reports question approval rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.08175
  bibtex_key: murakhovska2021mixqg
  confidence: high
---

# mixqg-qg-eval

> MixQG: Neural Question Generation with Mixed Answer Types — Murakhovs'ka et al. (2021) (arXiv:2110.08175, 2021)

## What this evaluates

Evaluates a neural question generation model's ability to produce fluent, relevant questions conditioned on a target answer and context. It probes both automatic n-gram/embedding-based similarity metrics and human-rated utility for educational quiz creation.

## Datasets

- **SQuAD** — total ?; splits: train (-1), dev (-1)
- **NQ** — total ?; splits: train (-1), dev (-1)
- **Quoref** — total ?; splits: train (-1), test (-1)
- **DROP** — total ?; splits: train (-1), dev (-1)

## Metrics

- `question approval rate` **(primary)** — range: percent
  - Percentage of generated questions approved by human teachers for inclusion in a quiz, calculated as approved_count / total_generated_questions.
- `BLEU` — range: [0, 1]
  - Standard n-gram precision metric with brevity penalty, measuring lexical overlap between generated and reference questions.
- `ROUGE` — range: [0, 1]
  - Recall-oriented n-gram overlap metric (typically ROUGE-L or ROUGE-1/2), measuring how much reference text is covered by the generation.
- `METEOR` — range: [0, 1]
  - Metric that aligns generated and reference words using synonymy and stemming, combining precision and recall with a penalty for fragmentation.
- `BERTScore` — range: [0, 1]
  - Computes cosine similarity between contextual embeddings of generated and reference tokens, then aggregates precision, recall, and F1 scores.

## Input / output format

**Input**: Context (e.g., Wikipedia article or sentence) and a target answer (quiz concept or answer span).

**Output**: A single generated question string.

## Scoring recipe

```python
def score(predictions, golds, human_approvals=None):
    scores = {}
    scores['BLEU'] = compute_bleu(predictions, golds)
    scores['ROUGE'] = compute_rouge(predictions, golds)
    scores['METEOR'] = compute_meteor(predictions, golds)
    scores['BERTScore'] = compute_bertscore(predictions, golds)
    if human_approvals is not None:
        approved = sum(1 for a in human_approvals if a)
        scores['question approval rate'] = (approved / len(predictions)) * 100
    return scores
```

## Common pitfalls

- n-gram based metrics (BLEU/ROUGE) may not correlate well with human judgments of question quality and fluency.
- Models trained solely on a target dataset can outperform joint-trained models in zero-shot settings, but fine-tuning the joint model yields the best overall performance.
- Human evaluation requires teachers to explicitly select a quiz concept (target answer) from the context to ensure generated questions match the intended answer specificity.

## Evidence (verbatim from paper)

> We report the commonly-used metrics applied in the QG research: BLEU, ROUGE, and METEOR scores. We also report BERTScore, which relies on contextual embeddings to produce the final score. The success of a QG model depends on its question approval rate. We compare seven QG models and collect 3,164 human-annotated samples from 10 recruited teachers. Teachers can then approve a generated question to be included on the quiz or reject it and provide a reason for rejection.

## Citation

```bibtex
@misc{murakhovska2021mixqg,
  title={MixQG: Neural Question Generation with Mixed Answer Types},
  author={Murakhovs'ka et al. (2021)},
  year={2021},
  note={arXiv:2110.08175}
}
```

- arXiv: 2110.08175