Qa4ie Eval

---
name: qa4ie-eval
description: Evaluates document-level information extraction by framing it as a question answering task. It probes a model's ability to extract cross-sentence relation triples from large documents using entity-relation queries and knowledge base alignment. Use when the user wants to benchmark on QA4IE, or asks about evaluating this task. Reports Exact Match (EM), F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.03396
  bibtex_key: qiu2018qa4ie
  confidence: high
---

# qa4ie-eval

> QA4IE: A Question Answering based Framework for Information Extraction — Lin Qiu et al. (2018) (arXiv:1804.03396, 2018)

## What this evaluates

Evaluates document-level information extraction by framing it as a question answering task. It probes a model's ability to extract cross-sentence relation triples from large documents using entity-relation queries and knowledge base alignment.

## Datasets

- **QA4IE** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/SJTU-lqiu/QA4IE

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - EM measures the percentage that the model prediction matches one of the ground truth answers exactly.
- `F1-score` **(primary)** — range: [0, 1]
  - F1-score measures the token-level overlap between the prediction and ground truth answers, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: A document (or sentence) and a candidate entity-relation query (entity-property pair).

**Output**: A predicted answer span or sequence; outputs an 'eos' symbol or no answer if the confidence score is below a threshold δ.

## Scoring recipe

```python
def compute_metrics(pred, golds):
    em = 1.0 if pred.strip() in [g.strip() for g in golds] else 0.0
    pred_tokens = pred.split()
    gold_tokens = [g.split() for g in golds]
    intersection = sum(Counter(pred_tokens) & Counter(g) for g in gold_tokens)
    precision = intersection / len(pred_tokens) if pred_tokens else 0
    recall = intersection / len(gold_tokens[0]) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return em, f1
```

## Common pitfalls

- The IE evaluation relies on confidence thresholds (Score_mul, Score_avg) to filter answers, causing precision/recall to vary drastically with the threshold.
- Baseline Open IE systems are evaluated on isolated sentences rather than full documents, creating an unfair comparison where baselines receive easier inputs.
- The benchmark assumes the first entity (subject) is known from the article title, so it does not evaluate entity recognition or linking.

## Evidence (verbatim from paper)

> Two metrics are introduced in the SQuAD dataset: Exact Match (EM) and F1-score. EM measures the percentage that the model prediction matches one of the ground truth answers exactly while F1-score measures the overlap between the prediction and ground truth answers. Our QA4IE benchmark also adopts these two metrics.

## Citation

```bibtex
@misc{qiu2018qa4ie,
  title={QA4IE: A Question Answering based Framework for Information Extraction},
  author={Lin Qiu et al. (2018)},
  year={2018},
  note={arXiv:1804.03396}
}
```

- arXiv: 1804.03396