Knesset Corpus Extraction Eval

---
name: knesset-corpus-extraction-eval
description: Evaluates the accuracy and robustness of an automated extraction pipeline that converts raw Hebrew parliamentary documents into structured metadata, speaker lists, and text. It probes the system's ability to correctly parse dates, identify speakers, extract sentences, and match speaker names to an official database of Knesset members. Use when the user wants to benchmark on Knesset Corpus, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.18115
  bibtex_key: sommer2024knesset
  confidence: high
---

# knesset-corpus-extraction-eval

> The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings — Sommer et al. (2024) (arXiv:2405.18115, 2024)

## What this evaluates

Evaluates the accuracy and robustness of an automated extraction pipeline that converts raw Hebrew parliamentary documents into structured metadata, speaker lists, and text. It probes the system's ability to correctly parse dates, identify speakers, extract sentences, and match speaker names to an official database of Knesset members.

## Datasets

- **Knesset Corpus** — total 44027; splits: evaluation (20)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Proportion of extracted speaker names that correctly match the official MP database after manual verification, calculated as correct_matches / total_records_analyzed.

## Input / output format

**Input**: Raw parliamentary document files containing Hebrew text, dates, and speaker mentions.

**Output**: Structured records containing document metadata (date), extracted text, identified speaker names, and matched MP database entries.

## Scoring recipe

```python
def score_success_rate(extracted_names, mp_db):
    correct = 0
    total = len(extracted_names)
    for name in extracted_names:
        clean = normalize(name)
        if clean in mp_db and mp_db[clean] == assigned_mp:
            correct += 1
    return (correct / total) * 100
```

## Common pitfalls

- The evaluation subset is very small (20 documents) compared to the full corpus (42k+ files), so results may not generalize to the remaining 85%.
- Name matching errors are heavily driven by orthographic variations between protocol text and the official database, causing false negatives that are hard to automate.
- Cleaning artifacts like '<' prefixes on speaker names are counted as errors but remain easily identifiable by humans, inflating error rates if not handled contextually.

## Evidence (verbatim from paper)

> Out of the total set, 285,790 records were automatically detected as correct since a match was found and the assigned name was identical to the clean name. We manually analyzed the remaining 400K records. About 38K records (5.52%) were found to be incorrect and were manually fixed. This reflects success rate of 94.48% .

## Citation

```bibtex
@misc{sommer2024knesset,
  title={The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings},
  author={Sommer et al. (2024)},
  year={2024},
  note={arXiv:2405.18115}
}
```

- arXiv: 2405.18115