Heq Eval

---
name: heq-eval
description: This benchmark evaluates extractive reading comprehension in Hebrew, a morphologically rich language. It probes a model's ability to accurately identify answer spans within a given context passage, handling challenges like affixation, spelling variations, and domain-specific vocabulary across news and encyclopedic text. Use when the user wants to benchmark on HeQ, or asks about evaluating this task. Reports TLNLS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01812
  bibtex_key: cohen2025heq
  confidence: high
---

# heq-eval

> HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark — Amir DN Cohen et al. (arXiv:2508.01812, 2025)

## What this evaluates

This benchmark evaluates extractive reading comprehension in Hebrew, a morphologically rich language. It probes a model's ability to accurately identify answer spans within a given context passage, handling challenges like affixation, spelling variations, and domain-specific vocabulary across news and encyclopedic text.

## Datasets

- **HeQ** — total 30147; splits: train (-1), test (-1); repo https://github.com/NNLP-IL/Hebrew-Question-Answering-Dataset

## Metrics

- `EM` — range: [0, 1]
  - Exact Match accuracy. Returns 1 if the predicted answer span exactly matches the gold answer span character-for-character, and 0 otherwise.
- `F1` — range: [0, 1]
  - Token-level F1 score. Computes the harmonic mean of precision and recall based on overlapping tokens between the predicted and gold answer spans.
- `TLNLS` **(primary)** — range: [0, 1]
  - Token-Level Normalized Levenshtein Similarity. Computes the normalized Levenshtein distance between the tokenized predicted and gold answers, then subtracts it from 1 to yield a similarity score that tolerates morphological variations and minor span boundary shifts.

## Input / output format

**Input**: A context passage (in Hebrew) and a corresponding question.

**Output**: An extracted answer span from the context, or an empty string if the answer is not present in the text.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em, f1, tlnls = [], [], []
    for pred, gold in zip(predictions, golds):
        pred, gold = pred.strip(), gold.strip()
        em.append(1.0 if pred == gold else 0.0)
        pred_tokens, gold_tokens = tokenize(pred), tokenize(gold)
        # F1 calculation omitted for brevity
        # TLNLS = 1 - (LevenshteinDistance(pred_tokens, gold_tokens) / max(len(pred_tokens), len(gold_tokens)))
        tlnls.append(1.0 - (levenshtein(pred_tokens, gold_tokens) / max(len(pred_tokens), len(gold_tokens))))
    return {'EM': sum(em)/len(em), 'F1': sum(f1)/len(f1), 'TLNLS': sum(tlnls)/len(tlnls)}
```

## Common pitfalls

- Span boundary inaccuracies: Models frequently return correct answers but with incorrect start/end indices, which heavily penalizes EM despite semantic correctness.
- Morphological sensitivity: Standard character-level exact match fails to account for Hebrew affixation and orthographic variations, leading to artificially low scores without token-level or normalized similarity metrics.
- Domain shift masking: Aggregate scores often hide significant performance drops when models trained on one domain (e.g., Wikipedia) are evaluated on another (e.g., tech news).

## Evidence (verbatim from paper)

> We evaluate several known pre-trained models on the HeQtest set. Table 4: Performance Comparison of Different Models on the HeQDataset. ... | Model | EM | $F_{1}$ | TLNLS |

## Citation

```bibtex
@misc{cohen2025heq,
  title={HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark},
  author={Amir DN Cohen et al.},
  year={2025},
  note={arXiv:2508.01812}
}
```

- arXiv: 2508.01812