Muharaf Htr Eval

---
name: muharaf-htr-eval
description: Evaluates handwritten text recognition (HTR) systems on historical Arabic manuscripts. It probes the model's ability to accurately transcribe cursive text at both the page and line levels, handling contextual character variations and layout structures. Use when the user wants to benchmark on Muharaf, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.09630
  bibtex_key: [REDACTED]
  confidence: high
---

# muharaf-htr-eval

> Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition — Saeed et al. (2024) (arXiv:2406.09630, 2024)

## What this evaluates

Evaluates handwritten text recognition (HTR) systems on historical Arabic manuscripts. It probes the model's ability to accurately transcribe cursive text at both the page and line levels, handling contextual character variations and layout structures.

## Datasets

- **Muharaf** — total 1646; splits: train (1500), val (50), test (96); repo https://github.com/mehreenmehreen/muharaf

## Metrics

- `CER` **(primary)** — range: [0, 1]
  - Character Error Rate computed as the Levenshtein distance between the predicted and ground truth strings, normalized by the length of the ground truth string.
- `WER` — range: [0, 1]
  - Word Error Rate computed as the Levenshtein distance between the predicted and ground truth strings, normalized by the length of the ground truth string.

## Input / output format

**Input**: Page images of handwritten Arabic manuscripts, and pre-segmented text line images for line-level evaluation.

**Output**: Recognized text string (sequence of characters or words) corresponding to the input image or line.

## Scoring recipe

```python
def compute_cer(pred, gt):
    dist = levenshtein_distance(pred, gt)
    return dist / len(gt)

def compute_wer(pred, gt):
    dist = levenshtein_distance(pred, gt)
    return dist / len(gt)

# Average over test set
cer = sum(compute_cer(p, g) for p, g in zip(predictions, ground_truths)) / len(ground_truths)
wer = sum(compute_wer(p, g) for p, g in zip(predictions, ground_truths)) / len(ground_truths)
```

## Common pitfalls

- Line-level CER/WER can be artificially inflated by lines containing only isolated numbers or single words, where a single error causes the error rate to jump to 1.0.
- Results are averaged over three random train/val/test splits, so single-run evaluations may not be reproducible without fixing the random seed or using the provided splits.

## Evidence (verbatim from paper)

> The metrics for evaluation are the character error rate (CER) and word error rate (WER), both computed using Levenshtein distance normalized by the length of the string representing the ground truth.

## Citation

```bibtex
@misc{saeed2024muharaf,
  title={Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition},
  author={Saeed et al. (2024)},
  year={2024},
  note={arXiv:2406.09630}
}
```

- arXiv: 2406.09630