Sard Ocr Eval

---
name: sard-ocr-eval
description: Evaluates the robustness and accuracy of OCR models on synthetic, book-style Arabic documents with high typographic diversity across 10 fonts. It measures character-level precision, word-level accuracy, and overall sequence fluency to benchmark vision-language and traditional OCR systems. Use when the user wants to benchmark on SARD, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24600
  bibtex_key: nacar2025sard
  confidence: high
---

# sard-ocr-eval

> SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition — Nacar et al. (2025) (arXiv:2505.24600, 2025)

## What this evaluates

Evaluates the robustness and accuracy of OCR models on synthetic, book-style Arabic documents with high typographic diversity across 10 fonts. It measures character-level precision, word-level accuracy, and overall sequence fluency to benchmark vision-language and traditional OCR systems.

## Datasets

- **SARD** — total 843622; splits: test (2000)

## Metrics

- `CER` **(primary)** — range: [0, 1]
  - Character Error Rate: normalized Levenshtein edit distance between predicted and ground truth character sequences. Lower is better.
- `WER` — range: [0, 1]
  - Word Error Rate: normalized Levenshtein edit distance between predicted and ground truth word sequences. Lower is better.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy score: geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short outputs. Higher is better.

## Input / output format

**Input**: Full-page synthetic Arabic document images (book-style layout).

**Output**: Recognized text string corresponding to the page content.

## Scoring recipe

```python
def compute_cer(pred, ref):
    return levenshtein_distance(pred, ref) / len(ref)
def compute_wer(pred, ref):
    return levenshtein_distance(pred.split(), ref.split()) / len(ref.split())
def compute_bleu(pred, ref):
    return nltk.translate.bleu_score.sentence_bleu([ref.split()], pred.split())
# Average scores across 2000 test images (200 per font)
```

## Common pitfalls

- Performance varies significantly across different fonts, so averaging across fonts can mask font-specific weaknesses.
- BLEU score measures sequence fluency but may not perfectly align with character/word-level accuracy for OCR tasks.
- Synthetic dataset evaluation may not generalize to real-world scanned documents with noise, skew, or complex layouts.

## Evidence (verbatim from paper)

> To demonstrate the utility of SARD for benchmarking and to provide initial performance metrics, we evaluated several representative OCR models. Our evaluation centered on standard OCR metrics: Character Error Rate (CER), Word Error Rate (WER), and Bilingual Evaluation Understudy (BLEU) score. Lower CER and WER values signify better accuracy at the character and word levels, respectively. A higher BLEU score indicates greater overall sequence similarity between the predicted and ground truth text, a metric commonly used in text generation tasks but also valuable here for assessing the fluency and overall correctness of the recognized text block. ... For the results presented in Tables[2], [3], and [4], 200 randomly selected images were used for each of the ten fonts incorporated in the dataset, resulting in a total test set of 2000 images for this comprehensive analysis.

## Citation

```bibtex
@misc{nacar2025sard,
  title={SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition},
  author={Nacar et al. (2025)},
  year={2025},
  note={arXiv:2505.24600}
}
```

- arXiv: 2505.24600