Historical Ocr Eval

---
name: historical-ocr-eval
description: Evaluates LLMs' ability to accurately transcribe historical 18th-century Russian documents while preserving period-specific orthography and avoiding anachronistic character insertions. It probes both standard OCR accuracy and historical fidelity under varying input contexts and prompt strategies. Use when the user wants to benchmark on 18th-century Russian Civil Font Texts, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06743
  bibtex_key: [REDACTED]
  confidence: high
---

# historical-ocr-eval

> Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities — Levchenko (2025) (arXiv:2510.06743, 2025)

## What this evaluates

Evaluates LLMs' ability to accurately transcribe historical 18th-century Russian documents while preserving period-specific orthography and avoiding anachronistic character insertions. It probes both standard OCR accuracy and historical fidelity under varying input contexts and prompt strategies.

## Datasets

- **18th-century Russian Civil Font Texts** — total ?; splits: test (-1); repo https://github.com/mary-lev/historical-ocr-analysis

## Metrics

- `CER` **(primary)** — range: [0, 1]
  - Character Error Rate calculated as the Levenshtein edit distance between the predicted transcription and the ground truth, normalized by the length of the ground truth string.
- `WER` — range: [0, 1]
  - Word Error Rate calculated as the Levenshtein edit distance between the predicted word sequence and the ground truth word sequence, normalized by the length of the ground truth word sequence.
- `HCPR` — range: [0, 1]
  - Historical Character Preservation Rate measuring the percentage of period-specific characters (e.g., i/ї, ъ) correctly preserved in the model's output relative to the source text.
- `AIR` — range: [0, 1]
  - Archaic Insertion Rate measuring the frequency of obsolete, pre-Petrine characters incorrectly inserted by the model that do not appear in the source document.
- `CaseER` — range: [0, 1]
  - Case Error Rate quantifying the proportion of case assignment mistakes, with specific emphasis on visually distinctive historical characters.

## Input / output format

**Input**: Image of a single text line, full page, or sliding window of lines from an 18th-century Russian historical document.

**Output**: Transcribed text string corresponding to the input image content.

## Scoring recipe

```python
def compute_cer(pred, gold):
    dist = levenshtein_distance(pred, gold)
    return dist / len(gold) if len(gold) > 0 else 0.0

def compute_hcpr(pred, gold, period_chars):
    gold_period = [c for c in gold if c in period_chars]
    if not gold_period: return 0.0
    correct = sum(1 for c in gold_period if c in pred)
    return correct / len(gold_period)

def compute_air(pred, gold, archaic_chars):
    inserted = [c for c in pred if c in archaic_chars and c not in gold]
    return len(inserted) / len(pred) if pred else 0.0
```

## Common pitfalls

- Standard metrics like CER and WER fail to detect 'over-historicization', where models systematically insert archaic characters from incorrect historical periods.
- Full-page processing inputs risk hallucinations or detail loss on dense layouts, making line-by-line or sliding-window modes more reliable for accuracy assessment.
- Case-insensitive evaluation masks critical orthographic errors, as historical Russian texts rely heavily on case distinctions for period-specific spelling.

## Evidence (verbatim from paper)

> We employed an evaluation framework with multiple metrics to assess OCR accuracy, historical fidelity, and case sensitivity: Standard OCR Metrics. Character Error Rate (CER) and Word Error Rate (WER), using Levenshtein distance between prediction and ground truth. Case-Insensitive Metrics. CER and WER after lowercasing, to isolate character recognition from case errors. Historical Fidelity Metrics. Historical Character Preservation Rate (HCPR) for period-specific characters (i/ї, , ъ); Archaic Insertion Rate (AIR) for insertion of obsolete, pre-Petrine characters.

## Citation

```bibtex
@misc{levchenko2025historical,
  title={Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities},
  author={Levchenko (2025)},
  year={2025},
  note={arXiv:2510.06743}
}
```

- arXiv: 2510.06743