Receiptsense Eval

---
name: receiptsense-eval
description: Evaluates multilingual receipt understanding across four tasks: question answering, object detection/classification, OCR, and information extraction. Probes model capabilities in handling real-world noise, mixed Arabic-English layouts, and complex formatting. Use when the user wants to benchmark on ReceiptSense, or asks about evaluating this task. Reports exact match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.04493
  bibtex_key: abdallah2024receiptsense
  confidence: high
---

# receiptsense-eval

> ReceiptSense: Beyond Traditional OCR -- A Dataset for Receipt Understanding — Abdelrahman Abdallah et al. (2024) (arXiv:2406.04493, 2024)

## What this evaluates

Evaluates multilingual receipt understanding across four tasks: question answering, object detection/classification, OCR, and information extraction. Probes model capabilities in handling real-world noise, mixed Arabic-English layouts, and complex formatting.

## Datasets

- **ReceiptSense** — total ?; splits: test (-1); repo https://github.com/Update-For-Integrated-Business-AI/CORU

## Metrics

- `exact match` **(primary)** — range: [0, 1]
  - Proportion of predictions that exactly match the ground truth answer or extracted field value.
- `contains` — range: [0, 1]
  - Proportion of predictions that contain the ground truth string as a substring.
- `precision` — range: [0, 1]
  - Proportion of predicted values that are correct relative to the total number of predictions.
- `recall` — range: [0, 1]
  - Proportion of ground truth values that are correctly predicted relative to the total number of ground truth instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
- `CER` — range: percent
  - Character Error Rate: normalized Levenshtein distance between predicted and ground truth character sequences.
- `WER` — range: percent
  - Word Error Rate: normalized Levenshtein distance between predicted and ground truth word sequences.
- `mAP50` — range: percent
  - Mean Average Precision at IoU threshold 0.50 across all object classes.
- `mAP50-95` — range: percent
  - Mean Average Precision averaged over IoU thresholds from 0.50 to 0.95.

## Input / output format

**Input**: Multilingual receipt images (Arabic-English) paired with questions or target information categories (e.g., Brand, Weight, #Units).

**Output**: Text answers for QA, bounding box coordinates and class labels for object detection, recognized character/word sequences for OCR, and extracted field values for information extraction.

## Scoring recipe

```python
def score(predictions, gold):
    exact_match = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    contains = sum(1 for p, g in zip(predictions, gold) if g in p) / len(gold)
    precision = sum(1 for p, g in zip(predictions, gold) if p in g) / len(predictions)
    recall = sum(1 for p, g in zip(predictions, gold) if p in g) / len(gold)
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    cer = sum(edit_distance(p, g) for p, g in zip(predictions, gold)) / sum(len(g) for g in gold)
    wer = sum(word_edit_distance(p, g) for p, g in zip(predictions, gold)) / sum(len(g.split()) for g in gold)
    return {'exact_match': exact_match, 'contains': contains, 'precision': precision, 'recall': recall, 'f1': f1, 'cer': cer, 'wer': wer}
```

## Common pitfalls

- Zero-shot settings yield very low performance for categories like Brand and Weight, making few-shot evaluation essential for meaningful comparison.
- Models show high variance across different receipt fields (e.g., #Units vs. Pack), so overall F1 can mask category-specific failures.
- Traditional OCR baselines (Tesseract) struggle significantly with multilingual layouts, requiring specialized models for competitive CER/WER.

## Evidence (verbatim from paper)

> Figure[7] provides a comparative analysis of various large language models on the Receipt QA subset of ReceiptSense across four key metrics: precision, recall, exact match, and contains. GPT-4o consistently outperforms other models, achieving the highest scores across all metrics — notably 37.7% precision, 36.4% recall, 35.0% exact match, and 29.1% contains.

## Citation

```bibtex
@misc{abdallah2024receiptsense,
  title={ReceiptSense: Beyond Traditional OCR -- A Dataset for Receipt Understanding},
  author={Abdelrahman Abdallah et al. (2024)},
  year={2024},
  note={arXiv:2406.04493}
}
```

- arXiv: 2406.04493