Pharmaship Eval

---
name: pharmaship-eval
description: Evaluates layout-aware document understanding models on Chinese pharmaceutical shipping documents. It probes semantic entity recognition, entity linking, and reading order prediction, specifically testing robustness to dense tabular layouts and long-range semantic dependencies. Use when the user wants to benchmark on PharmaShip, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.23714
  bibtex_key: xie2025pharmaship
  confidence: high
---

# pharmaship-eval

> PharmaShip: An Entity-Centric, Reading-Order-Supervised Benchmark for Chinese Pharmaceutical Shipping Documents — Xie et al. (2025) (arXiv:2512.23714, 2025)

## What this evaluates

Evaluates layout-aware document understanding models on Chinese pharmaceutical shipping documents. It probes semantic entity recognition, entity linking, and reading order prediction, specifically testing robustness to dense tabular layouts and long-range semantic dependencies.

## Datasets

- **PharmaShip** — total ?; splits: test (-1); repo https://github.com/KevinYuLei/PharmaShip

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Reported separately for SER, EL, and ROP tasks.

## Input / output format

**Input**: Chinese pharmaceutical shipping documents (images/PDFs) with text segments, layout coordinates, and geometric/pixel information.

**Output**: Predicted semantic entity boundaries, linked entity identifiers, and ordered sequences or graphs for word-level and segment-level reading order.

## Scoring recipe

```python
def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

# Applied per task (SER, EL, ROP)
# Precision = TP / (TP + FP)
# Recall = TP / (TP + FN)
```

## Common pitfalls

- Models overfit to layout heuristics and block-level 'false coupling' present in conventional benchmarks like FUNSD, failing on entity-centric annotations.
- Long-range medical relations and dense tabular layouts cause precision-recall trade-offs; geometry-only models are precision-leaning while pixel-aware models are recall-leaning.
- Segment-level reading order prediction suffers from boundary ambiguity and long-range crossings, making it significantly harder than word-level ordering.

## Evidence (verbatim from paper)

> Table III reports Performance comparison of different models on SER, EL, and ROP tasks of PharmaShip. Improvements $(\uparrow)$ denote F1 gains of RORE-enhanced variants.

## Citation

```bibtex
@misc{xie2025pharmaship,
  title={PharmaShip: An Entity-Centric, Reading-Order-Supervised Benchmark for Chinese Pharmaceutical Shipping Documents},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2512.23714}
}
```

- arXiv: 2512.23714