Meddocbench Eval

---
name: meddocbench-eval
description: Evaluates a model's ability to parse and reason over real-world medical documents, including laboratory test reports and general medical documents, through tasks like table extraction, simple/complex QA, and free-form scoring. Use when the user wants to benchmark on MedDocBench, or asks about evaluating this task. Reports Field-level micro Precision/Recall/F1 & Macro-Doc F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.19090
  bibtex_key: wang2025citrusv
  confidence: high
---

# meddocbench-eval

> Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning — Wang et al. (2025) (arXiv:2509.19090, 2025)

## What this evaluates

Evaluates a model's ability to parse and reason over real-world medical documents, including laboratory test reports and general medical documents, through tasks like table extraction, simple/complex QA, and free-form scoring.

## Datasets

- **MedDocBench** — total ?; splits: Hard test set (-1)

## Metrics

- `Field-level micro Precision/Recall/F1 & Macro-Doc F1` **(primary)** — range: [0, 1]
  - Bipartite matching between predicted and gold entry_names, then field-level correctness for entry_name, result, reference, and unit. Reports micro P/R/F1 at field level and macro-averaged scores at document level.
- `Exact-match accuracy` — range: [0, 1]
  - Computed after canonicalization for simple QA subtasks.
- `LLM judge mean score` — range: [0, 1]
  - Free-form answers scored by an LLM judge on a continuous scale s∈[0,1]; mean score reported.

## Input / output format

**Input**: Medical document images (laboratory test reports or general medical documents) paired with parsing instructions or questions.

**Output**: Markdown tables for full parsing; JSON containing result, reference, and abnormality label for complex QA; free-form text for simple QA and GMD.

## Scoring recipe

```python
def score_meddocbench(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        if pred.task == 'full_parsing':
            pred_table = canonicalize(pred.markdown)
            matches = bipartite_match(pred_table, gold.table)
            scores.append(f1_score(matches))
        elif pred.task == 'simple_qa':
            scores.append(exact_match(canonicalize(pred.text), gold.text))
        elif pred.task == 'gmd':
            scores.append(llm_judge_score(pred.text, gold.text, pred.question))
    return mean(scores)
```

## Common pitfalls

- Canonicalization is strictly required before exact-match or rule-based matching; skipping it causes severe score drops.
- LLM judge is used as a fallback for rule-based matching failures, introducing potential non-determinism and prompt sensitivity.
- Macro-averaged document-level scores can be skewed by rare entry types in sparse tables.

## Evidence (verbatim from paper)

> LTR: All predictions are canonicalized as a preprocessing step. For full parsing, models output a Markdown table; we perform bipartite matching between predicted and gold entry_names, then assess field-level correctness for entry_name, result, reference, and unit. We report micro Precision/Recall/F1 at the field level and macro-averaged scores at the document (image) level. For complex QA, models return JSON containing the result, reference, and an abnormality label; evaluation follows the same matching procedure with P/R/F1 reporting. For simple QA, we compute exact-match accuracy after canonicalization. In all LTR subtasks, an LLM judge is used as a fallback when rule-based matching failed. GMD: Free-form answers are scored by an LLM judge given the question, the predicted answer, and the gold answer. The judge produces a continuous score $s\in[0,1]$; we report the mean score.

## Citation

```bibtex
@misc{wang2025citrusv,
  title={Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.19090}
}
```

- arXiv: 2509.19090