Medlaybench V Eval

---
name: medlaybench-v-eval
description: Evaluates the ability of medical vision-language models to align expert clinical terminology with patient-accessible layman language while preserving diagnostic accuracy. It measures lexical overlap, readability, clinical factuality, and zero-shot image-text retrieval performance. Use when the user wants to benchmark on MedLayBench-V, or asks about evaluating this task. Reports Recall@K (R@1, R@5, R@10).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.05738
  bibtex_key: jang2026medlaybenchv
  confidence: high
---

# medlaybench-v-eval

> MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models — Jang et al. (2026) (arXiv:2604.05738, 2026)

## What this evaluates

Evaluates the ability of medical vision-language models to align expert clinical terminology with patient-accessible layman language while preserving diagnostic accuracy. It measures lexical overlap, readability, clinical factuality, and zero-shot image-text retrieval performance.

## Datasets

- **MedLayBench-V** — total 79789; splits: train (-1), val (-1), test (9927); repo https://github.com/janghana/MedLayBench-V

## Metrics

- `Recall@K (R@1, R@5, R@10)` **(primary)** — range: percent
  - Measures retrieval accuracy by checking if the ground-truth match appears within the top-K ranked candidates based on cosine similarity of L2-normalized embeddings.
- `BLEU-4, ROUGE-L, METEOR` — range: [0, 1]
  - Standard n-gram and sequence overlap metrics measuring structural similarity and lexical overlap between expert and layman captions.
- `LENS, RaTEScore, GREEN` — range: [0, 1]
  - LENS is a learnable metric for text simplification. RaTEScore and GREEN are model-based metrics designed to detect hallucinations and ensure clinical correctness in radiology reports.
- `FKGL, CLI, DCRS, SMOG, FRE` — range: other
  - Standard readability formulas quantifying text accessibility, grade level, and reading ease.

## Input / output format

**Input**: Image and text pair (for retrieval) or image with prompt (for captioning).

**Output**: Ranked list of candidate texts/images (retrieval) or generated natural language caption (captioning).

## Scoring recipe

```python
def compute_recall_at_k(sim_matrix, k):
    # sim_matrix: [N, N] cosine similarities between images and texts
    top_k_indices = np.argsort(sim_matrix, axis=1)[:, -k:]
    correct = np.sum(np.diag(top_k_indices) == np.arange(len(sim_matrix)))
    return correct / len(sim_matrix) * 100
```

## Common pitfalls

- Confusing expert vs. layman caption performance, as the benchmark reports both side-by-side.
- Applying fine-tuning or prompt engineering, which violates the zero-shot evaluation protocol.
- Ignoring domain adaptation effects, as general-domain VLMs significantly underperform medical-specific models on this benchmark.

## Evidence (verbatim from paper)

> To assess whether the simplified text preserves essential semantic information for automated analysis, we evaluate zero-shot text-to-image retrieval performance. We report Recall@K (R@1, R@5, R@10) to measure retrieval accuracy using the generated captions.

## Citation

```bibtex
@misc{jang2026medlaybenchv,
  title={MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2604.05738}
}
```

- arXiv: 2604.05738