Medrect Eval

---
name: medrect-eval
description: Evaluates large language models' ability to detect, localize, and correct clinical errors in medical texts across Japanese and English. It probes cross-lingual medical reasoning, precise error identification, and accurate text correction capabilities. Use when the user wants to benchmark on MedRECT-ja, MedRECT-en, or asks about evaluating this task. Reports Error Detection F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00421
  bibtex_key: iwase2025medrect
  confidence: high
---

# medrect-eval

> MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts — Iwase et al. (2025) (arXiv:2511.00421, 2025)

## What this evaluates

Evaluates large language models' ability to detect, localize, and correct clinical errors in medical texts across Japanese and English. It probes cross-lingual medical reasoning, precise error identification, and accurate text correction capabilities.

## Datasets

- **MedRECT-ja** — total ?; splits: test (-1)
- **MedRECT-en** — total ?; splits: test (-1)

## Metrics

- `Error Detection F1` **(primary)** — range: [0, 1]
  - F1 score for binary classification determining whether a clinical error is present in the text.
- `Sentence Extraction Accuracy` — range: [0, 1]
  - Accuracy of predicting the correct sentence number containing the error (multi-class classification).
- `ROUGE-1 F-score` — range: [0, 1]
  - ROUGE-1 F1 computed with MeCab tokenizer for Japanese and whitespace tokenizer for English.
- `BERTScore F1` — range: [0, 1]
  - BERTScore F1 using microsoft/deberta-xlarge-mnli as the base model with language-specific settings.
- `BLEURT` — range: [0, 1]
  - BLEURT score computed using the BLEURT-20 checkpoint.
- `Error Correction Average` — range: [0, 1]
  - Arithmetic average of ROUGE-1, BERTScore, and BLEURT scores.

## Input / output format

**Input**: Zero-shot prompt providing a clinical text snippet containing an error, instructing the model to identify the error, specify the sentence number, and provide the corrected text.

**Output**: Structured response containing: (1) binary error detection, (2) sentence number localization, (3) corrected clinical text.

## Scoring recipe

```python
for each sample in test_set:
  if gold_has_error:
    det_f1 = compute_f1(pred_detect, gold_detect)
    loc_acc = compute_accuracy(pred_sentence_num, gold_sentence_num)
    if pred_detect == 'error':
      rouge1 = compute_rouge1(pred_correction, gold_correction, tokenizer='mecab' if lang=='ja' else 'whitespace')
      bertscore = compute_bertscore(pred_correction, gold_correction, model='deberta-xlarge-mnli')
      bleurt = compute_bleurt(pred_correction, gold_correction, model='bleurt-20')
      corr_avg = (rouge1 + bertscore + bleurt) / 3
  else:
    corr_avg = None
return det_f1, loc_acc, corr_avg
```

## Common pitfalls

- Correction metrics (ROUGE-1, BERTScore, BLEURT) are only computed on samples where both the model prediction and ground truth indicate an error exists; ignoring this filtering inflates scores.
- Sentence extraction accuracy is only evaluated on samples with a ground-truth error, not on the full dataset.
- Japanese ROUGE-1 requires MeCab tokenization, while English uses whitespace tokenization; using a single tokenizer for both will yield incorrect scores.

## Evidence (verbatim from paper)

> We employed the following evaluation metrics: Error Detection F1 (binary classification), Sentence Extraction Accuracy (multi-class classification of sentence number), and Error Correction using ROUGE-1, BERTScore, BLEURT, and their arithmetic average. Following the MEDIQA-CORR 2024 evaluation protocol, sentence extraction is computed only on samples with a ground-truth error, and error correction metrics are computed only on samples where both prediction and ground-truth indicate the presence of an error.

## Citation

```bibtex
@misc{iwase2025medrect,
  title={MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts},
  author={Iwase et al. (2025)},
  year={2025},
  note={arXiv:2511.00421}
}
```

- arXiv: 2511.00421