Jmedbench Eval

---
name: jmedbench-eval
description: Evaluates Japanese biomedical large language models across five tasks: multiple-choice question answering, named entity recognition, machine translation, document classification, and semantic text similarity. It probes domain-specific knowledge, multilingual comprehension, and in-context learning capabilities. Use when the user wants to benchmark on JMedBench, or asks about evaluating this task. Reports F1-entity, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.13317
  bibtex_key: jiang2024jmedbench
  confidence: high
---

# jmedbench-eval

> JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models — Jiang et al. (2024) (arXiv:2409.13317, 2024)

## What this evaluates

Evaluates Japanese biomedical large language models across five tasks: multiple-choice question answering, named entity recognition, machine translation, document classification, and semantic text similarity. It probes domain-specific knowledge, multilingual comprehension, and in-context learning capabilities.

## Datasets

- **JMedBench** — total 38130; splits: test (-1)

## Metrics

- `F1-entity` **(primary)** — range: percent
  - Micro-averaged F1 score for entity recognition, computed over predicted and gold entity spans.
- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels or multiple-choice answers.
- `BLEU` — range: percent
  - Standard BLEU score for machine translation evaluation.
- `Pearson correlation` — range: [-1, 1]
  - Pearson correlation coefficient between predicted and gold similarity scores.

## Input / output format

**Input**: Text prompt formatted with one of four templates (Standard, English-centric, Instructed, Minimal), optionally containing zero-shot or few-shot demonstrations.

**Output**: Model-generated text response (e.g., selected option, entity tags, translated sentence, class label, or numerical similarity score).

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'NER':
        return micro_f1(predictions, golds)
    elif task in ['MCQA', 'DC']:
        return accuracy(predictions, golds)
    elif task == 'MT':
        return bleu_score(predictions, golds)
    elif task == 'STS':
        return pearson_corr(predictions, golds)
```

## Common pitfalls

- Few-shot NER evaluation is restricted to 1 shot for long-text datasets (MRNER, NRNER) to avoid exceeding model context windows.
- Continual pre-training on biomedical or Japanese text can cause catastrophic forgetting, degrading multilingual performance (e.g., Meditron).
- Machine translation performance correlates more with language generation ability than understanding, especially for cross-lingual directions.

## Evidence (verbatim from paper)

> We report the highest F1-entity score among four prompt templates as discussed in Section [3.3]. The best and second-best performances are highlighted in bold and underlined, respectively.

## Citation

```bibtex
@misc{jiang2024jmedbench,
  title={JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2409.13317}
}
```

- arXiv: 2409.13317