Multilingual Medical Benchmarks Eval

---
name: multilingual-medical-benchmarks-eval
description: Evaluates multilingual text-to-text models on medical argument mining (sequence labeling) and abstractive question answering across English, Spanish, French, and Italian. Use when the user wants to benchmark on AbstRCT, BioASQ 6B, or asks about evaluating this task. Reports sequence-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.07613
  bibtex_key: garciaferrero2024medicalmt5
  confidence: high
---

# multilingual-medical-benchmarks-eval

> Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain — García-Ferrero et al. (2024) (arXiv:2404.07613, 2024)

## What this evaluates

Evaluates multilingual text-to-text models on medical argument mining (sequence labeling) and abstractive question answering across English, Spanish, French, and Italian.

## Datasets

- **AbstRCT** — total ?; splits: train (-1), test (-1)
- **BioASQ 6B** — total ?; splits: test (-1)

## Metrics

- `sequence-level F1` **(primary)** — range: [0, 1]
  - Standard sequence-level F1 score for exact span matching (Tjong Kim Sang & De Meulder, 2003).
- `QA exact-match` — range: [0, 1]
  - Assessed by comparing generated answers against a set of ideal gold answers provided for each question.

## Input / output format

**Input**: Argument Mining: Raw medical and scientific text. QA: A biomedical question accompanied by a set of relevant text snippets.

**Output**: Argument Mining: Predicted spans labeled as 'Claim' or 'Premise'. QA: A generated natural language answer.

## Scoring recipe

```python
def compute_seq_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# For QA: compare generated answer to each gold answer, take max exact match or F1, then average across questions.
```

## Common pitfalls

- Using token-level F1 instead of the strictly required sequence-level F1 for argument mining.
- Treating glaucoma and mixed splits as in-domain; they are explicitly cross-domain evaluation splits.
- Assuming single gold answer per question for QA; the dataset provides a set of ideal gold answers.

## Evidence (verbatim from paper)

> However, in this paper we report results using the standard sequence level $F_{1}$ score Tjong Kim Sang and De Meulder ([2003]), a much more strict metric, which explains the lower results for all the models. ... Given a biomedical question and a set of snippets of text with relevant information about the question, the model must generate the ideal answer. A set of ideal gold answers are provided to assess the performance of the models.

## Citation

```bibtex
@misc{garciaferrero2024medicalmt5,
  title={Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain},
  author={García-Ferrero et al. (2024)},
  year={2024},
  note={arXiv:2404.07613}
}
```

- arXiv: 2404.07613