Bionpars Bench Eval

---
name: bionpars-bench-eval
description: Evaluates a Persian biomedical large language model's ability to generate accurate, domain-specific long-form answers and summaries. It probes subject-specific knowledge acquisition, knowledge synthesis, and evidence-based reasoning by comparing model outputs against human-written biomedical references. Use when the user wants to benchmark on BioPars-BENCH, or asks about evaluating this task. Reports BERTScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21567
  bibtex_key: [REDACTED]
  confidence: high
---

# bionpars-bench-eval

> BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining — Baqer M. Merzah et al. (2025) (arXiv:2506.21567, 2025)

## What this evaluates

Evaluates a Persian biomedical large language model's ability to generate accurate, domain-specific long-form answers and summaries. It probes subject-specific knowledge acquisition, knowledge synthesis, and evidence-based reasoning by comparing model outputs against human-written biomedical references.

## Datasets

- **BioPars-BENCH** — total 5231; splits: test (5231); repo https://github.com/amirap80/BioPars

## Metrics

- `BERTScore` **(primary)** — range: [0, 1]
  - Computes token-level cosine similarity using BERT embeddings (layer 9). Precision and Recall are averaged across tokens, and F1 is calculated as 2·P·R/(P+R). Scores are scaled to [0,1] using random text baselines.

## Input / output format

**Input**: Persian biomedical question or text prompt requiring a long-form answer.

**Output**: Generated Persian text response (long-form answer/summary).

## Scoring recipe

```python
def evaluate(pred, refs):
    # BERTScore F1
    R = mean(max(cos_sim(x, y) for y in refs) for x in pred)
    P = mean(max(cos_sim(y, x) for x in pred) for y in refs)
    bert_f1 = 2 * P * R / (P + R)
    # ROUGE-L
    rouge_l = (1 + beta**2) * (R_lcs * P_lcs) / (R_lcs + beta**2 * P_lcs)
    # MoverScore (EMD on IDF-weighted n-gram embeddings)
    moverscore = optimal_transport_cost(pred, refs)
    # BLEURT
    bleurt = bert_model.predict(pred, refs)
    return {'bertscore': bert_f1, 'rouge_l': rouge_l, 'moverscore': moverscore, 'bleurt': bleurt}
```

## Common pitfalls

- ROUGE metrics rely on exact n-gram overlap and may penalize valid semantic paraphrases common in biomedical text.
- BERTScore uses layer 9 of BERT and requires scaling to [0,1] via random text baselines for proper calibration.
- MoverScore computes Earth Mover’s Distance on IDF-weighted n-gram embeddings, making it computationally expensive and sensitive to embedding layer choices.

## Evidence (verbatim from paper)

> Four criteria were used to evaluate the results: BERTScore[[49]], ROUGE[[50]], Mover Score[[51]], and BLEURT[[52]]. The following is a general definition of these criteria.

## Citation

```bibtex
@misc{merzah2025bionpars,
  title={BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining},
  author={Baqer M. Merzah et al. (2025)},
  year={2025},
  note={arXiv:2506.21567}
}
```

- arXiv: 2506.21567