Microbiorel Re Eval

---
name: microbiorel-re-eval
description: Evaluates generative and discriminative models on document-level relation extraction in the microbiome domain. It probes the ability to correctly classify pairwise relations between biomedical entities (species, diseases, chemicals, etc.) under a low-resource setting. Use when the user wants to benchmark on MicrobioRel, or asks about evaluating this task. Reports Weighted F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.08647
  bibtex_key: [REDACTED]
  confidence: high
---

# microbiorel-re-eval

> Summarization for Generative Relation Extraction in the Microbiome Domain — El Khettari et al. (2025) (arXiv:2506.08647, 2025)

## What this evaluates

Evaluates generative and discriminative models on document-level relation extraction in the microbiome domain. It probes the ability to correctly classify pairwise relations between biomedical entities (species, diseases, chemicals, etc.) under a low-resource setting.

## Datasets

- **MicrobioRel** — total 1994; splits: (unstated); repo https://github.com/Stan8/MicrobioRel-dataset

## Metrics

- `Weighted F1-score` **(primary)** — range: [0, 1]
  - Weighted average of per-class F1 scores, where weights are proportional to class support (number of instances). Computed as Σ(support_i / total_support × F1_i). Only exact matches with gold labels count as correct predictions.

## Input / output format

**Input**: Paragraphs containing at least two named entities (species, diseases, chemicals, mutations, genes, cell lines). The model receives the text context and entity pairs to predict the relation type.

**Output**: A single relation class label from the predefined schema (or 'None'). For the summarization step, outputs follow the format 'Entity1 Relation Entity2' in a single sentence.

## Scoring recipe

```python
def compute_weighted_f1_exact_match(preds, golds, classes):
    tp = {c: 0 for c in classes}
    fp = {c: 0 for c in classes}
    fn = {c: 0 for c in classes}
    support = {c: 0 for c in classes}
    for p, g in zip(preds, golds):
        support[g] += 1
        if p == g:
            tp[g] += 1
        else:
            fp[p] += 1
            fn[g] += 1
    total = sum(support.values())
    weighted_f1 = 0.0
    for c in classes:
        if support[c] > 0:
            prec = tp[c] / (tp[c] + fp[c]) if (tp[c] + fp[c]) > 0 else 0
            rec = tp[c] / (tp[c] + fn[c]) if (tp[c] + fn[c]) > 0 else 0
            f1_c = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
            weighted_f1 += (support[c] / total) * f1_c
    return weighted_f1
```

## Common pitfalls

- LLM outputs often contain extra text; post-processing is required to extract only the predicted relation class before scoring.
- Partial matches or substring extractions do not count; predictions must exactly match the gold relation label.
- Weighted F1 is calculated by averaging per-class F1 scores, not by computing weighted precision and recall first.

## Evidence (verbatim from paper)

> Evaluation of the RE task is carried out using weighted precision, recall, and F1-score. Under an exact match criterion, a prediction is considered correct only if it matches the gold relation label. ... In the weighted setting, the precision, recall, and F1-score are computed as the weighted averages of the corresponding per-class scores, where the weights are based on the support of each class (see Equation 1). Note that weighted F1 is derived from individual per-class F1 scores, not from weighted precision and recall. Metrics are computed after post-processing the LLM outputs to remove text beyond the predicted relation class.

## Citation

```bibtex
@misc{elkhettari2025summarization,
  title={Summarization for Generative Relation Extraction in the Microbiome Domain},
  author={El Khettari et al. (2025)},
  year={2025},
  note={arXiv:2506.08647}
}
```

- arXiv: 2506.08647