Mets Cov Eval

---
name: mets-cov-eval
description: Evaluates named entity recognition capabilities on biomedical and social media text. It probes the model's ability to identify and classify domain-specific entities (e.g., diseases, drugs, vaccines) in both informal tweets and formal scientific abstracts under fully-supervised and few-shot learning conditions. Use when the user wants to benchmark on METS-CoV, BioRED, or asks about evaluating this task. Reports Micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.04001
  bibtex_key: zhang2025named
  confidence: high
---

# mets-cov-eval

> Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation — Zhang et al. (2025) (arXiv:2510.04001, 2025)

## What this evaluates

Evaluates named entity recognition capabilities on biomedical and social media text. It probes the model's ability to identify and classify domain-specific entities (e.g., diseases, drugs, vaccines) in both informal tweets and formal scientific abstracts under fully-supervised and few-shot learning conditions.

## Datasets

- **METS-CoV** — total 7000; splits: test (-1)
- **BioRED** — total 600; splits: test (-1)

## Metrics

- `Micro F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall calculated globally across all entity types. Precision = TP / (TP + FP), Recall = TP / (TP + FN), Micro F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: Raw text instances (tweets or PubMed abstracts) with implicit or explicit entity boundaries to be predicted.

**Output**: Predicted entity spans with corresponding labels (e.g., BIO-tagged sequence or list of (start, end, type) tuples).

## Scoring recipe

```python
def compute_micro_f1(preds, golds):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        tp += len(set(p) & set(g))
        fp += len(set(p) - set(g))
        fn += len(set(g) - set(p))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
```

## Common pitfalls

- Micro F1 aggregates across all entity types, which can mask poor performance on rare or domain-specific classes like 'vaccine' or 'chemical entity'.
- Few-shot settings use fixed k-shot splits (5, 10, 20) without specifying random seeds, leading to potential variance in reported scores.
- The paper reports per-entity-type F1 in tables but only highlights the average/micro F1 in text, which may obscure class imbalance effects.

## Evidence (verbatim from paper)

> Micro F1 scores are used as our evaluation metrics. We set up 5-shot, 10-shot, and 20-shot settings for few-shot experiments.

## Citation

```bibtex
@misc{zhang2025named,
  title={Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.04001}
}
```

- arXiv: 2510.04001