Nucleotide Transformer Benchmark Eval

---
name: nucleotide-transformer-benchmark-eval
description: Assesses a model's capability to predict various genomic features including histone markers, regulatory annotations, and splice sites. It tests fine-grained sequence understanding and multi-task classification across diverse genomic contexts. Use when the user wants to benchmark on Nucleotide Transformer Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17257
  bibtex_key: [REDACTED]
  confidence: high
---

# nucleotide-transformer-benchmark-eval

> JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model — Duan et al. (2025) (arXiv:2505.17257, 2025)

## What this evaluates

Assesses a model's capability to predict various genomic features including histone markers, regulatory annotations, and splice sites. It tests fine-grained sequence understanding and multi-task classification across diverse genomic contexts.

## Datasets

- **Nucleotide Transformer Benchmark** — total ?; splits: 10-fold CV (-1)

## Metrics

- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient, measuring the quality of binary classifications.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall.
- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly classified sequences.

## Input / output format

**Input**: DNA sequences for histone marker prediction, regulatory annotation, and splice site annotation tasks.

**Output**: Class labels for each specific genomic feature task.

## Scoring recipe

```python
if task in ['histone', 'enhancer']:
    score = mcc(gold, pred)
elif task in ['promoter', 'splice_acceptor', 'splice_donor']:
    score = f1_score(gold, pred)
else:
    score = accuracy(gold, pred)
```

## Common pitfalls

- Metrics vary by task type; using a single metric across all tasks would misrepresent performance.
- The benchmark includes tasks with highly imbalanced classes, making accuracy misleading without MCC or F1.
- Results are averaged over 10-fold CV; error bars represent max-min across 10 seeds.

## Evidence (verbatim from paper)

> Next, we evaluate our model on the Nucleotide Transformer tasks, which include 18 datasets covering histone marker prediction, regulatory annotation prediction, and splice site annotation prediction. Following the evaluation metrics outlined in [nucleotidebenchmark], we perform 10-fold cross-validation for each task... Metrics vary by task: MCC for histone markers and enhancer annotation, F1-score for promoter annotation and splice site acceptor/donor, and accuracy for splice site “all”.

## Citation

```bibtex
@misc{duan2025janusdna,
  title={JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model},
  author={Duan et al. (2025)},
  year={2025},
  note={arXiv:2505.17257}
}
```

- arXiv: 2505.17257