Hybridna Eval

---
name: hybridna-eval
description: Evaluates the capability of DNA foundation models to perform short-range and long-range genomic understanding tasks, as well as their ability to generate biologically plausible cis-regulatory elements. It probes sequence classification, variant effect prediction, and generative design across multiple species and cell types. Use when the user wants to benchmark on GUE, BEND, LRB, CRE (regLM), or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.10807
  bibtex_key: ma2025hybridna
  confidence: high
---

# hybridna-eval

> HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model — Mingqian Ma et al. (2025) (arXiv:2502.10807, 2025)

## What this evaluates

Evaluates the capability of DNA foundation models to perform short-range and long-range genomic understanding tasks, as well as their ability to generate biologically plausible cis-regulatory elements. It probes sequence classification, variant effect prediction, and generative design across multiple species and cell types.

## Datasets

- **GUE** — total 28; splits: test (-1)
- **BEND** — total 3; splits: test (-1)
- **LRB** — total 2; splits: test (-1)
- **CRE (regLM)** — total 670000; splits: train (670000), test (-1)

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthews Correlation Coefficient; measures the quality of binary/multiclass classifications considering true/false positives and negatives.
- `F-1 score` — range: [0, 1]
  - Harmonic mean of precision and recall; used specifically for the Covid Variant Classification task.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve; measures the ability of the model to distinguish between classes across all classification thresholds.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall Curve; evaluates performance on imbalanced datasets by focusing on positive class predictions.
- `Top-1 activity` — range: other
  - The highest predicted enhancer activity score among the generated sequences for a given cell type.
- `Mean activity` — range: other
  - The average of the top 100 predicted enhancer activity scores for a given cell type.
- `Diversity` — range: other
  - The mean of pair-wise edit distances of the top 100 predicted sequences, measuring the overall diversity of high-quality generated sequences.

## Input / output format

**Input**: DNA sequences (70–131,072 bp) with task-specific prompts or labels. For generation tasks, prompt tokens specifying cell type and activity strength (e.g., 3-digit labels for enhancer activity).

**Output**: Class probabilities or predicted labels for classification tasks; generated DNA sequences (200 bp) for CRE tasks.

## Scoring recipe

```python
# Classification
mcc = matthews_corrcoef(y_true, y_pred)
f1 = f1_score(y_true, y_pred, average='macro')
auroc = roc_auc_score(y_true, y_prob)
auprc = average_precision_score(y_true, y_prob)
# Generation
seqs = beam_search_decode(model, prompts, num=600)
scores = external_scoring_model.predict(seqs)
top1 = max(scores)
mean_act = mean(scores[:100])
diversity = mean(pairwise_edit_distance(s1, s2) for s1, s2 in combinations(seqs[:100], 2))
```

## Common pitfalls

- Decoder-only models require specific embedding extraction strategies (last token hidden state vs. mean sequence hidden state) which drastically changes performance and must match the benchmark's original setting.
- Fine-tuning protocols (learning rates, epochs, frozen vs. full parameter training) must strictly follow the original benchmark settings to ensure fair comparison.
- Zero-shot vs. fine-tune evaluation modes for LRB tasks are often confused, leading to incorrect metric reporting.

## Evidence (verbatim from paper)

> Following the same setting as DNABERT-2, we use metrics of Matthews Correlation Coefficient (MCC) for all tasks, except for the Covid task, where we use the F-1 score according to the GUE dataset’s original setting.

## Citation

```bibtex
@misc{ma2025hybridna,
  title={HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model},
  author={Mingqian Ma et al. (2025)},
  year={2025},
  note={arXiv:2502.10807}
}
```

- arXiv: 2502.10807