Soar Rna Eval

---
name: soar-rna-eval
description: Evaluates large language models on zero-shot and chain-of-thought cell type annotation tasks using single-cell RNA-seq gene expression profiles. It probes the models' ability to translate structured genomic data into textual descriptions and accurately predict cell type labels without fine-tuning. Use when the user wants to benchmark on SOAR-RNA, or asks about evaluating this task. Reports Average BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.02915
  bibtex_key: [REDACTED]
  confidence: medium
---

# soar-rna-eval

> Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data — Liu et al. (2024) (arXiv:2412.02915, 2024)

## What this evaluates

Evaluates large language models on zero-shot and chain-of-thought cell type annotation tasks using single-cell RNA-seq gene expression profiles. It probes the models' ability to translate structured genomic data into textual descriptions and accurately predict cell type labels without fine-tuning.

## Datasets

- **SOAR-RNA** — total ?; splits: test (-1)

## Metrics

- `BLEU-1` — range: [0, 100]
  - Standard n-gram overlap score measuring unigram precision between predicted and reference cell type names. Scaled to [0, 100].
- `BLEU-2` — range: [0, 100]
  - Standard n-gram overlap score measuring bigram precision between predicted and reference cell type names. Scaled to [0, 100].
- `Average BLEU` **(primary)** — range: [0, 100]
  - Arithmetic mean of BLEU-1 and BLEU-2 scores. Scaled to [0, 100].

## Input / output format

**Input**: Textual description of the top k differentially expressed genes (sorted by p-value) and associated cell metadata (e.g., tissue type), formatted as a question: 'Given the following markers [genes], what is the specific cell type in [metadata] corresponding to these markers?'

**Output**: A single predicted cell type name, parsed from the LLM's response.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    bleu_1 = corpus_bleu(predictions, references, weights=(1, 0, 0, 0)) * 100
    bleu_2 = corpus_bleu(predictions, references, weights=(0, 1, 0, 0)) * 100
    avg_bleu = (bleu_1 + bleu_2) / 2
    return {'BLEU-1': bleu_1, 'BLEU-2': bleu_2, 'Average BLEU': avg_bleu}
```

## Common pitfalls

- LLMs may generate verbose chain-of-thought reasoning or extra text instead of just the cell type name, requiring strict parsing rules to extract the final answer.
- BLEU scores are sensitive to exact string matching; minor variations in cell type nomenclature (e.g., 'T cell' vs 'T-cell') can artificially lower scores despite biological correctness.
- The top-k gene selection and p-value thresholds are not explicitly standardized in the prompt template, which may introduce variability in input quality across datasets.

## Evidence (verbatim from paper)

> Table 1: The BLEU evaluation of cell type annotation results on the SOAR-RNA benchmark using the zero-shot and zero-shot chain-of-thought (CoT) prompting strategies to prompt LLMs respectively.

## Citation

```bibtex
@misc{liu2024singlecellomicsarena,
  title={Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.02915}
}
```

- arXiv: 2412.02915