Textme Eval

---
name: textme-eval
description: Evaluates zero-shot cross-modal retrieval and classification across six modalities (image, video, audio, 3D, X-ray, molecules) using a text-only expansion framework. It probes whether unpaired text descriptions can bridge the geometric modality gap to align diverse modalities into a unified LLM embedding space without paired supervision. Use when the user wants to benchmark on COCO, Flickr30k, MSR-VTT, MSVD, DiDeMo, AudioCaps, Clotho, DrugBank, AudioSet, ESC-50, ModelNet40, ScanObjectNN, RSNA, or asks about evaluating this task. Reports Recall@k (R@k).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.03098
  bibtex_key: hong2026textme
  confidence: high
---

# textme-eval

> TextME: Bridging Unseen Modalities Through Text Descriptions — Soyeon Hong et al. (arXiv:2602.03098, 2026)

## What this evaluates

Evaluates zero-shot cross-modal retrieval and classification across six modalities (image, video, audio, 3D, X-ray, molecules) using a text-only expansion framework. It probes whether unpaired text descriptions can bridge the geometric modality gap to align diverse modalities into a unified LLM embedding space without paired supervision.

## Datasets

- **COCO** — total ?; splits: test (-1)
- **Flickr30k** — total ?; splits: test (-1)
- **MSR-VTT** — total ?; splits: test (-1)
- **MSVD** — total ?; splits: test (-1)
- **DiDeMo** — total ?; splits: test (-1)
- **AudioCaps** — total ?; splits: test (-1)
- **Clotho** — total ?; splits: test (-1)
- **DrugBank** — total ?; splits: test (-1)
- **AudioSet** — total ?; splits: test (-1)
- **ESC-50** — total ?; splits: test (-1)
- **ModelNet40** — total ?; splits: test (-1)
- **ScanObjectNN** — total ?; splits: test (-1)
- **RSNA** — total ?; splits: test (-1)

## Metrics

- `Recall@k (R@k)` **(primary)** — range: percent
  - Percentage of queries where the ground-truth item appears in the top-k retrieved results based on cosine similarity in the shared embedding space.
- `MRR@k` — range: percent
  - Mean Reciprocal Rank for molecule retrieval, averaging the inverse rank of the first correct match across queries.
- `Top-k accuracy` — range: percent
  - Percentage of correctly classified instances where the predicted label matches the ground truth within the top-k predictions.
- `Performance Preservation Ratio (PPR)` — range: percent
  - Relative metric calculated as (TextME score / Pretrained encoder score) × 100%, measuring how much of the original encoder's capability is retained after text-only expansion.

## Input / output format

**Input**: Text descriptions and unpaired modality instances (images, videos, audio, 3D models, X-rays, molecules) independently projected into a shared LLM embedding space.

**Output**: Ranked list of retrieved items for retrieval tasks, or predicted class labels for classification tasks.

## Scoring recipe

```python
def compute_recall_at_k(sim_matrix, k):
    ranks = np.argsort(-sim_matrix, axis=1)
    correct = np.any(ranks[:, :k] == np.arange(sim_matrix.shape[0]), axis=1)
    return np.mean(correct) * 100

def compute_mrr(sim_matrix):
    ranks = np.argsort(-sim_matrix, axis=1)
    ranks_correct = np.where(ranks == np.arange(sim_matrix.shape[0])[:, None])[1]
    return np.mean(1.0 / (ranks_correct + 1)) * 100

def compute_topk_accuracy(preds, labels, k=1):
    correct = np.sum(np.isin(preds, labels))
    return (correct / len(labels)) * 100

def compute_ppr(method_score, pretrained_score):
    return (method_score / pretrained_score) * 100
```

## Common pitfalls

- Assuming paired modality-text data is required for training; the protocol explicitly uses only unpaired text descriptions (100K per modality).
- Interpreting PPR as absolute performance rather than a relative measure of preserved pretrained encoder capability.
- Applying offset correction blindly without verifying gap consistency, which can degrade performance for modalities like molecules.

## Evidence (verbatim from paper)

> We report Recall@k (R@k) for retrieval, MRR@k for molecule retrieval following Liu et al. (2023), and Top-k accuracy for classification. We define Performance Preservation Ratio (PPR) as the percentage of pretrained encoder performance retained by our method: PPR = (TextME score / Pretrained score) × 100%.

## Citation

```bibtex
@misc{hong2026textme,
  title={TextME: Bridging Unseen Modalities Through Text Descriptions},
  author={Soyeon Hong et al.},
  year={2026},
  note={arXiv:2602.03098}
}
```

- arXiv: 2602.03098