Cmedteb Eval

---
name: cmedteb-eval
description: Evaluates Chinese medical text embedding models across retrieval, reranking, and semantic textual similarity tasks. It probes the model's ability to capture domain-specific semantic alignment while measuring the trade-off between retrieval accuracy and inference efficiency. Use when the user wants to benchmark on CMedTEB, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10937
  bibtex_key: [REDACTED]
  confidence: high
---

# cmedteb-eval

> CMedTEB & CARE: Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders — Jiang et al. (2026) (arXiv:2604.10937, 2026)

## What this evaluates

Evaluates Chinese medical text embedding models across retrieval, reranking, and semantic textual similarity tasks. It probes the model's ability to capture domain-specific semantic alignment while measuring the trade-off between retrieval accuracy and inference efficiency.

## Datasets

- **CMedTEB** — total ?; splits: CMed v1 (-1), CMed v2 (-1), Retrieval (-1), Rerank (-1), STS (-1); repo https://github.com/PhilipGAQ/CARE

## Metrics

- `MAP@10` — range: [0, 1]
  - Mean Average Precision at 10. Computes the average precision of retrieved documents up to rank 10, averaged over all queries.
- `nDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at 10. Measures ranking quality by discounting gains logarithmically based on position, normalized by the ideal ranking.
- `Pearson` — range: [-1, 1]
  - Pearson correlation coefficient measuring linear correlation between predicted similarity scores and ground-truth human ratings.
- `Avg` **(primary)** — range: [0, 1]
  - Arithmetic mean of MAP@10 (CMed v1), MAP@10 (CMed v2), nDCG@10 (Retrieval), MAP@10 (Rerank), and Pearson (STS).

## Input / output format

**Input**: Query-document pairs for embedding generation; queries with candidate document sets for retrieval/reranking; sentence pairs for STS.

**Output**: Dense embedding vectors for queries and documents; ranked lists of document IDs/scores for retrieval/reranking; similarity scores for STS.

## Scoring recipe

```python
def map_at_10(retrieved, relevant):
    hits = 0; scores = []
    for i, doc in enumerate(retrieved[:10]):
        if doc in relevant:
            hits += 1
            scores.append(hits / (i + 1))
    return sum(scores) / len(relevant) if relevant else 0

def ndcg_at_10(retrieved, relevant):
    dcg = sum(1 / log2(i + 2) for i, doc in enumerate(retrieved[:10]) if doc in relevant)
    ideal = sum(1 / log2(i + 2) for i in range(min(len(relevant), 10)))
    return dcg / ideal if ideal > 0 else 0

pearson = scipy.stats.pearsonr(pred_scores, gold_scores)[0]
avg = (map_v1 + map_v2 + ndcg + map_rerank + pearson) / 5
```

## Common pitfalls

- Confusing total model parameters with online inference parameters; the benchmark emphasizes that asymmetric models only count the lightweight query encoder for latency/cost.
- Averaging metrics across different tasks (retrieval, reranking, STS) without noting that STS uses Pearson correlation (0-1 range) while MAP/nDCG are percentages, which can skew the 'Avg' interpretation.
- Assuming symmetric baselines are directly comparable in latency; the paper explicitly notes that symmetric giants incur prohibitive computational costs despite higher accuracy.

## Evidence (verbatim from paper)

> Table 2 presents the evaluation of CARE series on the CMedTEB benchmark, alongside strong open-source baselines. We observe two key findings: (1) CARE establishes a new state of the art: the 0.3B-4B variant achieves an average score of 78.13, and the 0.3B-8B variant reaches 78.94, surpassing the strongest baseline gte-Qwen2-1.5B-instruct (77.61, a decoder-only model), despite using a much smaller query encoder.

## Citation

```bibtex
@misc{jiang2026cmedteb,
  title={CMedTEB & CARE: Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.10937}
}
```

- arXiv: 2604.10937