Danqing Eval

---
name: danqing-eval
description: Evaluates the quality of Chinese vision-language pre-training datasets by measuring downstream performance on cross-modal retrieval and multimodal reasoning benchmarks after continual pre-training. Use when the user wants to benchmark on Flickr30K-CN, MSCOCO-CN, MUGE, DCI-CN, DOCCI-CN, or asks about evaluating this task. Reports R@1/5/10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.10305
  bibtex_key: shen2026danqing
  confidence: high
---

# danqing-eval

> DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset — Shen et al. (2026) (arXiv:2601.10305, 2026)

## What this evaluates

Evaluates the quality of Chinese vision-language pre-training datasets by measuring downstream performance on cross-modal retrieval and multimodal reasoning benchmarks after continual pre-training.

## Datasets

- **Flickr30K-CN** — total ?; splits: test (-1)
- **MSCOCO-CN** — total ?; splits: test (-1)
- **MUGE** — total ?; splits: test (-1)
- **DCI-CN** — total ?; splits: test (-1)
- **DOCCI-CN** — total ?; splits: test (-1)

## Metrics

- `R@1/5/10` **(primary)** — range: percent
  - Recall@K measures the fraction of relevant items retrieved within the top K results. Computed separately for text-to-image and image-to-text directions.

## Input / output format

**Input**: Image and corresponding text caption for retrieval; image and question for LMM benchmarks.

**Output**: Retrieval ranking scores (R@1, R@5, R@10) per direction; accuracy scores for LMM benchmarks.

## Scoring recipe

```python
def compute_recall_at_k(retrieved_top_k, relevant_set, k):
    hits = sum(1 for idx in retrieved_top_k[:k] if idx in relevant_set)
    return hits / len(relevant_set)

def evaluate_retrieval(model, test_pairs):
    metrics = {'R@1': 0.0, 'R@5': 0.0, 'R@10': 0.0}
    for query, gold_idx in test_pairs:
        scores = model.encode(query)
        top_k = get_top_k_indices(scores, k=10)
        for k in [1, 5, 10]:
            metrics[f'R@{k}'] += compute_recall_at_k(top_k, {gold_idx}, k)
    for k in [1, 5, 10]:
        metrics[f'R@{k}'] /= len(test_pairs)
    return metrics
```

## Common pitfalls

- Competitor datasets (Zero, TaiSu) are randomly sampled to 100M pairs to match DanQing's size, which may introduce sampling variance and affect reproducibility.
- Long-caption retrieval is constrained to a 64-token context length, potentially truncating relevant information and lowering recall scores.
- Metrics are reported separately for text-to-image and image-to-text directions, but averages are sometimes combined without clear weighting or direction specification.

## Evidence (verbatim from paper)

> Table 3: Cross-modal retrieval performance on short-caption datasets for models pretrained on various large-scale Chinese image-text datasets. * indicates random sampling of 100 million image-text pairs. The best and second-best results are highlighted in bold and underlined, respectively. R@1 R@5 R@10

## Citation

```bibtex
@misc{shen2026danqing,
  title={DanQing: An Up-to-Date Large-Scale Chinese Vision-Language Pre-training Dataset},
  author={Shen et al. (2026)},
  year={2026},
  note={arXiv:2601.10305}
}
```

- arXiv: 2601.10305