Dualtoken Eval

---
name: dualtoken-eval
description: Evaluates a unified vision tokenizer's capacity to decouple and jointly optimize low-level perceptual reconstruction and high-level semantic understanding. It probes zero-shot classification, cross-modal retrieval, image reconstruction fidelity, and downstream multimodal reasoning capabilities. Use when the user wants to benchmark on ImageNet-1K, Flickr8K, VQAv2, POPE, MME, SEED-IMG, MMBench, MM-Vet, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.14324
  bibtex_key: song2025dualtoken
  confidence: high
---

# dualtoken-eval

> DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies — Wei Song et al. (arXiv:2503.14324, 2025)

## What this evaluates

Evaluates a unified vision tokenizer's capacity to decouple and jointly optimize low-level perceptual reconstruction and high-level semantic understanding. It probes zero-shot classification, cross-modal retrieval, image reconstruction fidelity, and downstream multimodal reasoning capabilities.

## Datasets

- **ImageNet-1K** — total ?; splits: val (-1)
- **Flickr8K** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **SEED-IMG** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: dev (-1)
- **MM-Vet** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Percentage of correctly classified images in zero-shot classification on ImageNet-1K validation set. Computed as (correct predictions / total samples) * 100.
- `R@1` — range: percent
  - Recall at rank 1 for text-to-image and image-to-text retrieval on Flickr8K. Measures the fraction of queries where the correct match appears as the top-ranked result.
- `rFID` — range: other
  - Reconstruction Fréchet Inception Distance. Computes the FID between the distribution of real images and the distribution of reconstructed images from predicted tokens. Lower is better.
- `PSNR` — range: other
  - Peak Signal-to-Noise Ratio in decibels, measuring the ratio between the maximum possible power of a signal and the distorting noise introduced by reconstruction. Higher is better.
- `SSIM` — range: [-1, 1]
  - Structural Similarity Index, measuring perceived change in structural information between original and reconstructed images. Ranges from -1 to 1, where 1 indicates identical images.

## Input / output format

**Input**: Images resized to 384×384 pixels, paired with text prompts, classification labels, or retrieval queries.

**Output**: Discrete semantic and pixel tokens (27×27×8 grid), generated images, or classification/retrieval labels.

## Scoring recipe

```python
acc = sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)
r1 = sum(1 for q in queries if gold_ids[q] in top_1[q]) / len(queries)
rfid = frechet_inception_distance(real_imgs, recon_imgs)
psnr = peak_signal_noise_ratio(real_imgs, recon_imgs)
ssim = structural_similarity(real_imgs, recon_imgs, channel_axis=-1)
```

## Common pitfalls

- Naively combining separate specialized encoders (e.g., MoVQGAN for pixels + SigLIP for semantics) without unified training severely degrades generation quality.
- Using only semantic tokens for downstream tasks underperforms on reasoning benchmarks (MMBench, MME) compared to concatenating semantic and perceptual tokens.
- Classifier-free guidance (CFG) must be fixed at 3.0 for generation evaluation to ensure fair comparison across methods.

## Evidence (verbatim from paper)

> We evaluate our model against widely used vision-language understanding benchmarks, including VQAv2, POPE, MME, SEED-IMG, MMBench, and MM-Vet. For evaluating the semantic capabilities of our unified vision tokenizer, we report the Top-1 accuracy for zero-shot image classification on ImageNet-1K (validation set), along with text-to-image and image-to-text retrieval performance (R@1) on Flickr8K. To evaluate reconstruction capability, we measured reconstruction FID (rFID), PSNR, and SSIM on the ImageNet-1K validation set.

## Citation

```bibtex
@misc{song2025dualtoken,
  title={DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies},
  author={Wei Song et al.},
  year={2025},
  note={arXiv:2503.14324}
}
```

- arXiv: 2503.14324