Yochameleon Personalized Eval

---
name: yochameleon-personalized-eval
description: Evaluates a multimodal model's ability to personalize to a few-shot visual concept for both understanding (recognition and QA) and pixel-level image generation. It probes whether learnable soft prompts can capture subject-specific details without catastrophic forgetting, while measuring token efficiency compared to standard prompting. Use when the user wants to benchmark on Yo’LLaVA dataset, or asks about evaluating this task. Reports Recognition Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.20998
  bibtex_key: nguyen2025yochameleon
  confidence: high
---

# yochameleon-personalized-eval

> YoChameleon: Personalized Vision and Language Generation — Nguyen et al. (2025) (arXiv:2504.20998, 2025)

## What this evaluates

Evaluates a multimodal model's ability to personalize to a few-shot visual concept for both understanding (recognition and QA) and pixel-level image generation. It probes whether learnable soft prompts can capture subject-specific details without catastrophic forgetting, while measuring token efficiency compared to standard prompting.

## Datasets

- **Yo’LLaVA dataset** — total 40; splits: test (-1)

## Metrics

- `Recognition Accuracy` **(primary)** — range: [0, 1]
  - Weighted accuracy balancing positive and negative classes to mitigate class imbalance on the 333 positive vs 13,000 negative test set.
- `CLIP Image Similarity Score` — range: [0, 1]
  - Cosine similarity between CLIP image embeddings of the generated output and the reference positive example, averaged over 100 generations per concept.
- `Question Answering Accuracy` — range: [0, 1]
  - Percentage of correct answers on 500 multiple-choice questions (A/B) covering visual and text-based attributes.
- `Facial Similarity Score` — range: [0, 1]
  - ArcFace similarity metric between generated and real images for 10 human concepts.

## Input / output format

**Input**: Recognition: single image + prompt 'Is <sks> in this photo?'. QA: multiple-choice question (A/B) with visual or textual context. Generation: prompt 'A photo of <sks>'.

**Output**: Recognition: 'Yes' or 'No'. QA: 'A' or 'B'. Generation: RGB image.

## Scoring recipe

```python
def compute_metrics(predictions, golds, generated_imgs=None, ref_imgs=None):
    # Recognition: weighted accuracy
    pos_acc = sum(1 for p, g in zip(predictions, golds) if p == g == 'Yes') / sum(1 for g in golds if g == 'Yes')
    neg_acc = sum(1 for p, g in zip(predictions, golds) if p == g == 'No') / sum(1 for g in golds if g == 'No')
    rec_acc = (pos_acc + neg_acc) / 2
    # Generation: CLIP Image Similarity
    clip_sim = sum(clip_cosine_similarity(g, r) for g, r in zip(generated_imgs, ref_imgs)) / len(generated_imgs)
    return {'recognition_accuracy': rec_acc, 'clip_i': clip_sim}
```

## Common pitfalls

- Using standard accuracy instead of the specified weighted accuracy, which would heavily penalize the model on the imbalanced 333 positive vs 13,000 negative test set.
- Confusing the ~1,100 training negative images (used for soft-positive augmentation) with the 13,000 test negative images.
- Comparing token counts directly without accounting for the base model's architecture, as Yo'Chameleon uses 32 latent tokens while baselines use ~64 to ~4k visible tokens.

## Evidence (verbatim from paper)

> In total, there are 333 positive and 13,000 negative images for recognition. During testing, we present a photo and ask the model “Is <sks> in this photo?” The ground-truth answer is either “Yes” or “No”. We use a weighted accuracy metric to balance the positive and negative classes, following the protocol in[[19]]. For question-answering, we provide multiple-choice questions (A or B) with 100 visual and 400 text-based questions. For image generation, we produce 100 images per concept using the prompt “A photo of <sks>” and compute the CLIP Image Similarity Score between the generated images and positive examples.

## Citation

```bibtex
@misc{nguyen2025yochameleon,
  title={YoChameleon: Personalized Vision and Language Generation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2504.20998}
}
```

- arXiv: 2504.20998