Multiid Bench Eval

---
name: multiid-bench-eval
description: Evaluates image generation models on their ability to produce identity-consistent portraits while maintaining controllability over pose, expression, and lighting. It specifically probes the trade-off between accurate identity preservation and the generation of copy-paste artifacts from reference images. Use when the user wants to benchmark on MultiID-Bench, or asks about evaluating this task. Reports face similarity (Sim(G)).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.14975
  bibtex_key: [REDACTED]
  confidence: high
---

# multiid-bench-eval

> WithAnyone: Towards Controllable and ID Consistent Image Generation — Hengyuan Xu et al. (2025) (arXiv:2510.14975, 2025)

## What this evaluates

Evaluates image generation models on their ability to produce identity-consistent portraits while maintaining controllability over pose, expression, and lighting. It specifically probes the trade-off between accurate identity preservation and the generation of copy-paste artifacts from reference images.

## Datasets

- **MultiID-Bench** — total ?; splits: single-person (-1), multi-person (-1); repo https://github.com/doby-xu/WithAnyone

## Metrics

- `face similarity (Sim(G))` **(primary)** — range: [0, 1]
  - Cosine similarity between face embeddings of the generated image and the ground-truth image, measuring identity preservation relative to the target.
- `face similarity (Sim(R))` — range: [0, 1]
  - Cosine similarity between face embeddings of the generated image and the reference image, measuring how closely the output matches the provided ID reference.
- `copy-paste (CP)` — range: [0, 1]
  - A score quantifying the severity of copy-paste artifacts in the generated face, where lower values indicate fewer direct replicative artifacts.
- `CLIP-I` — range: [0, 1]
  - CLIP image similarity score measuring the visual quality and feature alignment of the generated image.
- `CLIP-T` — range: [0, 1]
  - CLIP text-image similarity score measuring alignment between the generated image and the input text prompt.
- `Aes` — range: other
  - Aesthetic quality score of the generated image, typically derived from a dedicated aesthetic predictor.

## Input / output format

**Input**: Text prompt describing the scene, pose, expression, and lighting, plus one or more reference images containing the target identity/identities.

**Output**: Generated image(s) matching the prompt and reference identity/identities.

## Scoring recipe

```python
def compute_metrics(predictions, gold_prompts, reference_images, gt_images):
    gen_emb = get_face_embedding(predictions)
    ref_emb = get_face_embedding(reference_images)
    gt_emb = get_face_embedding(gt_images)
    
    sim_g = cosine_similarity(gen_emb, gt_emb)
    sim_r = cosine_similarity(gen_emb, ref_emb)
    cp_score = compute_copy_paste_artifact_score(predictions, reference_images)
    
    clip_i = compute_clip_image_similarity(predictions)
    clip_t = compute_clip_text_image_similarity(predictions, gold_prompts)
    aes = compute_aesthetic_score(predictions)
    
    return {
        'Sim(G)': sim_g,
        'Sim(R)': sim_r,
        'CP': cp_score,
        'CLIP-I': clip_i,
        'CLIP-T': clip_t,
        'Aes': aes
    }
```

## Common pitfalls

- High face similarity scores can be artificially inflated by models directly copy-pasting reference facial features rather than synthesizing the identity, creating a misleading trade-off curve.
- VLM-based evaluators often fail to distinguish individual identities and instead score non-identity attributes like pose, expression, or background, leading to inaccurate identity fidelity measurements.
- The copy-paste metric may not perfectly align with human perception in all cases, though the paper notes a moderate positive correlation.

## Evidence (verbatim from paper)

> We observe a clear trade-off between face similarity and copy-paste artifacts. As shown in Fig.[5], most methods align closely with a regression curve, where higher face similarity generally coincides with stronger copy-paste. This indicates that many existing models boost measured similarity by directly replicating reference facial features rather than synthesizing the identity.

## Citation

```bibtex
@misc{xu2025withanyone,
  title={WithAnyone: Towards Controllable and ID Consistent Image Generation},
  author={Hengyuan Xu et al. (2025)},
  year={2025},
  note={arXiv:2510.14975}
}
```

- arXiv: 2510.14975