Faceid 6m Eval

---
name: faceid-6m-eval
description: This evaluation protocol assesses the effectiveness of a large-scale dataset for training conditional diffusion models in FaceID customization. It probes the model's ability to preserve facial identity from a reference image while adhering to textual prompts and maintaining overall image quality. Use when the user wants to benchmark on COCO2017, Unsplash-50, or asks about evaluating this task. Reports Face Sim.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07091
  bibtex_key: wang2025faceid6m
  confidence: high
---

# faceid-6m-eval

> FaceID-6M: A Large-Scale, Open-Source FaceID Customization Dataset — Wang et al. (2025) (arXiv:2503.07091, 2025)

## What this evaluates

This evaluation protocol assesses the effectiveness of a large-scale dataset for training conditional diffusion models in FaceID customization. It probes the model's ability to preserve facial identity from a reference image while adhering to textual prompts and maintaining overall image quality.

## Datasets

- **COCO2017** — total 500; splits: test (500)
- **Unsplash-50** — total 50; splits: test (50)

## Metrics

- `Face Sim` **(primary)** — range: [0, 1]
  - Calculates the cosine similarity between the face embedding of the input reference image and the face embedding extracted from the generated image. Provides a direct estimate of facial identity preservation.
- `CLIP-T` — range: [0, 1]
  - Measures the cosine similarity between the text embedding of the input prompt and the image embedding of the generated image. Evaluates the model's ability to follow textual prompts.
- `CLIP-I` — range: [0, 1]
  - Measures the cosine similarity between the image embedding of the original reference face and the image embedding of the generated image. Assesses general image-level similarity after identity insertion.

## Input / output format

**Input**: A reference face image and a natural language text prompt describing the desired image content.

**Output**: A single generated RGB image that incorporates the visual features of the reference face while matching the textual description.

## Scoring recipe

```python
def score(predictions, gold):
    input_face, prompt, orig_face = gold
    face_sim = cosine_similarity(face_encoder(input_face), face_encoder(predictions))
    clip_t = cosine_similarity(text_encoder(prompt), image_encoder(predictions))
    clip_i = cosine_similarity(image_encoder(orig_face), image_encoder(predictions))
    return {'Face Sim': face_sim, 'CLIP-T': clip_t, 'CLIP-I': clip_i}
```

## Common pitfalls

- Using the full 5,000-image COCO2017 split instead of the manually curated 500 pairs, which introduces many non-face samples that skew FaceID fidelity metrics.
- Interpreting CLIP-I as a measure of facial identity; it actually measures general image-level similarity between the original reference and generated image, not necessarily preserving specific facial features.
- Comparing human evaluation scores (0–5 Likert scale) directly with automated cosine similarity scores (0–1 range) without normalization or clear separation of evaluation modalities.

## Evidence (verbatim from paper)

> For evaluation metrics, we use the following: (1) Face Sim, which calculates the FaceID cosine similarity between the input face and the face extracted from the generated image, providing a direct estimate of the difference between the generated and input faces. (2) CLIP-T *[[22]]*, which evaluates the model’s ability to follow prompts; and (3) CLIP-I *[[22]]*, which measures the CLIP image similarity between the original image and the image after FaceID insertion.

## Citation

```bibtex
@misc{wang2025faceid6m,
  title={FaceID-6M: A Large-Scale, Open-Source FaceID Customization Dataset},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.07091}
}
```

- arXiv: 2503.07091