Mmface Dit Eval

---
name: mmface-dit-eval
description: Evaluates multimodal face generation models conditioned on text and spatial inputs (semantic masks or sketches). It probes the model's ability to balance structural priors from spatial conditions with nuanced textual descriptions while maintaining photorealism and semantic alignment. Use when the user wants to benchmark on CelebA-HQ + FFHQ, or asks about evaluating this task. Reports FID.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29029
  bibtex_key: [REDACTED]
  confidence: high
---

# mmface-dit-eval

> MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation — Krishnamurthy et al. (2026) (arXiv:2603.29029, 2026)

## What this evaluates

Evaluates multimodal face generation models conditioned on text and spatial inputs (semantic masks or sketches). It probes the model's ability to balance structural priors from spatial conditions with nuanced textual descriptions while maintaining photorealism and semantic alignment.

## Datasets

- **CelebA-HQ + FFHQ** — total ?; splits: train (-1), test (-1)

## Metrics

- `FID` **(primary)** — range: other
  - Fréchet Inception Distance: measures the Fréchet distance between the multivariate Gaussian distributions of real and generated image features extracted by a pretrained Inception network.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity: computes perceptual distance using deep features from a pretrained network, with lower values indicating higher similarity.
- `SSIM` — range: [0, 1]
  - Multi-scale Structural Similarity Index Measure: evaluates luminance, contrast, and structure similarity between images across multiple scales.
- `CLIP Score` — range: other
  - Cosine similarity between image and text embeddings in the CLIP latent space.
- `CLIP Distance` — range: other
  - Euclidean distance between image and text embeddings in the CLIP latent space.
- `LLM Score` — range: other
  - Semantic consistency score generated by an LLM evaluating alignment between text prompt and generated image.

## Input / output format

**Input**: Text prompt + spatial condition (semantic mask or sketch)

**Output**: Generated face image (RGB)

## Scoring recipe

```python
def compute_metrics(real_imgs, gen_imgs, prompts, masks=None):
    fid = frechet_inception_distance(real_imgs, gen_imgs)
    lpips = learned_perceptual_similarity(real_imgs, gen_imgs)
    ssim = multi_scale_ssim(real_imgs, gen_imgs)
    clip_score = cosine_similarity(clip_encode(gen_imgs), clip_encode(prompts))
    clip_dist = euclidean_distance(clip_encode(gen_imgs), clip_encode(prompts))
    llm_sc = llm_evaluate_semantic_consistency(gen_imgs, prompts)
    acc, miou = 0.0, 0.0
    if masks is not None:
        acc, miou = pixel_accuracy_and_miou(gen_imgs, masks)
    return {'FID': fid, 'LPIPS': lpips, 'SSIM': ssim, 'CLIP_Score': clip_score, 'CLIP_Dist': clip_dist, 'LLM_Sc': llm_sc, 'ACC': acc, 'mIoU': miou}
```

## Common pitfalls

- FID and LPIPS measure perceptual quality but do not guarantee fine-grained attribute alignment or identity preservation.
- CLIP Score/Distance may fail to capture subtle semantic inconsistencies that LLM-based scoring detects.
- Spatial metrics (ACC, mIoU) are only applicable to mask-conditioned generation, not sketch-conditioned.

## Evidence (verbatim from paper)

> We assess performance using a comprehensive suite of metrics. Image realism is measured by Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS). For masks, we evaluate structural integrity with Pixel Accuracy (ACC) and mean Intersection-over-Union (mIoU). Spatial fidelity is further assessed with the multi-scale Structural Similarity Index Measure (SSIM). Finally, we quantify text-image alignment using CLIP Score and Distance, and capture more nuanced semantic consistency with an LLM Score (LLM Sc.).

## Citation

```bibtex
@misc{krishnamurthy2026mmfacedit,
  title={MMFace-DiT: A Dual-Stream Diffusion Transformer for High-Fidelity Multimodal Face Generation},
  author={Krishnamurthy et al. (2026)},
  year={2026},
  note={arXiv:2603.29029}
}
```

- arXiv: 2603.29029