Scendi Score

---
name: scendi-score
description: Evaluates the intrinsic diversity of text-to-image generative models by isolating model-driven variation from prompt-driven variation. It uses CLIP embeddings to construct a joint image-text kernel covariance matrix and applies Schur complement decomposition to remove text influence before computing spectral entropy. Use when the user has predictions and gold and needs to compute Scendi score.
metadata:
  skill_kind: metric
  source_arxiv: 2412.18645
  bibtex_key: ospanov2024scendi
  confidence: high
---

# scendi-score

> Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings — Ospanov et al. (2024) (arXiv:2412.18645, 2024)

## What this evaluates

Evaluates the intrinsic diversity of text-to-image generative models by isolating model-driven variation from prompt-driven variation. It uses CLIP embeddings to construct a joint image-text kernel covariance matrix and applies Schur complement decomposition to remove text influence before computing spectral entropy.

## Datasets

- **Cat breeds dataset** — total ?; splits: test (-1)

## Metrics

- `Scendi score` **(primary)** — range: other (≥ 1)
  - Computes the Schur complement of the image-text kernel covariance matrix to isolate model-driven variation: $\Lambda_I = C_{II} - C_{IT}C_{TT}^{-1}C_{IT}^\top$. Takes the eigenvalues $\lambda_j$ of $\Lambda_I$, computes the trace $\text{Tr}(\Lambda_I)$, and returns $\exp(\sum_j \lambda_j \log(\text{Tr}(\Lambda_I)/\lambda_j))$, which corresponds to the exponential of the conditional Shannon entropy of image clusters given the prompt.

## Input / output format

**Input**: A paired dataset of $n$ text prompts and corresponding generated images $(T_j, I_j)$.

**Output**: A single scalar diversity score.

## Scoring recipe

```python
# 1. Get CLIP embeddings for images (Phi_I) and texts (Phi_T)
# 2. Compute covariance matrices
C_II = (Phi_I.T @ Phi_I) / n
C_IT = (Phi_I.T @ Phi_T) / n
C_TT = (Phi_T.T @ Phi_T) / n
# 3. Compute Schur complement (model-driven component)
Lambda_I = C_II - C_IT @ np.linalg.inv(C_TT) @ C_IT.T
# 4. Compute eigenvalues and trace
eigenvalues = np.linalg.eigvalsh(Lambda_I)
trace_Lambda = np.sum(eigenvalues)
# 5. Compute Scendi score
score = np.exp(np.sum(eigenvalues * np.log(trace_Lambda / eigenvalues)))
return score
```

## Common pitfalls

- Forgetting to subtract the text-driven component ($C_{IT}C_{TT}^{-1}C_{IT}^\top$) before computing eigenvalues, which conflates prompt diversity with model diversity.
- Using raw covariance eigenvalues without accounting for the trace normalization in the entropy formula, leading to scale-dependent scores.
- Applying the metric to unpaired or mismatched image-text datasets, as the Schur complement decomposition strictly requires aligned $(T_j, I_j)$ pairs.

## Evidence (verbatim from paper)

> We define the Schur-Complement-ENtropy Diversity ($\mathrm{Scendi}$) score as follows: $\displaystyle\mathrm{Scendi}(x_{1},..,x_{n};t_{1},..,t_{n})\,:=$ $\displaystyle\,\exp\Bigl{(}\sum_{j\=1}^{d}\lambda^{(\Lambda_{I})}_{j}\log\frac{\mathrm{Tr}(\Lambda_{I})}{\lambda^{(\Lambda_{I})}_{j}}\Bigr{)}$ where $\lambda^{(\Lambda_{I})}_{j}$ denotes the $j$th eigenvalue of matrix $\Lambda_{I}$ and $\mathrm{Tr}(\Lambda_{I})\=\sum_{j\=1}^{d}\lambda^{(\Lambda_{I})}_{j}$ is the sum of the eigenvalues.

## Citation

```bibtex
@misc{ospanov2024scendi,
  title={Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings},
  author={Ospanov et al. (2024)},
  year={2024},
  note={arXiv:2412.18645}
}
```

- arXiv: 2412.18645