Causalverse Eval

---
name: causalverse-eval
description: Probes the ability of causal representation learning (CRL) models to recover ground-truth latent variables from high-fidelity visual simulations. It evaluates both component-wise and block-wise identifiability under realistic conditions where theoretical assumptions may be violated. Use when the user wants to benchmark on CausalVerse, or asks about evaluating this task. Reports Mean Correlation Coefficient (MCC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.14049
  bibtex_key: chen2025causalverse
  confidence: high
---

# causalverse-eval

> CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations — Chen et al. (2025) (arXiv:2510.14049, 2025)

## What this evaluates

Probes the ability of causal representation learning (CRL) models to recover ground-truth latent variables from high-fidelity visual simulations. It evaluates both component-wise and block-wise identifiability under realistic conditions where theoretical assumptions may be violated.

## Datasets

- **CausalVerse** — total 200000; splits: test (-1)

## Metrics

- `Mean Correlation Coefficient (MCC)` **(primary)** — range: [0, 1]
  - Computes Pearson correlations between ground-truth latents Z and estimated latents Z_hat. Selects an injective matching pi maximizing the sum of absolute correlations, then averages them: MCC = (1/D) * sum(|corr(Z_i, Z_hat_{pi(i)})|).
- `Coefficient of Determination (R^2)` — range: other
  - Measures block-wise identifiability by regressing ground-truth block z_b from estimated block z_hat_b. Formula: R^2 = 1 - Var(z_b - f(z_hat_b)) / Var(z_b), where f is the best linear or non-linear predictor.
- `Over-completed MCC` — range: [0, 1]
  - Variant of MCC for over-complete settings (D_hat > D). Selects the top D estimated variables that best match ground truth, then applies standard MCC computation over this subset.

## Input / output format

**Input**: Static images or video frames from configurable high-fidelity simulations. Models receive raw visual data without ground-truth latent labels during unsupervised training.

**Output**: Estimated latent vectors Z_hat in R^{D_hat} (or block-wise estimates z_hat_b) produced by the model's encoder.

## Scoring recipe

```python
def compute_mcc(Z, Z_hat):
    corr_matrix = pearson_corr(Z, Z_hat) # D x D_hat
    best_perm = argmax_permutation(corr_matrix, metric=abs)
    return mean(abs(corr_matrix[range(D), best_perm]))

def compute_r2(Z_b, Z_hat_b):
    f = fit_regression(Z_b, Z_hat_b) # linear or non-linear
    return 1 - var(Z_b - f(Z_hat_b)) / var(Z_b)
```

## Common pitfalls

- Assuming CRL methods satisfy theoretical assumptions (e.g., sufficient change, sparsity) when applied to realistic simulated data, leading to misleadingly low MCC scores.
- Confusing component-wise identifiability (MCC) with block-wise identifiability (R^2), as methods may excel at one metric while failing at the other.
- Interpreting negative R^2 values as implementation errors rather than valid indicators that the model performs worse than predicting the mean.

## Evidence (verbatim from paper)

> To evaluate both component-wise and block-wise identifiability in CausalVerse, we adopt three metrics, including the Mean Correlation Coefficient (MCC), the coefficient of determination $R^{2}$, and the over-completed MCC. Specifically, let $Z\in\mathbb{R}^{D}$ be the ground-truth latent vector and $\widehat{Z}\in\mathbb{R}^{\widehat{D}}$ the estimated vector, we have: Mean Correlation Coefficient (MCC): To calculate MCC, we first compute the Pearson correlations $R_{ij}=\mathrm{corr}(Z_{i},\widehat{Z}_{j}),$ then select an injective matching $\pi:{1,\dots,D}\to{1,\dots,\widehat{D}}$ maximizing $\sum_{i=1}^{D}|R_{i,\pi(i)}|$. Finally, the MCC value is defined as $\mathrm{MCC}=\frac{1}{D}\sum_{i=1}^{D}\bigl|R_{i,\pi(i)}\bigr|.$

## Citation

```bibtex
@misc{chen2025causalverse,
  title={CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2510.14049}
}
```

- arXiv: 2510.14049