Causal3d Eval

---
name: causal3d-eval
description: This benchmark evaluates a model's ability to discover and reason about causal structures from visual and tabular data. It probes whether models can infer correct causal graphs from observational data, learn disentangled representations from images, and perform valid causal interventions with limited visual samples. Use when the user wants to benchmark on Causal3D, or asks about evaluating this task. Reports correctness of inferred causal structures.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.04852
  bibtex_key: liu2025causal3d
  confidence: medium
---

# causal3d-eval

> CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data — Liu et al. (2025) (arXiv:2503.04852, 2025)

## What this evaluates

This benchmark evaluates a model's ability to discover and reason about causal structures from visual and tabular data. It probes whether models can infer correct causal graphs from observational data, learn disentangled representations from images, and perform valid causal interventions with limited visual samples.

## Datasets

- **Causal3D** — total 190000; splits: test (-1)

## Metrics

- `correctness of inferred causal structures` **(primary)** — range: [0, 1]
  - Measures the accuracy of the predicted causal graph against the ground truth DAG. Evaluated by comparing inferred edges and causal mechanisms across datasets.
- `intervention consistency` — range: [0, 1]
  - Assesses whether intervened images remain consistent with the underlying causal relations after applying the do-operator to a specific variable.

## Input / output format

**Input**: Tabular data containing variable values for causal discovery; or multi-perspective 3D rendered images (with/without real/virtual backgrounds) for representation learning and few-shot intervention tasks.

**Output**: Inferred causal graph/structure (for discovery tasks); or intervened/generated images reflecting the manipulated causal variables (for representation and intervention tasks).

## Scoring recipe

```python
def score_causal_discovery(predictions, ground_truth):
    pred_edges = set(predictions.edges)
    gt_edges = set(ground_truth.edges)
    correct = len(pred_edges.intersection(gt_edges))
    return correct / len(gt_edges)

def score_intervention(intervened_images, ground_truth_relations):
    consistent_count = 0
    for img in intervened_images:
        if check_causal_consistency(img, ground_truth_relations):
            consistent_count += 1
    return consistent_count / len(intervened_images)
```

## Common pitfalls

- Models may overfit to specific backgrounds or camera views rather than learning invariant causal relations.
- Causal discovery from few images is highly sensitive to initialization and may fail to converge without strong inductive biases.
- The benchmark mixes linear and nonlinear causal mechanisms, which can cause standard linear causal discovery algorithms to fail unexpectedly.

## Evidence (verbatim from paper)

> These tasks include: Causal discovery from tabular data... evaluated based on the correctness of inferred causal structures across various datasets and underlying causal mechanisms. ... Causal representation learning from images... Evaluation is based on generated images after intervening on learned representations, assessing whether they accurately reflect the corresponding causal variables and causal relations. ... Causal discovery & intervention from few images... Intervention evaluation is based on whether the intervened images still remain consistent with the underlying causal relations.

## Citation

```bibtex
@misc{liu2025causal3d,
  title={CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2503.04852}
}
```

- arXiv: 2503.04852