Cosmic Symmetry Benchmark Eval

---
name: cosmic-symmetry-benchmark-eval
description: Evaluates the ability of graph neural networks to extract cosmological parameters and local velocity fields from large-scale 3D point clouds of dark matter halos. It probes both global long-range correlation capture (via cosmological parameter regression) and local geometric dependency modeling (via per-node velocity prediction). Use when the user wants to benchmark on Quijote BSQ Point Cloud, or asks about evaluating this task. Reports MSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.20516
  bibtex_key: balla2024cosmic
  confidence: high
---

# cosmic-symmetry-benchmark-eval

> A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing — Balla et al. (2024) (arXiv:2410.20516, 2024)

## What this evaluates

Evaluates the ability of graph neural networks to extract cosmological parameters and local velocity fields from large-scale 3D point clouds of dark matter halos. It probes both global long-range correlation capture (via cosmological parameter regression) and local geometric dependency modeling (via per-node velocity prediction).

## Datasets

- **Quijote BSQ Point Cloud** — total 12384; splits: train (2048), val (512), test (512); repo https://github.com/smsharma/eqnn-jax

## Metrics

- `MSE` **(primary)** — range: other
  - Mean Squared Error between predicted values and ground truth targets. For graph-level tasks, averaged over the two cosmological parameters; for node-level tasks, averaged over all 5000 points and 3 velocity components.

## Input / output format

**Input**: 3D point cloud of 5000 dark matter halo positions, represented as a tensor of shape R^{5000x3}.

**Output**: Graph-level: 2D vector [Omega_m, sigma_8]. Node-level: 3D velocity tensor of shape R^{5000x3}.

## Scoring recipe

```python
def compute_mse(pred, target):
    return np.mean((pred - target) ** 2)

# Graph-level: pred/target shape (2,)
# Node-level: pred/target shape (5000, 3)
graph_mse = compute_mse(pred_graph, target_graph)
node_mse = compute_mse(pred_node, target_node)
```

## Common pitfalls

- The dataset uses only the 5000 most massive halos per simulation, not the full particle set, which alters point cloud density and correlation structure compared to raw N-body outputs.
- The node-level task predicts instantaneous velocity vectors from static positions alone, lacking explicit temporal or dynamical information, making it inherently challenging.
- Cosmological parameters Omega_m and sigma_8 are often degenerate in summary statistics, so independent regression performance can be sensitive to architectural inductive biases rather than just data fitting.

## Evidence (verbatim from paper)

> We consider two benchmark tasks to evaluate the performance of our models: a graph-level and a node-level prediction task. ... We train the models using the mean squared error (MSE) loss between the predicted output and true target parameters. ... We train the models using the MSE loss on the predicted velocities.

## Citation

```bibtex
@misc{balla2024cosmic,
  title={A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing},
  author={Balla et al. (2024)},
  year={2024},
  note={arXiv:2410.20516}
}
```

- arXiv: 2410.20516