Symile Synthetic Eval

---
name: symile-synthetic-eval
description: Probes a model's ability to capture higher-order conditional dependencies between modalities by predicting one modality's representation from two others under varying information dynamics. It specifically tests whether a model can leverage joint information when pairwise mutual information is zero. Use when the user wants to benchmark on Synthetic dataset, or asks about evaluating this task. Reports mean accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.01053
  bibtex_key: saporta2024symile
  confidence: high
---

# symile-synthetic-eval

> Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities — Saporta et al. (2024) (arXiv:2411.01053, 2024)

## What this evaluates

Probes a model's ability to capture higher-order conditional dependencies between modalities by predicting one modality's representation from two others under varying information dynamics. It specifically tests whether a model can leverage joint information when pairwise mutual information is zero.

## Datasets

- **Synthetic dataset** — total ?; splits: test (-1); repo https://github.com/rajesh-lab/symile

## Metrics

- `mean accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted target representations given input pairs. Reported as the mean across 10 bootstrap samples of the test set.

## Input / output format

**Input**: Pair of learned representations $(\mathbf{r}_{\mathbf{a}}, \mathbf{r}_{\mathbf{c}})$ derived from affine linear functions mapping input vectors $\mathbf{a}, \mathbf{c} \in \mathbb{R}^5$.

**Output**: Predicted representation vector $\mathbf{r}_{\mathbf{b}} \in \mathbb{R}^{16}$.

## Scoring recipe

```python
accuracy = 0
for each sample in test_set:
    pred = model.predict(r_a, r_c)
    if pred == r_b:
        accuracy += 1
return accuracy / len(test_set)
```

## Common pitfalls

- Assuming pairwise mutual information is sufficient for prediction; the task explicitly relies on higher-order conditional dependencies that are zero pairwise.
- Not accounting for the bootstrap sampling procedure; results must be averaged over 10 test set resamples to match reported values.

## Evidence (verbatim from paper)

> Mean accuracy is reported across 10 bootstrap samples of the test set. Both models start with a mean accuracy of $0.032\pm 0.001$ (SE) at $\hat{p}\=0$. As $\hat{p}$ increases, Symile’s accuracy progressively climbs, reaching perfect accuracy at $\hat{p}\=1\pm 0.0$ (SE).

## Citation

```bibtex
@misc{saporta2024symile,
  title={Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities},
  author={Saporta et al. (2024)},
  year={2024},
  note={arXiv:2411.01053}
}
```

- arXiv: 2411.01053