Synlogic Eval

---
name: synlogic-eval
description: Evaluates the logical reasoning and cross-domain generalization capabilities of models trained with verifiable synthetic data. It measures accuracy on mathematical, coding, and logical reasoning benchmarks using a multi-sample generation and verification protocol. Use when the user wants to benchmark on MATH 500, AIME 2024, AMC 2023, LiveCodeBench, SynLogic coding validation split, or asks about evaluating this task. Reports avg@8.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19641
  bibtex_key: liu2025synlogic
  confidence: medium
---

# synlogic-eval

> SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond — Liu et al. (2025) (arXiv:2505.19641, 2025)

## What this evaluates

Evaluates the logical reasoning and cross-domain generalization capabilities of models trained with verifiable synthetic data. It measures accuracy on mathematical, coding, and logical reasoning benchmarks using a multi-sample generation and verification protocol.

## Datasets

- **MATH 500** — total ?; splits: test (-1)
- **AIME 2024** — total ?; splits: test (-1)
- **AMC 2023** — total ?; splits: test (-1)
- **LiveCodeBench** — total ?; splits: test (-1)
- **SynLogic coding validation split** — total ?; splits: val (-1)

## Metrics

- `avg@8` **(primary)** — range: percent
  - Computes the fraction of correctly solved problems when generating up to 8 independent responses per prompt and accepting if at least one is correct. Reported as a percentage.

## Input / output format

**Input**: Standard prompt template for mathematical, coding, or logical reasoning problems.

**Output**: Model generates up to 8 independent responses per prompt; correctness is determined by rule-based or symbolic verification.

## Scoring recipe

```python
def compute_avg_at_8(predictions, gold):
    correct_count = 0
    for i in range(len(predictions)):
        # predictions[i] contains up to 8 generated responses
        if any(verify_response(resp, gold[i]) for resp in predictions[i]):
            correct_count += 1
    return (correct_count / len(predictions)) * 100
```

## Common pitfalls

- The paper uses avg@8 scoring rather than avg@1 or majority voting, which can significantly inflate reported accuracy compared to single-sample baselines.
- Evaluations are performed on held-out standard benchmarks rather than the synthetic training data itself, so performance on the training distribution is not reported.

## Evidence (verbatim from paper)

> Figure[10]: Comparative accuracy (%) of models trained with mixed data and math-only data across three mathematical benchmarks: MATH 500, AIME 2024, and AMC 2023. All evaluations of the figure use avg@8 scoring.

## Citation

```bibtex
@misc{liu2025synlogic,
  title={SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.19641}
}
```

- arXiv: 2505.19641