Csymr Eval

---
name: csymr-eval
description: Evaluates compositional symbolic music reasoning by requiring models to chain atomic analyses across multiple musical dimensions (e.g., rhythm, harmony, key, structure) to answer multiple-choice questions derived from expert forums and professional exams. Use when the user wants to benchmark on CSyMR-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.11556
  bibtex_key: wang2025csymr
  confidence: high
---

# csymr-eval

> CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration — Wang et al. (2025) (arXiv:2601.11556, 2025)

## What this evaluates

Evaluates compositional symbolic music reasoning by requiring models to chain atomic analyses across multiple musical dimensions (e.g., rhythm, harmony, key, structure) to answer multiple-choice questions derived from expert forums and professional exams.

## Datasets

- **CSyMR-Bench** — total 126; splits: test (126)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total number of questions.

## Input / output format

**Input**: Symbolic score excerpts, a natural-language question, and four answer options.

**Output**: Answer letter selection (A, B, C, or D) for direct prompting; ReAct-style tool invocation steps and final answer for the tool-augmented agent.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(predictions)) * 100
```

## Common pitfalls

- Models may hallucinate symbolic music details (e.g., non-existent pitches or chords) when reasoning without tool augmentation, leading to incorrect harmonic analysis.
- Tool-augmented methods may underperform on metadata- or style-driven categories if the model's reasoning focus shifts away from leveraging the analysis tools effectively.

## Evidence (verbatim from paper)

> Table[1] summarizes per‐category results across prompting and reasoning methods. Few‐shot prompting and explicit reasoning improve over direct prompting, but ReAct (without musical tools) is only comparable to CoT. Our method achieves the best overall accuracy, consistently outperforming all baselines.

## Citation

```bibtex
@misc{wang2025csymr,
  title={CSyMR: Benchmarking Compositional Symbolic Muisc Reasoning With MIR Tool Integration},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2601.11556}
}
```

- arXiv: 2601.11556