Mmar Eval

---
name: mmar-eval
description: Evaluates deep audio reasoning capabilities by testing both final answer correctness and the logical quality of intermediate reasoning steps. It covers single-domain (sound, music, speech) and mixed-domain audio tasks to measure how well models avoid spurious correlations and follow verifiable reasoning paths. Use when the user wants to benchmark on MMAR, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12527
  bibtex_key: li2026audiocogito
  confidence: high
---

# mmar-eval

> Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models — Longhao Li et al. (2026) (arXiv:2604.12527, 2026)

## What this evaluates

Evaluates deep audio reasoning capabilities by testing both final answer correctness and the logical quality of intermediate reasoning steps. It covers single-domain (sound, music, speech) and mixed-domain audio tasks to measure how well models avoid spurious correlations and follow verifiable reasoning paths.

## Datasets

- **MMAR** — total ?; splits: test (-1)

## Metrics

- `Avg` **(primary)** — range: percent
  - Average accuracy over the dataset: Avg = (1/N) * sum(c_i), where c_i is 1 if the final answer is correct and 0 otherwise.
- `Rubrics` — range: [0, 1]
  - Average reasoning score across all samples: Rubrics = (1/N) * sum(r_i), where r_i is the proportion of satisfied rubric criteria for correct answers, and 0 for incorrect answers.
- `CRS` — range: [0, 1]
  - Correct Reasoning Score: CRS = sum(r_i) / sum(c_i), representing the average reasoning score conditioned only on correctly answered samples.

## Input / output format

**Input**: Audio sample paired with a natural language question or prompt.

**Output**: A final answer string and a step-by-step reasoning trace (Chain-of-Thought).

## Scoring recipe

```python
def compute_metrics(predictions, golds, rubrics, judge):
    N = len(predictions)
    correct = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    avg_acc = sum(correct) / N
    
    reasoning_scores = []
    for i in range(N):
        if correct[i] == 0:
            reasoning_scores.append(0.0)
        else:
            satisfied = sum(1 for rubric in rubrics[i] if judge(predictions[i], rubric))
            reasoning_scores.append(satisfied / len(rubrics[i]))
            
    rubrics_score = sum(reasoning_scores) / N
    correct_count = sum(correct)
    crs = sum(reasoning_scores) / correct_count if correct_count > 0 else 0.0
    return avg_acc, rubrics_score, crs
```

## Common pitfalls

- Reasoning scores are explicitly set to 0 for incorrect answers, meaning high Rubrics/CRS requires high accuracy first.
- Evaluation relies on an LLM judge (GPT-4o) against auto-generated rubrics, which can introduce judge bias or miss nuanced reasoning flaws.
- Results are not from a single run; the protocol requires five runs and reports the mean of the middle three scores to reduce variance.

## Evidence (verbatim from paper)

> Specifically, for each sample $i$, let $c_{i}\in{0,1}$ denote the correctness of the answer, where $c_{i}\=1$ indicates a correct prediction and $c_{i}\=0$ otherwise. The answer's correctness is measured by the average accuracy (Avg) over the dataset: ... Each MMAR sample is associated with an instance-level rubric, automatically generated by Gemini-2.5-Pro from the ground-truth reasoning path. The rubric contains five verifiable criteria that capture the key reasoning steps for that specific example. Given a model's predicted reasoning trace, an LLM judge evaluates whether each criterion is satisfied. Following the official challenge protocol, we use GPT-4o as the LLM judge. For a correctly answered sample, the judge assigns a binary score (0 or 1) to each criterion, and the reasoning score $r_{i}$ is computed as the proportion of satisfied criteria: ... The overall Rubrics Score across the dataset is defined as: ... We further introduce Correct Reasoning Score (CRS) to evaluate reasoning quality on the correct answer only as follows: ...

## Citation

```bibtex
@misc{li2026audiocogito,
  title={Audio-Cogito: Towards Deep Audio Reasoning in Large Audio Language Models},
  author={Longhao Li et al. (2026)},
  year={2026},
  note={arXiv:2604.12527}
}
```

- arXiv: 2604.12527