M3cot Eval

---
name: m3cot-eval
description: Evaluates vision-language models' ability to perform multi-step, multi-modal chain-of-thought reasoning across diverse domains like science, commonsense, and mathematics. It probes the model's capacity to integrate visual information with textual reasoning steps and produce accurate final answers under various prompting and fine-tuning setups. Use when the user wants to benchmark on M3CoT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.16473
  bibtex_key: chen2024m3cot
  confidence: high
---

# m3cot-eval

> M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought — Chen et al. (2024) (arXiv:2405.16473, 2024)

## What this evaluates

Evaluates vision-language models' ability to perform multi-step, multi-modal chain-of-thought reasoning across diverse domains like science, commonsense, and mathematics. It probes the model's capacity to integrate visual information with textual reasoning steps and produce accurate final answers under various prompting and fine-tuning setups.

## Datasets

- **M3CoT** — total ?; splits: train (-1), test (-1); repo https://github.com/LightChen233/M3CoT

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of test instances where the model's final extracted answer exactly matches the ground truth label. Computed per domain and overall.

## Input / output format

**Input**: An image paired with a question/prompt. Depending on the evaluation setup, the prompt may include few-shot examples, chain-of-thought instructions (e.g., 'Let's think step-by-step!'), or tool-use directives.

**Output**: A chain-of-thought reasoning trace followed by a final answer. The final answer is extracted using regular expressions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        extracted = extract_final_answer(pred)  # via regex as specified
        if extracted == gold:
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Regex-based answer extraction may fail if models output answers in unexpected formats or include extra conversational text.
- Multi-step reasoning performance drops significantly with more steps, so evaluating only final accuracy without step-wise analysis can mask intermediate reasoning failures.
- Tool-usage and ICL setups can degrade performance if the model lacks interleaved image-text training or proper multi-modal planning capabilities.

## Evidence (verbatim from paper)

> In order to further understand the difference in model reasoning with different numbers of steps, we calculated the accuracy of different steps. As illustrated in Figure[7] (b), an increase in the number of reasoning steps is associated with a significant decline in the model’s performance. Following the settings of Kojima et al. (2022); Qin et al. (2023), we extract the final generated answer through regular expressions.

## Citation

```bibtex
@misc{chen2024m3cot,
  title={M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2405.16473}
}
```

- arXiv: 2405.16473