Visual Cot Eval

---
name: visual-cot-eval
description: Evaluates multi-modal large language models' ability to perform chain-of-thought reasoning with dynamic visual focusing on specific image regions. It probes localized visual understanding, intermediate bounding box prediction, and multi-turn reasoning across document, chart, general VQA, relation reasoning, and fine-grained domains. Use when the user wants to benchmark on Visual CoT Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.16999
  bibtex_key: hao2024visualcot
  confidence: high
---

# visual-cot-eval

> Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning — Hao Shao et al. (2024) (arXiv:2403.16999, 2024)

## What this evaluates

Evaluates multi-modal large language models' ability to perform chain-of-thought reasoning with dynamic visual focusing on specific image regions. It probes localized visual understanding, intermediate bounding box prediction, and multi-turn reasoning across document, chart, general VQA, relation reasoning, and fine-grained domains.

## Datasets

- **Visual CoT Benchmark** — total ?; splits: test (-1); repo https://github.com/deepcs233/Visual-CoT

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted answers, reported as a decimal between 0 and 1. For qualitative/CoT evaluation, a ChatGPT-based score between 0 and 1 is also used to rate prediction accuracy.

## Input / output format

**Input**: An image and a natural language question.

**Output**: A textual answer, optionally accompanied by intermediate chain-of-thought reasoning steps and predicted bounding boxes for key visual regions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return correct / len(gold_answers)
```

## Common pitfalls

- The benchmark aggregates 12 heterogeneous datasets; performance varies significantly across domains (e.g., Doc/Text vs. Fine-grained), so reporting only the average can mask domain-specific weaknesses.
- Zero-shot evaluation on SROIE, DUDE, and Visual7W uses only their test splits, which may not reflect full generalization or dataset-specific biases.
- ChatGPT-based scoring for CoT quality is subjective and highly prompt-dependent, making cross-study comparisons difficult without identical prompts.

## Evidence (verbatim from paper)

> We utilize 12 source datasets, as shown in Fig. 1, and when an official training/evaluation split exists, we adopt it. In cases where such a split does not exist, we randomly divide the dataset. Additionally, we incorporate the test split of SROIE, DUDE, and Visual7W to evaluate the model's zero-shot visual CoT capabilities. Following the methodology of previous MLLM studies [33, 46], we employ ChatGPT [52] and ask it to assign a numerical score between 0 and 1, where a higher score indicates better prediction accuracy.

## Citation

```bibtex
@misc{hao2024visualcot,
  title={Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning},
  author={Hao Shao et al. (2024)},
  year={2024},
  note={arXiv:2403.16999}
}
```

- arXiv: 2403.16999