Llava Cot Eval

---
name: llava-cot-eval
description: Evaluates the effectiveness of structured chain-of-thought prompting and test-time scaling algorithms on multimodal reasoning tasks. It probes whether enforcing a specific reasoning order (summary, caption, reasoning, conclusion) and selecting among multiple generated candidates improves answer accuracy over baseline prompting or dense supervision. Use when the user wants to benchmark on Unspecified multimodal reasoning benchmarks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10440
  bibtex_key: xu2024llavacot
  confidence: medium
---

# llava-cot-eval

> LLaVA-CoT: Let Vision Language Models Reason Step-by-Step — Xu et al. (2024) (arXiv:2411.10440, 2024)

## What this evaluates

Evaluates the effectiveness of structured chain-of-thought prompting and test-time scaling algorithms on multimodal reasoning tasks. It probes whether enforcing a specific reasoning order (summary, caption, reasoning, conclusion) and selecting among multiple generated candidates improves answer accuracy over baseline prompting or dense supervision.

## Datasets

- **Unspecified multimodal reasoning benchmarks** — total ?; splits: (unstated)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match or numerical comparison between the model's final answer in the <CONCLUSION> tag and the ground truth label. Performance is reported as the proportion of correctly answered instances.

## Input / output format

**Input**: An image paired with a natural language question, formatted with structured CoT prompting tags (<SUMMARY>, <CAPTION>, <REASONING>, <CONCLUSION>) to guide the model's step-by-step generation.

**Output**: A structured response containing four distinct sections: <SUMMARY> (problem restatement), <CAPTION> (visual description), <REASONING> (step-by-step calculation/logic), and <CONCLUSION> (final answer).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_val in zip(predictions, gold):
        # Extract final answer from <CONCLUSION> tag
        final_answer = extract_tag(pred, 'CONCLUSION')
        if normalize(final_answer) == normalize(gold_val):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Assuming dense GPT-generated supervision alone drives performance; ablation shows the structured CoT format itself is the key driver.
- Assuming any stage ordering works; training with shuffled stage orders yields almost no improvement, proving natural reasoning order is crucial.
- Confusing test-time scaling (inference-time candidate selection) with training-time scaling; the paper explicitly evaluates scaling parameters like N for Best-of-N and retracing iterations for SWIRES during inference.

## Evidence (verbatim from paper)

> The performance is significantly worse, suggesting that denser supervision from GPT-4o is not the reason for improvement. From the figure, it can be observed that test-time scaling effectively corrects the errors made by the model during generation.

## Citation

```bibtex
@misc{xu2024llavacot,
  title={LLaVA-CoT: Let Vision Language Models Reason Step-by-Step},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2411.10440}
}
```

- arXiv: 2411.10440