Vg Cot Eval

---
name: vg-cot-eval
description: Evaluates the visual reasoning and grounding capabilities of Large Vision-Language Models (LVLMs) by measuring the quality of their step-by-step rationales, the accuracy of their final answers, and the alignment between the generated reasoning and the prediction. Use when the user wants to benchmark on VG-CoT, or asks about evaluating this task. Reports Rationale Quality (RQ), Answer Accuracy (AA), Reasoning-Answer Alignment (RAA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.21396
  bibtex_key: lim2026vgcot
  confidence: high
---

# vg-cot-eval

> VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought — Lim et al. (2026) (arXiv:2604.21396, 2026)

## What this evaluates

Evaluates the visual reasoning and grounding capabilities of Large Vision-Language Models (LVLMs) by measuring the quality of their step-by-step rationales, the accuracy of their final answers, and the alignment between the generated reasoning and the prediction.

## Datasets

- **VG-CoT** — total ?; splits: test (-1)

## Metrics

- `Rationale Quality (RQ)` **(primary)** — range: percent
  - Composite score assessing visual evidence utilization, logical coherence, and reasoning completeness of the generated chain-of-thought.
- `Answer Accuracy (AA)` **(primary)** — range: percent
  - Overall correctness of the model's final answer, calculated as the percentage of exact matches against ground truth.
- `Reasoning-Answer Alignment (RAA)` **(primary)** — range: percent
  - Measures the consistency and faithfulness between the generated rationale and the final prediction, typically via LLM-judgment or logical entailment.
- `mAP@0.5 / mAP@0.75` — range: percent
  - Mean Average Precision for bounding box localization of visual evidence against pseudo-label ground truths at IoU thresholds of 0.5 and 0.75.

## Input / output format

**Input**: An image and a natural language question.

**Output**: A step-by-step reasoning process explicitly grounded in visual evidence, followed by a final answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    aa = sum(1 for p, g in zip(predictions['answers'], golds['answers']) if p == g) / len(predictions['answers'])
    raa = consistency_score(predictions['rationales'], predictions['answers'])
    rq = composite_score(predictions['rationales'], golds['evidence'])
    mAP = mean_average_precision(predictions['boxes'], golds['boxes'], iou_thresh=0.5)
    return {'AA': aa, 'RAA': raa, 'RQ': rq, 'mAP@0.5': mAP}
```

## Common pitfalls

- Visual evidence grounding scores are consistently lower than text-based reasoning scores, indicating spatial grounding is inherently harder than logical generation.
- Alignment anomalies can occur where fine-tuning improves overall accuracy but slightly decreases consistency between rationale and answer.
- Scene-text tasks pose a significant bottleneck for precise spatial grounding compared to relational or general VQA tasks.

## Evidence (verbatim from paper)

> The models are evaluated across three core dimensions. First, Rationale Quality (RQ) assesses visual evidence utilization, logical coherence, and reasoning completeness. Second, Answer Accuracy (AA) represents the overall correctness of the model’s output. Finally, Reasoning-Answer Alignment (RAA) measures the consistency and faithfulness between the generated rationale and the final prediction.

## Citation

```bibtex
@misc{lim2026vgcot,
  title={VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought},
  author={Lim et al. (2026)},
  year={2026},
  note={arXiv:2604.21396}
}
```

- arXiv: 2604.21396