Visual Text Grounding Eval

---
name: visual-text-grounding-eval
description: Evaluates multimodal large language models' ability to perform precise spatial reasoning and visual text grounding in document images. It tests whether models can generate accurate bounding boxes that support their textual answers, both from scratch (OCR-free) and when provided with OCR text (OCR-based), while also measuring their instruction-following capability. Use when the user wants to benchmark on ChartQA, DocVQA, InfographicsVQA, TRINS, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.04974
  bibtex_key: li2025trig
  confidence: high
---

# visual-text-grounding-eval

> Towards Visual Text Grounding of Multimodal Large Language Model — Ming Li et al. (2025) (arXiv:2504.04974, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform precise spatial reasoning and visual text grounding in document images. It tests whether models can generate accurate bounding boxes that support their textual answers, both from scratch (OCR-free) and when provided with OCR text (OCR-based), while also measuring their instruction-following capability.

## Datasets

- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **InfographicsVQA** — total ?; splits: test (-1)
- **TRINS** — total ?; splits: test (-1)

## Metrics

- `IoU` **(primary)** — range: percent
  - Pixel-level Intersection over Union between the predicted bounding box and the ground-truth bounding box, averaged across all matched boxes and samples.
- `Precision` — range: percent
  - Proportion of predicted bounding boxes that correctly match a ground-truth box (typically above an IoU threshold).
- `Recall` — range: percent
  - Proportion of ground-truth bounding boxes that are successfully matched by at least one predicted box.
- `F1` — range: percent
  - Harmonic mean of Precision and Recall at the bounding-box level.
- `Instruction-following rate` — range: percent
  - Proportion of test samples for which the model generates at least one bounding box, regardless of its correctness.

## Input / output format

**Input**: Document image and a question. In Evaluation Setting 2, the input also includes OCR-extracted text and their bounding boxes.

**Output**: Text answer and one or more bounding boxes (in pixel coordinates) that support the answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    ious, follows = [], 0
    for pred, gt in zip(predictions, golds):
        if len(pred) > 0: follows += 1
        for p_box, g_box in zip(pred, gt):
            inter = intersection_area(p_box, g_box)
            union = union_area(p_box, g_box)
            ious.append(inter / union if union > 0 else 0)
    iou_avg = sum(ious) / len(ious) if ious else 0
    # P, R, F1 computed via greedy matching of predicted to GT boxes
    instr_rate = follows / len(predictions)
    return {'IoU': iou_avg, 'Precision': p, 'Recall': r, 'F1': f1, 'Instruction-following rate': instr_rate}
```

## Common pitfalls

- Most open-source MLLMs fail to follow the instruction to output bounding boxes, resulting in zero outputs rather than incorrect ones.
- The OCR-based setting (Setting 2) simplifies the task to bounding box selection rather than true spatial grounding, inflating scores for models that can follow simple selection instructions.
- IoU scores are extremely low across all models, making it difficult to differentiate fine-grained spatial reasoning capabilities.

## Evidence (verbatim from paper)

> IoU, P, R, F1 represent bounding-box-level IoU score, precision, recall and F1 score. Avg represents the average score on all datasets and evaluation metrics, and the ordering is decided by this score.

## Citation

```bibtex
@misc{li2025trig,
  title={Towards Visual Text Grounding of Multimodal Large Language Model},
  author={Ming Li et al. (2025)},
  year={2025},
  note={arXiv:2504.04974}
}
```

- arXiv: 2504.04974