Tir Bench Eval

---
name: tir-bench-eval
description: Evaluates multimodal language models' ability to perform agentic reasoning with images, specifically requiring dynamic visual manipulation and tool-use to solve complex tasks like rotation, jigsaw assembly, and instrument reading. It probes whether models can iteratively process, crop, or transform visual inputs to extract information or solve spatial problems. Use when the user wants to benchmark on TIR-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.01833
  bibtex_key: li2025tirbench
  confidence: high
---

# tir-bench-eval

> TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning — Ming Li et al. (2025) (arXiv:2511.01833, 2025)

## What this evaluates

Evaluates multimodal language models' ability to perform agentic reasoning with images, specifically requiring dynamic visual manipulation and tool-use to solve complex tasks like rotation, jigsaw assembly, and instrument reading. It probes whether models can iteratively process, crop, or transform visual inputs to extract information or solve spatial problems.

## Datasets

- **TIR-Bench** — total ?; splits: test (-1); repo https://github.com/agents-x-project/TIR-Bench

## Metrics

- `accuracy` **(primary)** — range: percent
  - For multiple-choice and short-form answers, the final answer is extracted via GPT-4o and compared directly against the ground-truth value. For grounding-type problems (e.g., Jigsaw Game, Spot the Difference), intersection over union (IoU) is calculated. Results are reported as percentage accuracy across tasks.

## Input / output format

**Input**: A single image and a text prompt/question. Models are evaluated in a zero-shot setting without task-specific fine-tuning or in-context examples.

**Output**: Text response containing reasoning steps and/or a final answer. For grounding tasks, models output coordinates or list-type answers. Final answers are extracted by GPT-4o before scoring.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, task_types):
    correct = 0
    for pred, gold, task in zip(predictions, golds, task_types):
        if task in ['grounding', 'jigsaw', 'spot_difference']:
            iou = calculate_iou(pred, gold)
            correct += (1.0 if iou > 0.5 else 0.0)
        else:
            extracted = gpt4o_extract_answer(pred)
            correct += (1.0 if extracted == gold else 0.0)
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Answer extraction relies on GPT-4o, which may introduce parsing inconsistencies or model-specific biases compared to deterministic string matching.
- IoU calculation for list-type grounding answers requires explicit handling of coordinate formats and list ordering, which is not detailed in the paper.
- Agentic models require external tool-use environments (e.g., code interpreters, sandboxes) to function, making zero-shot API evaluation setup-dependent and potentially unfair to non-agentic baselines.

## Evidence (verbatim from paper)

> We follow previous works*(lu2023mathvista; li2025eee)* to first generate answers from models and subsequently using GPT-4o to extract the final answer from the answer content. For multiple-choice and short-form answers, we compare the extracted value directly against the ground-truth to calculate accuracy; for grounding type problems such as Jigsaw Game and Spot the Difference with list type answer, we calculate the intersection over union (IoU).

## Citation

```bibtex
@misc{li2025tirbench,
  title={TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning},
  author={Ming Li et al. (2025)},
  year={2025},
  note={arXiv:2511.01833}
}
```

- arXiv: 2511.01833