Sifthinker Eval

---
name: sifthinker-eval
description: Evaluates a model's spatial reasoning, fine-grained visual perception, and 3D spatial grounding capabilities. It also measures self-correction ability through dynamic bounding box refinement and general visual-language understanding across multiple benchmarks. Use when the user wants to benchmark on SpatialBench, SAT-Static, CV-Bench, VisCoT_s, V*Bench, RefCOCO, RefCOCO+, RefCOCOg, OVDEval, MME, MMBench, SEED-Bench, VQAv2, POPE, or asks about evaluating this task. Reports Top-1 Accuracy@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.06259
  bibtex_key: [REDACTED]
  confidence: high
---

# sifthinker-eval

> SIFThinker: Spatially-Aware Image Focus for Visual Reasoning — Chen et al. (2025) (arXiv:2508.06259, 2025)

## What this evaluates

Evaluates a model's spatial reasoning, fine-grained visual perception, and 3D spatial grounding capabilities. It also measures self-correction ability through dynamic bounding box refinement and general visual-language understanding across multiple benchmarks.

## Datasets

- **SpatialBench** — total ?; splits: test (-1)
- **SAT-Static** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)
- **VisCoT_s** — total ?; splits: test (-1)
- **V*Bench** — total ?; splits: test (-1)
- **RefCOCO** — total ?; splits: val (-1), test-A (-1), test-B (-1)
- **RefCOCO+** — total ?; splits: val (-1), test-A (-1), test-B (-1)
- **RefCOCOg** — total ?; splits: val-u (-1), test-u (-1)
- **OVDEval** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1), dev (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **VQAv2** — total ?; splits: test-dev (-1)
- **POPE** — total ?; splits: val (-1)

## Metrics

- `Top-1 Accuracy@0.5` **(primary)** — range: percent
  - Percentage of correctly predicted bounding boxes where the Intersection over Union (IoU) with the ground truth exceeds 0.5.
- `NMS-AP` — range: percent
  - Mean Average Precision computed after Non-Maximum Suppression, evaluating multi-object detection robustness.
- `Average F1 score` — range: percent
  - Macro-averaged F1 score computed across three predefined categories (e.g., Adversarial, Common, Hateful) on the COCO validation set.
- `Accuracy / Score` — range: percent
  - Standard exact-match or normalized score depending on the benchmark (e.g., VQAv2, MME, MMBench, SEED-I, VisCoT_s, V*Bench, SpatialBench).

## Input / output format

**Input**: An input image paired with a textual question or prompt requiring spatial reasoning, visual perception, or referring expression comprehension.

**Output**: A textual response containing the final answer. During the reasoning process, the model may generate interleaved text and bounding box coordinates to represent dynamic attention shifts.

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name == 'Top-1 Accuracy@0.5':
        correct = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
        return correct / len(gold)
    elif metric_name == 'NMS-AP':
        return compute_nms_ap(predictions, gold, iou_thresh=0.5)
    elif metric_name == 'Average F1 score':
        f1s = [f1_score(gold_cat, pred_cat) for cat in ['adversarial', 'common', 'hateful']]
        return sum(f1s) / len(f1s)
    else:
        return exact_match_or_normalized_score(predictions, gold)
```

## Common pitfalls

- Evaluating on different base models (LLaVA-1.5-7B, Bunny-Llama3-8B, Qwen2.5-VL-7B) without explicitly controlling for architectural differences when comparing SOTA methods.
- Using VisCoT_s (scene-related non-planar subsets) instead of the full VisCoT benchmark, which may skew fine-grained perception results.
- Confusing RefCOCO val/test-A/test-B splits with RefCOCO+ and RefCOCOg splits, as they use different partitioning strategies.

## Evidence (verbatim from paper)

> Specifically, we select two structurally similar tasks—Referring Expression Comprehension (REC) and Open-Vocabulary Detection (OVD)—both of which require the model to generate bounding boxes conditioned on textual descriptions. SIFThinker outperforms all previous generalist models with comparable parameters, achieving an average improvement of 1% to 3% over VisRL (93.08 vs. 91.78 on RefCOCO*(Kazemzadeh et al. [2014])*, 85.76 vs. 83.90 on RefCOCO+*(Mao et al. [2016])*, 90.47 vs. 88.82 on RefCOCOg*(Mao et al. [2016])*). Moreover, in most of cases, SIFThinker even surpasses previous state-of-the-art specialist models (e.g. Grounding-DINO, UNINEXT). To further assess multi-object grounding performance of our method, we adopt OVDEval*(Yao et al. [2023])* with NMS-AP as the evaluation metric.

## Citation

```bibtex
@misc{chen2025sifthinker,
  title={SIFThinker: Spatially-Aware Image Focus for Visual Reasoning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.06259}
}
```

- arXiv: 2508.06259