Thyme Multimodal Eval

---
name: thyme-multimodal-eval
description: Evaluates multimodal large language models on image manipulation, visual perception, mathematical reasoning, and general vision-language tasks. It probes whether autonomous code generation and execution for image processing improves downstream accuracy and reduces hallucination. Use when the user wants to benchmark on MME-RealWorld, HR Bench, MathVista, Hallucination bench, MMStar, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.11630
  bibtex_key: zhang2025thyme
  confidence: high
---

# thyme-multimodal-eval

> Thyme: Think Beyond Images — Zhang et al. (2025) (arXiv:2508.11630, 2025)

## What this evaluates

Evaluates multimodal large language models on image manipulation, visual perception, mathematical reasoning, and general vision-language tasks. It probes whether autonomous code generation and execution for image processing improves downstream accuracy and reduces hallucination.

## Datasets

- **MME-RealWorld** — total ?; splits: Perception (-1), Reasoning (-1), Overall (-1)
- **HR Bench** — total ?; splits: FSP (-1), FCP (-1), Overall (-1)
- **MathVista** — total ?; splits: Mini (-1)
- **Hallucination bench** — total ?; splits: aAcc (-1), fAcc (-1), qAcc (-1), Overall (-1)
- **MMStar** — total ?; splits: Overall (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances. Computed as (number of correct predictions / total instances) * 100. For multi-part benchmarks, separate accuracy metrics are reported (e.g., FSP/FCP for HR Bench, aAcc/fAcc/qAcc for Hallucination bench, Perception/Reasoning for MME-RealWorld).

## Input / output format

**Input**: Multimodal input consisting of one or more images paired with a text prompt/question requiring visual understanding, code generation, or reasoning.

**Output**: Textual response, optionally containing executable Python code blocks and simulated sandbox outputs. For benchmarks, the final answer or generated code is extracted for scoring.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100
# Note: For benchmarks like Hallucination, separate metrics (aAcc, fAcc, qAcc) are computed on their respective subsets.
# Code-based tasks may require executing the generated code and comparing sandbox output or final answer.
```

## Common pitfalls

- Failing to report perception and reasoning splits separately for MME-RealWorld and HR Bench.
- Confusing the three Hallucination bench metrics (aAcc, fAcc, qAcc) or averaging them incorrectly.
- Assuming code generation is always required; the model may output direct text, which must be parsed correctly for scoring.
- Overlooking that some benchmarks (e.g., Chart QA, BLINK) have specific validation/test splits that must be matched.

## Evidence (verbatim from paper)

> We mainly select three categories of benchmarks. The first category focuses on perception tasks because Thyme’s image operations mainly aim to enhance perception ability. These benchmarks include the MME-RealWorld series, HR Bench, V*, RealWorld QA, etc. We report results for different splits of each benchmark. For example, for the MME-RealWorld series, we report perception and reasoning accuracy separately. For HR Bench, we report Fine-grained Single-instance Perception (FSP) and Fine-grained Cross-instance Perception (FCP) separately.

## Citation

```bibtex
@misc{zhang2025thyme,
  title={Thyme: Think Beyond Images},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.11630}
}
```

- arXiv: 2508.11630