Tfq Bench Eval

---
name: tfq-bench-eval
description: Evaluates a model's ability to understand visual metaphors and image implications by verifying multiple factual and inferential propositions per image. It probes fine-grained visual perception, multi-hop reasoning, and theory of mind through structured true-false questioning. Use when the user wants to benchmark on TFQ-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10575
  bibtex_key: zhang2026metaphorstar
  confidence: high
---

# tfq-bench-eval

> MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning — Zhang et al. (2026) (arXiv:2602.10575, 2026)

## What this evaluates

Evaluates a model's ability to understand visual metaphors and image implications by verifying multiple factual and inferential propositions per image. It probes fine-grained visual perception, multi-hop reasoning, and theory of mind through structured true-false questioning.

## Datasets

- **TFQ-Bench** — total 14099; splits: Lite (492), Full (14099)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered True/False questions. Calculated as the number of predictions matching the ground truth label divided by the total number of questions.

## Input / output format

**Input**: An image paired with a single true-false proposition (question).

**Output**: Structured text containing a reasoning process enclosed in <think>...</think> tags, followed by a final answer enclosed in <answer>...</answer> tags. The final answer must be 'T' or 'F'.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        start = pred.find('<answer>') + len('<answer>')
        end = pred.find('</answer>')
        pred_ans = pred[start:end].strip().upper()
        if pred_ans == gold.strip().upper():
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Models may output reasoning but fail to strictly follow the <answer> tag format, causing parsing failures if not handled robustly.
- The benchmark includes both basic visual perception questions and complex metaphorical implication questions; reporting aggregate accuracy without splitting by difficulty or question type may mask specific capability gaps.
- TFQ-Bench-Lite is strictly disjoint from TFQ-Data-Full, so evaluating on the full set without checking for data leakage is critical.

## Evidence (verbatim from paper)

> The accuracy reward assesses whether the final answer is correct, while the format reward penalizes poorly structured outputs. Formally, given a reasoning trajectory $\tau$, the total reward is defined as:

## Citation

```bibtex
@misc{zhang2026metaphorstar,
  title={MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.10575}
}
```

- arXiv: 2602.10575