Fine R1 Eval

---
name: fine-r1-eval
description: Evaluates multi-modal large language models on fine-grained visual recognition (FGVR) tasks across six standard datasets. It probes the model's ability to distinguish visually similar sub-categories in both closed-world (seen categories) and open-world (unseen categories) settings using chain-of-thought reasoning. Use when the user wants to benchmark on CaltechUCSD Bird-200, Stanford Car-196, Stanford Dog-120, Flower-102, Oxford-IIIT Pet-37, FGVC-Aircraft, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.07605
  bibtex_key: he2026finer1
  confidence: high
---

# fine-r1-eval

> Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning — He et al. (2026) (arXiv:2602.07605, 2026)

## What this evaluates

Evaluates multi-modal large language models on fine-grained visual recognition (FGVR) tasks across six standard datasets. It probes the model's ability to distinguish visually similar sub-categories in both closed-world (seen categories) and open-world (unseen categories) settings using chain-of-thought reasoning.

## Datasets

- **CaltechUCSD Bird-200** — total ?; splits: train (-1), test (-1)
- **Stanford Car-196** — total ?; splits: train (-1), test (-1)
- **Stanford Dog-120** — total ?; splits: train (-1), test (-1)
- **Flower-102** — total ?; splits: train (-1), test (-1)
- **Oxford-IIIT Pet-37** — total ?; splits: train (-1), test (-1)
- **FGVC-Aircraft** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Success rate where the ground-truth sub-category is included in the model's generation. Calculated as the fraction of test examples where the prediction matches or contains the gold label.
- `text inclusion` — range: percent
  - Strict string matching between the predicted text and the ground-truth sub-category name.
- `relative semantic similarity` — range: percent
  - SS_relative = max(0, (Sim(c, c*) - Sim(c_hat, c*)) / (1 - Sim(c_hat, c*))), where c is the predicted sub-category, c* is the ground truth, and c_hat is the super-category. Sim denotes cosine similarity from the SigLIP text encoder.

## Input / output format

**Input**: An image of an object and a text prompt asking the model to identify its sub-category from a candidate list (closed-world) or generate a prediction (open-world).

**Output**: Natural language text containing the predicted sub-category name, optionally preceded by chain-of-thought reasoning.

## Scoring recipe

```python
def score_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if g in p)
    return correct / len(golds)

def score_ss_relative(predictions, golds, super_categories):
    sim = lambda a, b: cosine_similarity(text_encoder(a), text_encoder(b))
    scores = []
    for p, g, s in zip(predictions, golds, super_categories):
        num = max(0, sim(p, g) - sim(s, g))
        den = 1 - sim(s, g)
        scores.append(num / den if den > 0 else 0)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Open-world predictions may differ in granularity (e.g., 'Boeing 737' vs 'Boeing 737-200'), requiring semantic similarity rather than exact match.
- Ground truth labels can contain redundant distinguishing phrases (e.g., 'Coupe 2012' in 'Audi A5 Coupe 2012'), complicating strict string matching.
- The relative semantic similarity formula uses the super-category as a baseline to normalize predictions, which is non-standard for simple accuracy metrics.

## Evidence (verbatim from paper)

> We define success on a single example as whether the ground-truth choice is included in the MLLM generation. We report the success rate of all test examples as the accuracy in the closed-world setting. Since evaluating models in the open-world setting presents additional challenges, as predictions may differ in granularity (e.g., Boeing 737 vs. Boeing 737-200), or ground truth may include redundancy for distinguishing from others (e.g.,“Coupe 2012” in Audi A5 Coupe 2012 and Audi S5 Coupe 2012), we use two complementary metrics: (1) text inclusion (Zhang et al., [2024e]), evaluating strict string matching. (2) relative semantic similarity between the text embeddings of predictions and ground truth calculated by the SigLIP (Zhai et al., [2023]) text encoder.

## Citation

```bibtex
@misc{he2026finer1,
  title={Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2602.07605}
}
```

- arXiv: 2602.07605