Diningbench Eval

---
name: diningbench-eval
description: This benchmark evaluates Vision-Language Models on fine-grained visual discrimination, nutritional quantification from images, and complex food-related visual question answering. It probes the models' ability to fuse multi-view imagery, perform volumetric reasoning, and avoid parametric knowledge biases when identifying dishes and estimating macronutrients. Use when the user wants to benchmark on DiningBench, or asks about evaluating this task. Reports Accuracy, MAPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10425
  bibtex_key: [REDACTED]
  confidence: high
---

# diningbench-eval

> DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain — Jin et al. (2026) (arXiv:2604.10425, 2026)

## What this evaluates

This benchmark evaluates Vision-Language Models on fine-grained visual discrimination, nutritional quantification from images, and complex food-related visual question answering. It probes the models' ability to fuse multi-view imagery, perform volumetric reasoning, and avoid parametric knowledge biases when identifying dishes and estimating macronutrients.

## Datasets

- **DiningBench** — total ?; splits: test (-1); repo https://github.com/meituan/DiningBench

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correct predictions out of total instances. Calculated as the number of exact matches between model output and ground truth divided by the total number of samples.
- `MAPE` **(primary)** — range: percent
  - Mean Absolute Percentage Error. Computed as the average of |(actual - predicted) / actual| across all samples, typically expressed as a percentage to quantify regression error in nutrition estimation.

## Input / output format

**Input**: One to four images of a dish (multi-view) paired with a text prompt. Prompts vary by task: classification options, nutrition estimation queries, or open-ended VQA questions.

**Output**: Text response containing either a selected dish name, numerical nutritional values, or a natural language answer. Models may optionally generate chain-of-thought reasoning before the final prediction.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    if task_type in ['classification', 'vqa']:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
        return correct / len(golds)
    elif task_type == 'nutrition':
        mape_vals = []
        for p, g in zip(predictions, golds):
            pred = extract_numeric(p)
            gold = extract_numeric(g)
            if gold != 0:
                mape_vals.append(abs(pred - gold) / gold)
        return (sum(mape_vals) / len(mape_vals)) * 100
```

## Common pitfalls

- Chain-of-Thought prompting often degrades performance on fine-grained classification and nutrition regression tasks by introducing noise or hallucination instead of refining predictions.
- Smaller models suffer from information overload when provided with 3+ multi-view images, causing performance to plateau or drop rather than improve.
- Models frequently default to parametric knowledge priors (e.g., guessing common dishes) rather than relying on visual evidence, especially for long-tail or visually similar dishes.

## Evidence (verbatim from paper)

> Quantifying nutritional content from visual cues proves to be the most demanding task. Even the state-of-the-art Gemini-3-Pro-Preview yields an Average MAPE of 24.45%, reflecting a non-negligible margin of error. The challenge is more pronounced for GPT-4o, which suffers from a high error rate of 42.43%.

## Citation

```bibtex
@misc{jin2026diningbench,
  title={DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2604.10425}
}
```

- arXiv: 2604.10425