Mm Food 100k Eval

---
name: mm-food-100k-eval
description: Evaluates the predictive capability of vision-language models on food-related tasks, specifically testing their ability to estimate nutritional content (kilocalories) and identify categorical food attributes (dish name, ingredients, cooking method) from images. The protocol isolates the value of structured, human-verified data by comparing base foundation models against their supervised fine-tuned counterparts on a frozen test split. Use when the user wants to benchmark on MM-Food-100K, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.10429
  bibtex_key: dong2025mmfood100k
  confidence: high
---

# mm-food-100k-eval

> MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance — Dong et al. (2025) (arXiv:2508.10429, 2025)

## What this evaluates

Evaluates the predictive capability of vision-language models on food-related tasks, specifically testing their ability to estimate nutritional content (kilocalories) and identify categorical food attributes (dish name, ingredients, cooking method) from images. The protocol isolates the value of structured, human-verified data by comparing base foundation models against their supervised fine-tuned counterparts on a frozen test split.

## Datasets

- **MM-Food-100K** — total 100000; splits: train (80000), val (10000), test (10000)

## Metrics

- `Win Rate` — range: percent
  - Percentage of test cases where one model's categorical output (dish name, ingredients, or cooking method) is judged more accurate or comprehensive than a baseline's output using text similarity.
- `MAE` **(primary)** — range: other
  - Mean Absolute Error between predicted and ground-truth kilocalories.
- `RMSE` — range: other
  - Root Mean Square Error between predicted and ground-truth kilocalories.
- `R2` — range: [0, 1]
  - Coefficient of determination for kilocalorie prediction, measuring the proportion of variance in the target explained by the model.

## Input / output format

**Input**: Food image (and associated metadata/context for categorical tasks)

**Output**: Predicted kilocalories (float) for regression; predicted dish name, ingredients, and cooking method (text strings) for categorical tasks.

## Scoring recipe

```python
def score_regression(y_pred, y_true):
    mae = mean(abs(y_pred - y_true))
    rmse = sqrt(mean((y_pred - y_true) ** 2))
    ss_res = sum((y_true - y_pred) ** 2)
    ss_tot = sum((y_true - mean(y_true)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    return mae, rmse, r2

def score_win_rate(preds_a, preds_b, golds):
    wins = 0
    for pa, pb, g in zip(preds_a, preds_b, golds):
        if text_similarity(pa, g) > text_similarity(pb, g):
            wins += 1
    return (wins / len(golds)) * 100
```

## Common pitfalls

- Win rate is a pairwise comparative metric, not an absolute accuracy score, so it cannot be interpreted as standalone model capability.
- The evaluation intentionally holds seeds, batch sizes, and learning rates constant to isolate data value from optimization variables, meaning results are not directly comparable to models trained with different hyperparameters.
- Stratification by cuisine and source type is applied during splitting but not yet reported in the initial results, which may mask performance disparities across food categories.

## Evidence (verbatim from paper)

> We use standard regression metrics: MAE (Mean Absolute Error), RMSE (Root Mean Square Error), and R2. In each comparison between two models (e.g., base vs. fine-tuned), we report the percentage of test cases where one model’s output is judged to be more accurate or comprehensive.

## Citation

```bibtex
@misc{dong2025mmfood100k,
  title={MM-Food-100K: A 100,000-Sample Multimodal Food Intelligence Dataset with Verifiable Provenance},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2508.10429}
}
```

- arXiv: 2508.10429