Mirage Eval

---
name: mirage-eval
description: Evaluates multimodal vision-language models on expert-level agricultural reasoning, including grounded entity identification, causal explanation quality, and dialogue management decisions (clarify vs. respond) under partial observability. Use when the user wants to benchmark on MIRAGE-MMST, MIRAGE-MMMT, or asks about evaluating this task. Reports Identification Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.20100
  bibtex_key: dongre2025mirage
  confidence: high
---

# mirage-eval

> MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations — Dongre et al. (2025) (arXiv:2506.20100, 2025)

## What this evaluates

Evaluates multimodal vision-language models on expert-level agricultural reasoning, including grounded entity identification, causal explanation quality, and dialogue management decisions (clarify vs. respond) under partial observability.

## Datasets

- **MIRAGE-MMST** — total ?; splits: train (-1), test (-1); repo https://github.com/MIRAGE-Benchmark/MIRAGE-Benchmark
- **MIRAGE-MMMT** — total ?; splits: test (-1); repo https://github.com/MIRAGE-Benchmark/MIRAGE-Benchmark

## Metrics

- `Identification Accuracy` **(primary)** — range: [0, 1]
  - Binary metric: 1 if the predicted entity string exactly matches any reference field (entity name, scientific name, or common names); 0 otherwise.
- `Reasoning Score` — range: [0, 4]
  - LLM-judge graded 0–4 scale based on the presence of key visual clues, descriptive specificity, and causal coherence in the model's justification.
- `W-Sum` — range: [0, 1]
  - Composite score: (2*Accuracy + Relevance + Completeness + Parsimony) / 20, aggregating four 0–4 dimension scores.
- `Decision Accuracy` — range: percent
  - Percentage of turns where the model's chosen action (clarify vs. respond) matches the gold annotation.

## Input / output format

**Input**: Multimodal inputs comprising agricultural images, user queries, and dialogue history (for multi-turn tasks). Optional metadata (location, time) may be provided.

**Output**: Model must output a predicted biological entity, a textual justification/explanation, and a dialogue action decision (clarify or respond).

## Scoring recipe

```python
def score_mmst_id(pred, gold):
    return 1.0 if pred in [gold['name'], gold['sci_name'], gold['common']] else 0.0

def score_mmst_mg(pred, gold, judge_ensemble):
    acc = judge_ensemble.evaluate(pred, gold, rubric='factual_alignment')
    rel = judge_ensemble.evaluate(pred, gold, rubric='on_topic')
    comp = judge_ensemble.evaluate(pred, gold, rubric='covers_key_info')
    pars = judge_ensemble.evaluate(pred, gold, rubric='concise_actionable')
    return (2*acc + rel + comp + pars) / 20.0

def score_mmmt(pred_action, gold_action):
    return 1.0 if pred_action == gold_action else 0.0
```

## Common pitfalls

- Relies on an LLM-as-a-Judge ensemble (3 models × 3 generations = 9 evaluations per sample) to ensure reliability; single-pass judge scoring will yield high variance.
- Models frequently treat provided metadata (location/time) as distracting rather than helpful, causing slight performance degradation when metadata is included.
- Significant open-world generalization gap: accuracy drops ~14 points on unseen biological entities compared to seen ones, indicating poor long-tail generalization.

## Evidence (verbatim from paper)

> We evaluate model performance on the MMST-ID task using two complementary metrics: 1.) Identification Accuracy which is a binary metric that measures whether the entity identified by the model matches the expert’s answer. A response is scored as correct (1) if the predicted entity string exactly matches any of the reference fields: entity name, scientific name, or common names; otherwise, it receives a score of 0. Reasoning Score, evaluates the quality of the model’s visual and linguistic justification for its prediction. It is graded on a 0–4 scale by the judges, based on the presence of key visual clues, descriptive specificity, and causal coherence.

## Citation

```bibtex
@misc{dongre2025mirage,
  title={MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations},
  author={Dongre et al. (2025)},
  year={2025},
  note={arXiv:2506.20100}
}
```

- arXiv: 2506.20100