Mmdr Bench Eval

---
name: mmdr-bench-eval
description: This benchmark evaluates a model's ability to handle complex, multi-turn visually-grounded dialogue and follow intricate instructions. It probes sustained contextual understanding, visual entity tracking across turns, and multi-step reasoning depth in dynamic multi-modal interactions. Use when the user wants to benchmark on MMDR-Bench, or asks about evaluating this task. Reports average human evaluation ratings.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.15164
  bibtex_key: han2025contextualvlmagent
  confidence: high
---

# mmdr-bench-eval

> ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following — Han et al. (2025) (arXiv:2508.15164, 2025)

## What this evaluates

This benchmark evaluates a model's ability to handle complex, multi-turn visually-grounded dialogue and follow intricate instructions. It probes sustained contextual understanding, visual entity tracking across turns, and multi-step reasoning depth in dynamic multi-modal interactions.

## Datasets

- **MMDR-Bench** — total 300; splits: test (300)

## Metrics

- `average human evaluation ratings` **(primary)** — range: [1, 5]
  - Human experts score each model response per turn on a 1-5 scale across six dimensions: visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency. The final metric is the arithmetic mean of all turn-level scores across all scenarios and evaluators.

## Input / output format

**Input**: Multi-turn dialogue scenarios (5-7 turns on average) centered around one or more images, containing complex question-answering or instruction-following prompts per turn.

**Output**: Textual model responses for each dialogue turn.

## Scoring recipe

```python
scores = []
for scenario in dataset:
    for turn in scenario.turns:
        rating = human_rater.evaluate(turn.prompt, turn.image, turn.model_response, dimensions=6)
        scores.append(rating)
return sum(scores) / len(scores)
```

## Common pitfalls

- Human evaluation relies on expert raters, introducing potential subjectivity and inter-rater variability that automated metrics cannot capture.
- LLM-based automatic evaluation is mentioned only as supplementary validation without a defined formula or prompt, making it non-reproducible.
- Latency measurements are hardware-dependent and may vary significantly across different deployment environments.

## Evidence (verbatim from paper)

> Table[I] presents the core experimental results, showcasing the average human evaluation ratings of different models across the six critical dimensions on the MMDR-Bench dataset. Human evaluators, who are experts in multi-modal AI interactions, meticulously review model responses for each dialogue turn and assign a score on a 1-5 scale across the six predefined dimensions: visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency. A score of 5 indicates excellent performance, while 1 indicates poor performance. The final reported scores are the average ratings across all scenarios and evaluators.

## Citation

```bibtex
@misc{han2025contextualvlmagent,
  title={ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following},
  author={Han et al. (2025)},
  year={2025},
  note={arXiv:2508.15164}
}
```

- arXiv: 2508.15164