Mmerealworld Eval

---
name: mmerealworld-eval
description: This benchmark evaluates multimodal large language models on high-resolution real-world image perception and complex reasoning tasks. It probes the models' ability to extract fine-grained details from large images and perform logical inference across diverse domains like autonomous driving, remote sensing, and document understanding. Use when the user wants to benchmark on MME-RealWorld, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.13257
  bibtex_key: zhang2024mmerealworld
  confidence: high
---

# mmerealworld-eval

> MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans? — Yi-Fan Zhang et al. (2024) (arXiv:2408.13257, 2024)

## What this evaluates

This benchmark evaluates multimodal large language models on high-resolution real-world image perception and complex reasoning tasks. It probes the models' ability to extract fine-grained details from large images and perform logical inference across diverse domains like autonomous driving, remote sensing, and document understanding.

## Datasets

- **MME-RealWorld** — total 29429; splits: test (29429)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100.
- `weighted average accuracy` — range: percent
  - Average accuracy across subtasks weighted by the number of QA pairs in each subtask.
- `unweighted average accuracy` — range: percent
  - Average accuracy across subtasks calculated without weighting by subtask size.

## Input / output format

**Input**: High-resolution images (average 2000×1500 pixels) paired with natural language questions covering perception and reasoning tasks.

**Output**: Text responses answering the provided questions. Models may also output specific tokens like "E" to indicate object absence.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Closed-source models often have strict image resolution/size limits, requiring compression that degrades performance on high-res inputs.
- Models may adopt conservative strategies (e.g., outputting "E" for missing objects) or refuse to answer due to safety filters, artificially lowering accuracy.
- Accuracy is computed per subtask and then averaged; failing to account for subtask size differences can misrepresent overall capability.

## Evidence (verbatim from paper)

> “Avg” and “Avg-C” indicate the weighted average accuracy and the unweighted average accuracy across subtasks in each domain.

## Citation

```bibtex
@misc{zhang2024mmerealworld,
  title={MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?},
  author={Yi-Fan Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.13257}
}
```

- arXiv: 2408.13257