Marvel Eval

---
name: marvel-eval
description: Evaluates multimodal large language models on multidimensional abstract visual reasoning and perceptual grounding. It probes the model's ability to recognize complex geometric and abstract patterns, track temporal/spatial changes, and perform multi-step visual reasoning across diverse puzzle configurations. Use when the user wants to benchmark on MARVEL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.13591
  bibtex_key: jiang2024marvel
  confidence: high
---

# marvel-eval

> MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning — Jiang et al. (2024) (arXiv:2404.13591, 2024)

## What this evaluates

Evaluates multimodal large language models on multidimensional abstract visual reasoning and perceptual grounding. It probes the model's ability to recognize complex geometric and abstract patterns, track temporal/spatial changes, and perform multi-step visual reasoning across diverse puzzle configurations.

## Datasets

- **MARVEL** — total 770; splits: test (770); repo https://github.com/1171-jpg/MARVEL_AVR

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Extracted via regex matching from model responses (e.g., 'choice 4'). Failed extractions are re-extracted using GPT-4. Instance-based accuracy computes the fraction of correctly answered individual questions. Group-based accuracy requires the model to answer all questions within a predefined group correctly to receive a score of 1.

## Input / output format

**Input**: Visual puzzle image(s) containing abstract/geometric shapes and multiple-choice options (typically 4 choices), presented in zero-shot or few-shot settings.

**Output**: Text response containing the selected choice (e.g., 'The correct answer is choice 4').

## Scoring recipe

```python
def extract_choice(response):
    match = re.search(r'choice\s+(\d+)', response)
    if not match:
        match = gpt4_extract(response) # fallback
    return match.group(1) if match else None

def instance_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def group_accuracy(group_predictions, group_golds):
    correct_groups = 0
    for preds, golds in zip(group_predictions, group_golds):
        if all(p == g for p, g in zip(preds, golds)):
            correct_groups += 1
    return correct_groups / len(group_predictions)
```

## Common pitfalls

- Group-based accuracy is an all-or-nothing metric per group; a single error in a group yields a score of 0, which heavily penalizes partial reasoning failures compared to standard per-instance accuracy.
- Models frequently fail at basic perceptual grounding (e.g., counting panels or identifying shapes), causing near-random performance on perception questions even when reasoning capabilities are theoretically present.
- Safety filters in some models (e.g., Gemini-pro-vision) refuse to answer a subset of puzzles (~7%), artificially deflating reported accuracy if not explicitly excluded from the denominator.

## Evidence (verbatim from paper)

> Following a similar setting as previous research evaluating MLLMs on the AVR benchmark (Ahrabian et al., 2024), we use regex matching to extract the choices picked (e.g., "choice  $4''$  in the response "The correct answer is choice  $4''$ ), with failure cases re-extracted by GPT-4 (Aher et al., 2023). We use accuracy as the metric, which is commonly used for evaluating multiple-choice questions and has been utilized by many AVR systems (Zhang et al., 2019; Hill et al., 2019). Based on the hierarchical evaluation framework, we evaluate MLLMs with two types of accuracy-based metrics:

1. Instance-based Accuracy considers questions separately. We report accuracy results for AVR question and fine-grained perception question.  
2. Group-based Accuracy considers questions as group to assess the consistency in model reasoning (Jiang et al., 2023; Yuan et al., 2021). The model receives a score of 1 only if it correctly answers all questions within the same group. We report the group-based

accuracy result of combining all three coarse-grained perception questions and the further result after introducing fine-grained and AVR questions into the group.

## Citation

```bibtex
@misc{jiang2024marvel,
  title={MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2404.13591}
}
```

- arXiv: 2404.13591