Dori Eval

---
name: dori-eval
description: Evaluates multimodal large language models' ability to understand and reason about object orientation across four dimensions: frontal alignment, rotational transformations, relative directional relationships, and canonical orientation. It distinguishes between coarse categorical judgments and fine-grained angular estimations to probe 3D spatial reasoning capabilities. Use when the user wants to benchmark on DORI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21649
  bibtex_key: nichols2025dori
  confidence: high
---

# dori-eval

> Right Side Up? Disentangling Orientation Understanding in MLLMs with Fine-grained Multi-axis Perception Tasks — Nichols et al. (2025) (arXiv:2505.21649, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to understand and reason about object orientation across four dimensions: frontal alignment, rotational transformations, relative directional relationships, and canonical orientation. It distinguishes between coarse categorical judgments and fine-grained angular estimations to probe 3D spatial reasoning capabilities.

## Datasets

- **DORI** — total 33656; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct answers chosen from multiple options. Calculated as (number of correct predictions / total number of questions) × 100.

## Input / output format

**Input**: Image input paired with a standardized multiple-choice question prompt describing an orientation task (e.g., frontal alignment, rotation, relative direction, or canonical orientation).

**Output**: Model must output a selected answer option from the provided multiple-choice set, along with a reasoning explanation.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Models often perform worse than random chance on certain sub-tasks, indicating systematic failure rather than mere noise.
- Performance heavily depends on whether the question requires coarse categorical judgment or fine-grained angular estimation, with a large drop in the latter.
- Results may be inflated for datasets like COCO due to potential train/test distribution overlap from pretraining data.

## Evidence (verbatim from paper)

> We evaluate models on accuracy—the percentage of correct answers chosen from multiple options—across both coarse and granular question types. All models were prompted with the standardized task descriptions and prompted to provide both an answer selection and a reasoning explanation.

## Citation

```bibtex
@misc{nichols2025dori,
  title={Right Side Up? Disentangling Orientation Understanding in MLLMs with Fine-grained Multi-axis Perception Tasks},
  author={Nichols et al. (2025)},
  year={2025},
  note={arXiv:2505.21649}
}
```

- arXiv: 2505.21649