Re Mi Eval

---
name: re-mi-eval
description: Evaluates multimodal LLMs' ability to reason across multiple images, including sequential and set-based consumption, interleaved text-image processing, and heterogeneous visual inputs like charts, equations, maps, and code. It probes cross-image contextual integration, precise visual reading, and step-by-step logical deduction. Use when the user wants to benchmark on ReMI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.09175
  bibtex_key: kazemi2024remi
  confidence: high
---

# re-mi-eval

> ReMI: A Dataset for Reasoning with Multiple Images — Kazemi et al. (2024) (arXiv:2406.09175, 2024)

## What this evaluates

Evaluates multimodal LLMs' ability to reason across multiple images, including sequential and set-based consumption, interleaved text-image processing, and heterogeneous visual inputs like charts, equations, maps, and code. It probes cross-image contextual integration, precise visual reading, and step-by-step logical deduction.

## Datasets

- **ReMI** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact match for textual outputs, ignoring spacing and case. For numeric answers, relaxed accuracy with a task-specific tolerance epsilon: default 1%, 3% for GeomShapes/GeomCost, and 10 minutes for Clocks. A prediction p is correct if (1-epsilon)*l <= p <= (1+epsilon)*l, where l is the label.
- `error reduction percentage (ERP)` — range: percent
  - Measures how much a model reduces error compared to a baseline B: 100 * (Error_T(B) - Error_T(M)) / Error_T(B). Normalizes the error reduction by the baseline's error magnitude.

## Input / output format

**Input**: Multiple images (up to six) interleaved with or presented as a set/sequence of text prompts/questions. Tasks span math, physics, code, spatial/temporal reasoning, and table/chart analysis.

**Output**: Final answer (numeric value, text string, or multiple-choice selection). For CodeEdit, a line of code ending in a semicolon.

## Scoring recipe

```python
def compute_metric(pred, gold, task_name):
    if task_name in ['GeomShapes', 'GeomCost']:
        tol = 0.03
    elif task_name == 'Clocks':
        tol = 10  # minutes
    else:
        tol = 0.01  # 1%
    if isinstance(gold, str):
        return normalize_whitespace_case(pred) == normalize_whitespace_case(gold)
    return (1 - tol) * gold <= pred <= (1 + tol) * gold
```

## Common pitfalls

- Relaxed accuracy tolerances are task-specific (1% default, 3% for geometry tasks, 10 minutes for clocks). Using a uniform tolerance will misreport scores.
- The naive baseline strategy varies significantly across tasks (e.g., uniform random 1/c for MC, fixed coordinates (0,0) for charts, 12*60 for clock diffs, 0 for RefCOCO).
- Models often achieve higher scores when images are provided sequentially/separately rather than concatenated into a single composite image, which can skew comparative evaluations if input format is not standardized.

## Evidence (verbatim from paper)

> Metrics: We mainly report accuracy for our tasks. For textual outputs, we compute exact match while handling slight variations such as spacing issues, lowercase vs uppercase, etc. For numeric answers, we compute a relaxed accuracy with 1% tolerance, mainly to avoid penalizing rounding errors. In the case of relaxed accuracy with tolerance $\epsilon$, a numeric prediction $p$ is considered correct if $(1-\epsilon)l\leq p\leq(1+\epsilon)l$ where $l$ is the label.

## Citation

```bibtex
@misc{kazemi2024remi,
  title={ReMI: A Dataset for Reasoning with Multiple Images},
  author={Kazemi et al. (2024)},
  year={2024},
  note={arXiv:2406.09175}
}
```

- arXiv: 2406.09175