Mem Gallery Eval

---
name: mem-gallery-eval
description: This benchmark evaluates multimodal long-term conversational memory in MLLM agents across multi-session dialogues. It probes the agent's ability to extract, adapt, reason over, and manage evolving visual and textual information, including handling temporal dependencies, conflicting updates, and knowledge gaps. Use when the user wants to benchmark on Mem-Gallery, or asks about evaluating this task. Reports answer correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.03515
  bibtex_key: bei2026memgallery
  confidence: medium
---

# mem-gallery-eval

> Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents — Bei et al. (2026) (arXiv:2601.03515, 2026)

## What this evaluates

This benchmark evaluates multimodal long-term conversational memory in MLLM agents across multi-session dialogues. It probes the agent's ability to extract, adapt, reason over, and manage evolving visual and textual information, including handling temporal dependencies, conflicting updates, and knowledge gaps.

## Datasets

- **Mem-Gallery** — total 1711; splits: test (1711); repo https://github.com/YuanchenBei/Mem-Gallery

## Metrics

- `answer correctness` **(primary)** — range: [0, 1]
  - Percentage of QA pairs where the model's generated answer exactly matches the ground-truth answer. Evaluated per subtask (e.g., Factual Retrieval, Knowledge Resolution) and aggregated across the benchmark.

## Input / output format

**Input**: A sequence of multi-session multimodal conversations (text and images) with temporal gaps, followed by a question requiring the model to retrieve or reason over accumulated memory.

**Output**: A textual answer to the question.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / total
```

## Common pitfalls

- Models often fail to persist visual information across sessions without explicit captioning or retrieval mechanisms.
- Evaluating 'answer refusal' requires checking if the model correctly abstains when information is missing or contradictory, rather than just penalizing wrong answers.
- Long-horizon conversations require incremental memory updates; models relying solely on context windows will fail.

## Evidence (verbatim from paper)

> LLMs are first used to check answer correctness and question clarity. This verification step is followed by careful human revision.

## Citation

```bibtex
@misc{bei2026memgallery,
  title={Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents},
  author={Bei et al. (2026)},
  year={2026},
  note={arXiv:2601.03515}
}
```

- arXiv: 2601.03515