Polarmem Multimodal Eval

---
name: polarmem-multimodal-eval
description: Evaluates training-free multimodal agents on retrieval-augmented generation, general reasoning, and hallucination robustness by testing a polarized latent graph memory that injects logical constraints at inference time. Use when the user wants to benchmark on MRAMG-Bench, MRAG-Bench, Visual-RAG, MMMU, MMStar, HallusionBench, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.00415
  bibtex_key: chen2026polarmem
  confidence: high
---

# polarmem-multimodal-eval

> PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Multimodal Agents — Chen et al. (2026) (arXiv:2602.00415, 2026)

## What this evaluates

Evaluates training-free multimodal agents on retrieval-augmented generation, general reasoning, and hallucination robustness by testing a polarized latent graph memory that injects logical constraints at inference time.

## Datasets

- **MRAMG-Bench** — total ?; splits: test (-1)
- **MRAG-Bench** — total ?; splits: test (-1)
- **Visual-RAG** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **HallusionBench** — total ?; splits: test (-1)

## Metrics

- `performance` **(primary)** — range: percent
  - Standard benchmark performance score calculated as the percentage of correctly answered questions or appropriately graded responses per each benchmark's official evaluation protocol.

## Input / output format

**Input**: Multimodal query (image + text prompt) from the benchmark, augmented with retrieved evidence snippets formatted as '[Fact Check: {Status}] {Content}', truncated to a fixed token limit per item to fit the context window.

**Output**: Text response/answer generated by the VLM backbone to the query.

## Scoring recipe

```python
def compute_performance(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if evaluate_answer(p, g) == 'correct')
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Residual retrieval noise in long contexts can cause diminishing or negative returns on stronger models.
- Retrieved items are truncated to a fixed maximum length (in tokens) to fit within the model context window, which may discard relevant evidence.
- All memory-based methods must use the same memory sources and the same retrieval budget (Top-K evidence) to ensure fair comparison.

## Evidence (verbatim from paper)

> Table 1: Performance comparison on six benchmarks. PolarMem (PolarMem) significantly boosts retrieval-augmented benchmarks (MRAMG/MRAG/Visual-RAG) for smaller and mid-size backbones, while showing diminishing or occasionally negative returns on stronger models for general reasoning (MMMU/MMStar) and hallucination robustness (HallusionBench), likely due to residual retrieval noise in long contexts.

## Citation

```bibtex
@misc{chen2026polarmem,
  title={PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Multimodal Agents},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.00415}
}
```

- arXiv: 2602.00415