Memoryagentbench Eval

---
name: memoryagentbench-eval
description: Evaluates four core memory competencies in LLM agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. It transforms long-context datasets into session-based multi-turn interactions to simulate real-world memory accumulation and retrieval. Use when the user wants to benchmark on MemoryAgentBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.05257
  bibtex_key: [REDACTED]
  confidence: high
---

# memoryagentbench-eval

> Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions — Hu et al. (2025) (arXiv:2507.05257, 2025)

## What this evaluates

Evaluates four core memory competencies in LLM agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. It transforms long-context datasets into session-based multi-turn interactions to simulate real-world memory accumulation and retrieval.

## Datasets

- **MemoryAgentBench** — total ?; splits: test (-1); repo https://github.com/HUST-AI-HYZ/MemoryAgentBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered queries against ground-truth labels. Calculated as (number of correct predictions / total number of instances) * 100.

## Input / output format

**Input**: Multi-turn conversational sessions containing long-context documents or prior interaction history, segmented into chunks (typically 512 or 4096 tokens) for retrieval-augmented or long-context processing.

**Output**: Natural language response or structured answer to a specific query posed within the multi-turn session.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize(pred) == normalize(gold):
        correct += 1
return (correct / len(predictions)) * 100
```

## Common pitfalls

- Chunk size selection significantly biases results: smaller chunks (512) favor RAG agents on retrieval tasks but hurt long-range understanding, while larger chunks (4096) favor long-context models.
- Retrieval top-k is fixed at 10 in main results; increasing it beyond 10 exceeds typical context windows (~40k tokens) and is not evaluated.
- Selective forgetting tasks are extremely difficult for multi-hop scenarios, with most agents achieving ≤7% accuracy, making it a key differentiator from simple retrieval benchmarks.

## Evidence (verbatim from paper)

> The evaluation metrics for all datasets are shown in Table [1], along with more dataset details. ... We observe that all methods fail on the multi-hop situation (with achieving at most 7% accuracy).

## Citation

```bibtex
@misc{hu2025evaluatingmemoryllmagents,
  title={Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2507.05257}
}
```

- arXiv: 2507.05257