Memcollab Eval

---
name: memcollab-eval
description: Evaluates LLM agents' ability to solve mathematical reasoning and code generation tasks by leveraging a shared, contrastively distilled memory system. It probes cross-agent knowledge transfer, reasoning invariance extraction, and task-aware memory retrieval efficiency. Use when the user wants to benchmark on MATH500, GSM8K, MBPP, HumanEval, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.23234
  bibtex_key: chang2026memcollab
  confidence: high
---

# memcollab-eval

> MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation — Chang et al. (2026) (arXiv:2603.23234, 2026)

## What this evaluates

Evaluates LLM agents' ability to solve mathematical reasoning and code generation tasks by leveraging a shared, contrastively distilled memory system. It probes cross-agent knowledge transfer, reasoning invariance extraction, and task-aware memory retrieval efficiency.

## Datasets

- **MATH500** — total 1500; splits: memory (1000), test (500)
- **GSM8K** — total 1500; splits: memory (1000), test (500)
- **MBPP** — total ?; splits: test (-1)
- **HumanEval** — total ?; splits: test (-1)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly solved instances out of the total evaluated instances.
- `Average Accuracy (%)` — range: percent
  - Mean of Accuracy (%) across MATH500, GSM8K, MBPP, and HumanEval.
- `Average reasoning turns` — range: other
  - Mean number of reasoning turns required to solve an instance.

## Input / output format

**Input**: Task prompts for mathematical reasoning or code generation problems.

**Output**: Model-generated reasoning trajectories and final answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_correct(p, g))
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Data leakage: The 1000 instances used to construct the memory system must be strictly disjoint from the 500 instances used for evaluation.
- Naive memory transfer: Directly transferring memory from a single larger model can degrade performance compared to vanilla baselines; contrastive construction is required.
- Retrieval budget: Performance degrades if more than 3 memory entries are retrieved due to noise and attention dispersion.

## Evidence (verbatim from paper)

> From each dataset, we randomly sample 1000 instances to construct the memory system and evaluate performance on a disjoint set of 500 randomly selected instances, reporting accuracy as the metric*(kang2025distilling)*. For code generation, we evaluate on MBPP*(austin2021program)* and HumanEval*(chen2021evaluating)*.

## Citation

```bibtex
@misc{chang2026memcollab,
  title={MemCollab: Cross-Agent Memory Collaboration via Contrastive Trajectory Distillation},
  author={Chang et al. (2026)},
  year={2026},
  note={arXiv:2603.23234}
}
```

- arXiv: 2603.23234