Reasoning Collab Memory Eval

---
name: reasoning-collab-memory-eval
description: Evaluates LLM logical and spatial reasoning capabilities under multi-agent collaboration and memory-augmented prompting. It probes how different reasoning styles, exemplar retrieval methods, and answer aggregation strategies impact accuracy on formal logic and object-tracking tasks. Use when the user wants to benchmark on FOLIO, RACO, TSO, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.05944
  bibtex_key: [REDACTED]
  confidence: high
---

# reasoning-collab-memory-eval

> Enhancing Reasoning with Collaboration and Memory — Michelman et al. (2025) (arXiv:2503.05944, 2025)

## What this evaluates

Evaluates LLM logical and spatial reasoning capabilities under multi-agent collaboration and memory-augmented prompting. It probes how different reasoning styles, exemplar retrieval methods, and answer aggregation strategies impact accuracy on formal logic and object-tracking tasks.

## Datasets

- **FOLIO** — total ?; splits: test (-1)
- **RACO** — total ?; splits: test (-1)
- **TSO** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy computed after converting model outputs to a canonical form. For multi-agent setups, the final answer is determined by majority voting or a summarizer agent, then compared to the ground truth label.

## Input / output format

**Input**: Natural language or formal logical prompts describing scenes, object states, or logical premises, optionally augmented with few-shot exemplars retrieved from a memory bank.

**Output**: Canonicalized answer string (e.g., 'True', 'False', 'Unknown', or specific object/color/position/count). Multi-agent pipelines aggregate outputs via voting or a summarizer LLM call.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    canonical_preds = [convert_to_canonical(p) for p in predictions]
    correct = sum(1 for p, g in zip(canonical_preds, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Outputs must be strictly converted to a canonical form before comparison; minor formatting differences cause false negatives.
- Multi-agent voting requires consistent answer formatting across all agents to avoid split votes.
- Memory retrieval method (similarity vs. random) significantly impacts performance, contrary to typical assumptions about semantic search.

## Evidence (verbatim from paper)

> LLM answers are converted to a canonical form before comparing to the label or counting votes for multi-agent collaboration. FOLIO (Han et al., [2022]) is a first-order logic task. It presents a series of formal logical statements then asks whether the conclusion is True, False, or Unknown. Reasoning About Colored Objects (RACO) from BIG-bench (Srivastava et al., [2022]) describes a scene involving several objects of various colors, optionally makes a modification, then asks about colors, positions, or counts.

## Citation

```bibtex
@misc{michelman2025enhancing,
  title={Enhancing Reasoning with Collaboration and Memory},
  author={Michelman et al. (2025)},
  year={2025},
  note={arXiv:2503.05944}
}
```

- arXiv: 2503.05944