Locomo Eval

---
name: locomo-eval
description: This benchmark evaluates the long-term conversational memory of LLM agents by testing their ability to answer questions, summarize events, and generate multi-modal dialogues over very long, multi-session conversations. It probes how well models retain and reason over temporal and causal information across hundreds of turns and thousands of tokens. Use when the user wants to benchmark on LoCoMo, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.17753
  bibtex_key: maharana2024locomo
  confidence: high
---

# locomo-eval

> Evaluating Very Long-Term Conversational Memory of LLM Agents — Maharana et al. (2024) (arXiv:2402.17753, 2024)

## What this evaluates

This benchmark evaluates the long-term conversational memory of LLM agents by testing their ability to answer questions, summarize events, and generate multi-modal dialogues over very long, multi-session conversations. It probes how well models retain and reason over temporal and causal information across hundreds of turns and thousands of tokens.

## Datasets

- **LoCoMo** — total ?; splits: test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Standard F1 score computed between predicted answer text and ground truth answer text, calculated as the harmonic mean of precision and recall.
- `Recall Accuracy (R@$k$)` — range: [0, 1]
  - Measures whether the correct retrieval unit (dialog, observation, or summary) appears in the top-k retrieved items.

## Input / output format

**Input**: Dialogue history (text-only with image captions for QA/summarization, or raw images for multi-modal generation) interleaved with a question or prompt.

**Output**: Predicted answer text (for QA) or generated dialogue turn (for multi-modal generation).

## Scoring recipe

```python
def compute_f1(pred, gold):
    pred_tokens = set(pred.lower().split())
    gold_tokens = set(gold.lower().split())
    if not pred_tokens or not gold_tokens:
        return 0.0
    precision = len(pred_tokens & gold_tokens) / len(pred_tokens)
    recall = len(pred_tokens & gold_tokens) / len(gold_tokens)
    if precision + recall == 0:
        return 0.0
    return 2 * precision * recall / (precision + recall)
```

## Common pitfalls

- Models are evaluated under different context length constraints (e.g., 4K, 8K, 16K), so results are not directly comparable without accounting for the truncated history.
- RAG experiments use different retrieval units (dialog, observation, summary) and top-k values, making it difficult to isolate the effect of the retriever versus the reader model.

## Evidence (verbatim from paper)

> Results are based on F1-score for answer prediction; higher is better.

## Citation

```bibtex
@misc{maharana2024locomo,
  title={Evaluating Very Long-Term Conversational Memory of LLM Agents},
  author={Maharana et al. (2024)},
  year={2024},
  note={arXiv:2402.17753}
}
```

- arXiv: 2402.17753