Streammecoeval

---
name: streammecoeval
description: Evaluates the ability of long-term agent memory compression methods to retain and retrieve critical information from streaming and long-form videos. It probes how effectively a model can maintain temporal consistency and answer complex queries while significantly reducing memory graph size, measured via answer accuracy across robotic, web, and general video benchmarks. Use when the user wants to benchmark on M3-Bench-robot, M3-Bench-web, Video-MME-Long, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09000
  bibtex_key: wang2026streammecol
  confidence: high
---

# streammecoeval

> StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding — Wang et al. (2026) (arXiv:2604.09000, 2026)

## What this evaluates

Evaluates the ability of long-term agent memory compression methods to retain and retrieve critical information from streaming and long-form videos. It probes how effectively a model can maintain temporal consistency and answer complex queries while significantly reducing memory graph size, measured via answer accuracy across robotic, web, and general video benchmarks.

## Datasets

- **M3-Bench-robot** — total ?; splits: test (-1)
- **M3-Bench-web** — total ?; splits: test (-1)
- **Video-MME-Long** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered video-based queries, evaluated by GPT-4o. Reported per category (ME, MH, CM, PU, GK) and averaged across datasets.

## Input / output format

**Input**: Video sequences (streaming or offline) paired with natural language queries/questions.

**Output**: Natural language answers generated by the model, which are then scored by GPT-4o for correctness.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if gpt4o_judge(pred, gold) == 'correct':
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Using GPT-4o as the sole judge for answer quality may introduce LLM-as-a-judge bias or inconsistency across different video domains.
- Comparing compression performance across datasets with different inherent redundancies (e.g., robotic vs. YouTube) without normalizing for baseline accuracy differences.
- Assuming streaming benchmarks behave identically to offline video benchmarks, ignoring temporal continuity and retrieval latency constraints.

## Evidence (verbatim from paper)

> Consistent with the original M3-Agent setup, we use GPT-4o to assess the answer quality and employ text-embedding-3-large to encode the query content generated by M3-Agent to supporting subsequent memory graph retrieval. ... Even with 70% of text nodes compressed, StreamMeCo and TMR mechanism still achieves an average accuracy improvement of 1.0% across all datasets compared to the uncompressed M3-Agent.

## Citation

```bibtex
@misc{wang2026streammecol,
  title={StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.09000}
}
```

- arXiv: 2604.09000