Egomem Eval

---
name: egomem-eval
description: Evaluates a lifelong memory agent's ability to perform real-time audiovisual user retrieval, detect dialog session boundaries in continuous streams, and generate personalized, fact-consistent responses in full-duplex omnimodal interactions. Use when the user wants to benchmark on LFW, VoxCeleb, EgoMem Custom Text Retrieval, EgoMem Episodic Trigger, or asks about evaluating this task. Reports pass@5, Fact Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.11914
  bibtex_key: yao2025egomem
  confidence: high
---

# egomem-eval

> EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models — Yao et al. (2025) (arXiv:2509.11914, 2025)

## What this evaluates

Evaluates a lifelong memory agent's ability to perform real-time audiovisual user retrieval, detect dialog session boundaries in continuous streams, and generate personalized, fact-consistent responses in full-duplex omnimodal interactions.

## Datasets

- **LFW** — total ?; splits: test (-1); repo https://github.com/serengil/deepface/tree/master/benchmarks
- **VoxCeleb** — total ?; splits: test (-1)
- **EgoMem Custom Text Retrieval** — total 200; splits: test (200)
- **EgoMem Episodic Trigger** — total 1000; splits: test (1000)

## Metrics

- `Accuracy` — range: [0, 1]
  - Fraction of correctly verified face pairs out of total test pairs.
- `pass@1` — range: [0, 1]
  - Binary indicator: 1 if the top-1 retrieved speaker matches the query speaker, else 0.
- `EER` — range: [0, 1]
  - Equal Error Rate: the decision threshold where false acceptance rate equals false rejection rate.
- `pass@5` **(primary)** — range: [0, 1]
  - Fraction of queries where all relevant facts are returned within the top-5 retrieved entries.
- `Jaccard score` — range: [0, 1]
  - Intersection over union of predicted and ground-truth dialog session spans.
- `span_match@N` — range: [0, 1]
  - Precision, recall, and F1 for detected boundaries allowing ±N step tolerance from ground truth.
- `Fact Score` **(primary)** — range: [0, 1]
  - Binary 0/1 score from an LLM judge indicating if the response is personalized, consistent with the user profile, and factually correct.
- `Answer Quality` — range: [0, 10]
  - LLM-assigned score from 0 to 10 measuring general helpfulness and quality independent of personalization.

## Input / output format

**Input**: Retrieval: query (face image, speech utterance, or text keyword) vs candidate pool. Episodic Trigger: continuous token stream segments. Personalized Dialog: user instruction, ground-truth response, MemChunks contents, and generated response (for evaluator).

**Output**: Retrieval: ranked candidate list or binary match decision. Episodic Trigger: detected dialog boundary timestamps. Personalized Dialog: generated textual/audio response.

## Scoring recipe

```python
def compute_pass_at5(retrieved_top5, gold_facts):
    return 1.0 if all(f in retrieved_top5 for f in gold_facts) else 0.0

def compute_span_match_f1(predicted_spans, gold_spans, tolerance=5):
    tp = sum(1 for p in predicted_spans if any(abs(p - g) <= tolerance for g in gold_spans))
    precision = tp / len(predicted_spans) if predicted_spans else 0
    recall = tp / len(gold_spans) if gold_spans else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

def compute_fact_score(prediction, user_profile, evaluator_model):
    prompt = f"Check if '{prediction}' matches profile: {user_profile}. Return 1 if consistent, 0 otherwise."
    return 1 if evaluator_model(prompt) == "1" else 0
```

## Common pitfalls

- Speaker verification EER threshold is manually tuned to 6 for deployment based on human case studies rather than using a fixed benchmark threshold.
- Episodic trigger span_match@0 degrades significantly in noisy environments due to inherent VAD latency, which is expected but often misinterpreted as a model failure.
- Fact Score relies on an external LLM judge (DeepSeek-V3), introducing non-determinism and prompt sensitivity that can affect reproducibility.

## Evidence (verbatim from paper)

> For each dialog turn, we provide an evaluator model with the following inputs: the user instruction (textual transcript), the ground-truth textual response, the contents of the MemChunks, and the textual monologue response generated by RoboEgo. The evaluator is implemented with the DeepSeek-V3 API, prompted to return two scores: – Fact Score: A binary 0/1 metric for each turn indicating whether the model’s response is personalized to the user and consistent with the user profile, without factual errors. – Answer Quality: A score from 0 to 10 for each turn measuring the general helpfulness and quality of the response with respect to the user instruction, independent of personalization.

## Citation

```bibtex
@misc{yao2025egomem,
  title={EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.11914}
}
```

- arXiv: 2509.11914