Es Memeval Eval

---
name: es-memeval-eval
description: This benchmark evaluates conversational agents' long-term memory capabilities in personalized emotional support contexts. It probes five core competencies—information extraction, temporal reasoning, conflict detection, abstention, and user modeling—across question answering, summarization, and dialogue generation tasks. Use when the user wants to benchmark on ES-MemEval, or asks about evaluating this task. Reports F1-Score, LLM-as-Judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.01885
  bibtex_key: chen2026esmemeval
  confidence: high
---

# es-memeval-eval

> ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support — Chen et al. (2026) (arXiv:2602.01885, 2026)

## What this evaluates

This benchmark evaluates conversational agents' long-term memory capabilities in personalized emotional support contexts. It probes five core competencies—information extraction, temporal reasoning, conflict detection, abstention, and user modeling—across question answering, summarization, and dialogue generation tasks.

## Datasets

- **ES-MemEval** — total 1368; splits: QA (1209), Summarization (125), Dialogue Generation (34); repo https://github.com/slptongji/ES-MemEval

## Metrics

- `F1-Score` **(primary)** — range: [0, 1]
  - Token-level F1 overlap between the model's generated answer and the reference answer.
- `BERTScore` — range: [0, 1]
  - Semantic similarity computed using contextual embeddings between the model response and reference answer.
- `LLM-as-Judge` **(primary)** — range: [0, 5]
  - GPT-4o scores semantic consistency and faithfulness on a scale of 0-2 for QA, 0-5 for summarization, and 5-point for dialogue generation based on provided prompts.
- `ROUGE-1/2/L` — range: [0, 1]
  - Lexical overlap metrics measuring unigram, bigram, and longest common subsequence similarity between generated and reference summaries.
- `Event-based F1` — range: [0, 1]
  - F1 score computed on discrete events extracted from both reference and generated summaries to assess factual coverage alignment.
- `LLM Rating` — range: [0, 5]
  - GPT-4o rates overall dialogue quality on a 5-point scale across long-term memory, personalization, and emotional support dimensions.

## Input / output format

**Input**: Varies by task: (1) QA: question text, reference answer, and supporting evidence passage; (2) Summarization: thematic groups of sessions and event timelines; (3) Dialogue Generation: scenario overview, specific details, user physical/psychological state, and relevant prior sessions.

**Output**: Task-specific: (1) QA: natural language answer; (2) Summarization: cross-session summary text; (3) Dialogue Generation: model's conversational response turn.

## Scoring recipe

```python
def score_qa(pred, ref, question):
    f1 = token_f1(pred, ref)
    bert = bertscore(pred, ref)
    llm_judge = gpt4o_score_semantic(question, ref, pred, scale=0-2)
    return f1, bert, llm_judge

def score_sum(pred, ref):
    rouge = rouge_score(pred, ref, ['ROUGE-1','ROUGE-2','ROUGE-L'])
    llm_judge = gpt4o_score_faithfulness(ref, pred, scale=0-5)
    events_ref = extract_events(ref)
    events_pred = extract_events(pred)
    event_f1 = f1_score(events_ref, events_pred)
    return rouge, llm_judge, event_f1

def score_dialogue(pred_history, scenario):
    llm_rating = gpt4o_rate_quality(pred_history, scale=5)
    obs_recall = observation_recall(pred_history, scenario_annotations)
    return llm_rating, obs_recall
```

## Common pitfalls

- LLM-as-Judge scores are highly sensitive to prompt phrasing and temperature, making cross-run comparisons difficult without strict prompt locking.
- Event-based metrics require consistent event extraction; minor paraphrasing in summaries can artificially lower precision/recall scores.
- Models frequently fail to abstain when information is insufficient, generating hallucinated responses that negatively impact QA and dialogue metrics.

## Evidence (verbatim from paper)

> Answer quality is assessed using F1-Score (Rajpurkar et al., [2016]), BERTScore (Zhang et al., [2020]), and LLM-as-Judge (Zheng et al., [2023]). F1-Score measures token-level overlap with the reference answer, while BERTScore computes semantic similarity through contextual embeddings. In addition, LLM-as-Judge enables flexible evaluation of model responses: GPT-4o receives the question, reference answer, and model response, and assigns a score of 0, 1, or 2 reflecting semantic consistency.

## Citation

```bibtex
@misc{chen2026esmemeval,
  title={ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.01885}
}
```

- arXiv: 2602.01885