Ms Tod Eval

---
name: ms-tod-eval
description: Evaluates an LLM agent's ability to retrieve and utilize long-term memory across multiple dialogue sessions to complete goal-oriented tasks. It probes intent-aligned memory selection, slot-level tracking, and dialogue efficiency in maintaining task continuity over extended interactions. Use when the user wants to benchmark on MS-TOD, SGD, MultiWOZ 2.2, or asks about evaluating this task. Reports Success Rate (S.R.).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.20231
  bibtex_key: du2025memguide
  confidence: high
---

# ms-tod-eval

> MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents — Du et al. (2025) (arXiv:2505.20231, 2025)

## What this evaluates

Evaluates an LLM agent's ability to retrieve and utilize long-term memory across multiple dialogue sessions to complete goal-oriented tasks. It probes intent-aligned memory selection, slot-level tracking, and dialogue efficiency in maintaining task continuity over extended interactions.

## Datasets

- **MS-TOD** — total ?; splits: test (-1)
- **SGD** — total ?; splits: test (-1)
- **MultiWOZ 2.2** — total ?; splits: test (-1)

## Metrics

- `Success Rate (S.R.)` **(primary)** — range: [0, 1]
  - Binary indicator: 1 if the user's final goal is explicitly confirmed as achieved in the dialogue, 0 otherwise.
- `Joint Goal Accuracy (JGA)` — range: [0, 1]
  - Exact-match accuracy of predicted slot values against gold-standard slot-value sets across all dialogue turns.
- `Dialogue Turn Efficiency (DTE)` — range: turns
  - Average number of dialogue turns required to complete a task.
- `GPT-4 score` — range: [1, 10]
  - 1–10 rating assigned by GPT-4-as-a-judge evaluating response fluency, coherence, and informativeness.

## Input / output format

**Input**: Dialogue history, current user utterance, and retrieved memory units (intent-aligned QA pairs or raw history).

**Output**: Model-generated dialogue response (text).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions: list of model responses per turn
    # gold: dict with 'goal', 'slots', 'confirmed'
    sr = 1.0 if gold['confirmed'] else 0.0
    jga = 0.0
    for turn in predictions:
        predicted_slots = extract_slots(turn)
        if predicted_slots == gold['slots']:
            jga += 1.0
    jga /= len(predictions)
    dte = len(predictions)
    return {'S.R.': sr, 'JGA': jga, 'DTE': dte}
```

## Common pitfalls

- Evaluating on non-confirmation-type sessions inflates dialogue length and deflates success rate; only sessions with explicit goal confirmation should be included.
- JGA requires exact slot-value matching; partial matches or synonym variations are typically counted as incorrect unless explicitly normalized.
- GPT-4 scoring is sensitive to prompt formatting and temperature; results may vary if the judge prompt differs from the appendix version.

## Evidence (verbatim from paper)

> We use four core automatic metrics and human evaluation to evaluate response performance: 1) GPT-4 score, (1–10) 333GPT-4-as-the-judge prompts can be found in Appendix evaluates response quality in terms of fluency, coherence, and informativeness; 2) Joint Goal Accuracy (JGA) measures slot prediction accuracy; 3) Dialogue Turn Efficiency (DTE) captures the number of turns required to complete a task, and 4) Success Rate (S.R.) indicates whether the user goal is achieved.

## Citation

```bibtex
@misc{du2025memguide,
  title={MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2505.20231}
}
```

- arXiv: 2505.20231