Simmc2.0 Eval

---
name: simmc2.0-eval
description: Evaluates multimodal task-oriented dialogue capabilities, specifically focusing on dialogue state tracking, disambiguation, coreference resolution, and response generation using visual scene representations. Use when the user wants to benchmark on SIMMC 2.0, or asks about evaluating this task. Reports Intent-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.04907
  bibtex_key: hemanthage2023simplemtod
  confidence: high
---

# simmc2.0-eval

> SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation — Hemanthage et al. (2023) (arXiv:2307.04907, 2023)

## What this evaluates

Evaluates multimodal task-oriented dialogue capabilities, specifically focusing on dialogue state tracking, disambiguation, coreference resolution, and response generation using visual scene representations.

## Datasets

- **SIMMC 2.0** — total ?; splits: dev-test (-1), test-standard (-1)

## Metrics

- `Intent-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for predicting the correct dialogue intent class. Computed as 2 * (precision * recall) / (precision + recall).
- `Slot-F1` — range: [0, 1]
  - Harmonic mean of precision and recall for predicting dialogue state slot values. Matches predicted slot-value pairs against ground truth.
- `Joint Accuracy` — range: [0, 1]
  - Exact match accuracy where the entire predicted belief state (all intents and slots) must exactly match the ground truth state.
- `Request Slot-F1` — range: [0, 1]
  - F1 score specifically for predicting request slots in the dialogue state.

## Input / output format

**Input**: Multimodal dialogue context $C_t$ including de-localized object tokens, spatial information, and bounding box areas; optionally includes ground-truth system belief state for response generation.

**Output**: Task-dependent: YES/NO tokens for disambiguation, de-localized object tokens with spatial info for coreference, natural language system responses for generation, and structured belief states (intent, slots) for DST.

## Scoring recipe

```python
pred_intent = extract_intent(prediction)
gold_intent = extract_intent(gold)
intent_f1 = f1_score(gold_intent, pred_intent)

pred_slots = extract_slots(prediction)
gold_slots = extract_slots(gold)
slot_f1 = f1_score(gold_slots, pred_slots)

joint_acc = mean([1.0 if p == g else 0.0 for p, g in zip(pred_slots, gold_slots)])
```

## Common pitfalls

- The test-standard split is held-out and not publicly available; final evaluation must use the dev-test split.
- End-to-end vs task-specific setups differ: in E2E mode, disambiguation/coref labels are derived from the predicted system intent, not ground truth.
- Coreference resolution tie-breaking relies on bounding box area when multiple objects share the same de-localized token in a region.

## Evidence (verbatim from paper)

> We follow the experimental setup of the SIMMC 2.0 challenge with same dataset-splits, inference time limitations, and performance metrics. See Appendix:B for details. It should be noted that the test-standard split of the SIMMC2.0 dataset is not publicly available and is a held-out set for evaluating submissions to SIMMC2.0 challenge. Therefore, the final version of our model could only be evaluated on the dev-test split. Table columns report Intent-F1, Slot-F1, Request Slot-F1, and Joint Accuracy.

## Citation

```bibtex
@misc{hemanthage2023simplemtod,
  title={SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation},
  author={Hemanthage et al. (2023)},
  year={2023},
  note={arXiv:2307.04907}
}
```

- arXiv: 2307.04907