Mctaco Eval

---
name: mctaco-eval
description: Evaluates a model's ability to reason about temporal commonsense, including event duration, ordering, typical time, frequency, and stationarity. It tests whether systems can correctly classify candidate answers as 'likely' or 'unlikely' given a context sentence and a question. Use when the user wants to benchmark on MCTACO, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.03065
  bibtex_key: zhou2019going
  confidence: high
---

# mctaco-eval

> "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding — Ben Zhou et al. (2019) (arXiv:1909.03065, 2019)

## What this evaluates

Evaluates a model's ability to reason about temporal commonsense, including event duration, ordering, typical time, frequency, and stationarity. It tests whether systems can correctly classify candidate answers as 'likely' or 'unlikely' given a context sentence and a question.

## Datasets

- **MCTACO** — total ?; splits: dev (-1), test (-1)

## Metrics

- `exact match (EM)` — range: percent
  - Fraction of questions where the model's set of predicted 'likely' answers exactly matches the gold set of 'likely' answers. Formula: EM = (1/|D|) * sum_{q in D} prod_{a in q} f(a;q).
- `F1` **(primary)** — range: percent
  - Macro-averaged F1 score across all questions. Question-level F1 is the harmonic mean of precision and recall for predicting 'likely' answers. Formula: F1 = (1/|D|) * sum_{q in D} F1(q).

## Input / output format

**Input**: Concatenation of a context sentence and a temporal commonsense question, paired with a single candidate answer.

**Output**: Binary classification label: 'likely' or 'unlikely' for each candidate answer.

## Scoring recipe

```python
em_scores = []
f1_scores = []
for q in dataset:
    gold_likely = {a for a in q.candidates if a.is_gold}
    pred_likely = {a for a in q.candidates if a.prediction == 'likely'}
    em_scores.append(1.0 if gold_likely == pred_likely else 0.0)
    if len(pred_likely) == 0:
        f1_scores.append(0.0)
    else:
        prec = len(gold_likely & pred_likely) / len(pred_likely)
        rec = len(gold_likely & pred_likely) / len(gold_likely)
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}
```

## Common pitfalls

- EM requires perfect prediction of all 'likely' answers for a question; a single missed or extra 'likely' prediction yields 0 for that question.
- The task is binary classification per candidate ('likely'/'unlikely'), not selecting a single correct answer from a multiple-choice list.
- Human baseline is not 100% due to subjective commonsense; gold labels are crowdsourced, so strict exact-match evaluation can be overly punitive for reasonable alternative temporal interpretations.

## Evidence (verbatim from paper)

> Two question-level metrics are adopted in this work: exact match (EM) and F1 . ... EM measures how many questions a system is able to correctly label all candidate answers, while F1 is more relaxed and measures the average overlap between one's predictions and the ground truth.

## Citation

```bibtex
@misc{zhou2019going,
  title={"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding},
  author={Ben Zhou et al. (2019)},
  year={2019},
  note={arXiv:1909.03065}
}
```

- arXiv: 1909.03065