Cosmos Qa Eval

---
name: cosmos-qa-eval
description: This benchmark evaluates a model's ability to perform contextual commonsense reasoning in machine reading comprehension. It probes whether systems can make non-literal, implicit inferences about causes, effects, and counterfactuals based on personal narratives, rather than relying on explicit textual evidence or simple semantic matching. Use when the user wants to benchmark on Cosmos QA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.00277
  bibtex_key: huang2019cosmosqa
  confidence: high
---

# cosmos-qa-eval

> Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning — Huang et al. (2019) (arXiv:1909.00277, 2019)

## What this evaluates

This benchmark evaluates a model's ability to perform contextual commonsense reasoning in machine reading comprehension. It probes whether systems can make non-literal, implicit inferences about causes, effects, and counterfactuals based on personal narratives, rather than relying on explicit textual evidence or simple semantic matching.

## Datasets

- **Cosmos QA** — total 35600; splits: dev (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions / total questions) × 100.

## Input / output format

**Input**: A context paragraph, a question about the paragraph, and a set of multiple-choice candidate answers.

**Output**: A single selected candidate answer from the provided options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models often overfit to lexical/semantic overlap between the paragraph and distractor answers, failing to perform actual commonsense inference.
- The dataset contains >83% questions where the correct answer is not explicitly stated in the text, so retrieval or exact-match strategies will fail.

## Evidence (verbatim from paper)

> Table 3: Comparison of varying approaches (Accuracy %). ... Most of the reading comprehension approaches apply attention to capture the correlation between paragraph, question and each candidate answer and tend to select the answer which is the most semantically closed to the paragraph.

## Citation

```bibtex
@misc{huang2019cosmosqa,
  title={Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1909.00277}
}
```

- arXiv: 1909.00277