Label Accuracy Eval

---
name: label-accuracy-eval
description: Evaluates GPT-3's ability to predict ground-truth labels for given instances, and analyzes whether explanation quality correlates with prediction correctness across different datasets. Use when the user wants to benchmark on CommonsenseQA, SNLI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.08674
  bibtex_key: wiegreffe2021reframing
  confidence: medium
---

# label-accuracy-eval

> Reframing Human-AI Collaboration for Generating Free-Text Explanations — Wiegreffe et al. (2021) (arXiv:2112.08674, 2021)

## What this evaluates

Evaluates GPT-3's ability to predict ground-truth labels for given instances, and analyzes whether explanation quality correlates with prediction correctness across different datasets.

## Datasets

- **CommonsenseQA** — total ?; splits: test (-1)
- **SNLI** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's predicted label matches the ground-truth label.

## Input / output format

**Input**: Question/context instance formatted as a prompt, explicitly excluding the 'why?' token and any gold explanations.

**Output**: A single predicted label corresponding to the instance.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Low accuracy on SNLI (46%) reflects known difficulties with sentence-comparison tasks, not necessarily poor explanation generation.
- High label accuracy does not guarantee that generated explanations are faithful or 'right for the right reasons'.

## Evidence (verbatim from paper)

> GPT-3 achieves 50.8% accuracy on CommonsenseQA compared to a 20% random baseline, and 46% accuracy on SNLI compared to a 33.33% random baseline.

## Citation

```bibtex
@misc{wiegreffe2021reframing,
  title={Reframing Human-AI Collaboration for Generating Free-Text Explanations},
  author={Wiegreffe et al. (2021)},
  year={2021},
  note={arXiv:2112.08674}
}
```

- arXiv: 2112.08674