Cola Eval

---
name: cola-eval
description: This benchmark evaluates a model's ability to classify English sentences as grammatically acceptable or unacceptable. It probes syntactic competence by measuring performance on both in-domain and out-of-domain linguistic data. Use when the user wants to benchmark on CoLA, or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1805.12471
  bibtex_key: warstadt2018neural
  confidence: high
---

# cola-eval

> Neural Network Acceptability Judgments — Warstadt et al. (2018) (arXiv:1805.12471, 2018)

## What this evaluates

This benchmark evaluates a model's ability to classify English sentences as grammatically acceptable or unacceptable. It probes syntactic competence by measuring performance on both in-domain and out-of-domain linguistic data.

## Datasets

- **CoLA** — total 10657; splits: train (-1), dev (-1), test (-1)

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthews Correlation Coefficient, measuring the quality of binary classifications. Calculated as (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
- `Accuracy` — range: [0, 1]
  - The proportion of correctly classified sentences out of the total number of sentences.

## Input / output format

**Input**: A single English sentence.

**Output**: A binary label: 1 for acceptable, 0 for unacceptable.

## Scoring recipe

```python
def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    denom = ((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) ** 0.5
    mcc = (tp * tn - fp * fn) / denom if denom > 0 else 0.0
    return {'accuracy': acc, 'mcc': mcc}
```

## Common pitfalls

- Threshold tuning for unsupervised baselines (e.g., Lau et al.'s WLPM) is performed via 10-fold cross-validation on the CoLA test set itself, which can inflate performance if not strictly separated from training.
- Human performance is reported as both 'Average' and 'Aggregate' agreement, which differ significantly and should not be conflated with model scores.
- The out-of-domain evaluation set is not described in this section, making replication of that specific metric difficult.

## Evidence (verbatim from paper)

> We evaluate these classifiers on CoLA without CoLA training. ... We train 20 pooling classifiers end-to-end on real/fake data with BNC embeddings, 20 with GloVe, and 20 with ELMo-style embeddings for up to 7 days or until completing 4 epochs without improving in development MCC.

## Citation

```bibtex
@misc{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt et al. (2018)},
  year={2018},
  note={arXiv:1805.12471}
}
```

- arXiv: 1805.12471