Jcola Eval

---
name: jcola-eval
description: This benchmark evaluates the ability of neural language models to judge the grammatical acceptability of Japanese sentences. It probes deep syntactic knowledge, particularly long-distance dependencies and linguistic phenomena, by measuring performance on both in-domain and out-of-domain acceptability judgments. Use when the user wants to benchmark on JCoLA, or asks about evaluating this task. Reports Matthews Correlation Coefficient (MCC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.12676
  bibtex_key: someya2023jcola
  confidence: high
---

# jcola-eval

> JCoLA: Japanese Corpus of Linguistic Acceptability — Someya et al. (2023) (arXiv:2309.12676, 2023)

## What this evaluates

This benchmark evaluates the ability of neural language models to judge the grammatical acceptability of Japanese sentences. It probes deep syntactic knowledge, particularly long-distance dependencies and linguistic phenomena, by measuring performance on both in-domain and out-of-domain acceptability judgments.

## Datasets

- **JCoLA** — total 10020; splits: dev (-1); repo https://github.com/osekilab/JCoLA

## Metrics

- `Matthews Correlation Coefficient (MCC)` **(primary)** — range: [-1, 1]
  - A correlation coefficient between the observed and predicted binary classifications. It accounts for true and false positives and negatives, making it robust for unbalanced binary classification tasks.

## Input / output format

**Input**: A single Japanese sentence.

**Output**: Binary acceptability label (acceptable vs. unacceptable).

## Scoring recipe

```python
def evaluate_jcola(model, dev_data, test_data):
    best_avg_test_mcc = -1.0
    for lr in [5e-5, 3e-5, 2e-5]:
        dev_mccs = []
        test_mccs = []
        for seed in range(20):
            m = train_and_finetune(model, dev_data, lr, seed, epochs=5)
            dev_mcc = compute_mcc(m.predict(dev_data), dev_data.labels)
            if dev_mcc < 0:
                continue
            dev_mccs.append(dev_mcc)
            test_mccs.append(compute_mcc(m.predict(test_data), test_data.labels))
        if test_mccs:
            avg_test_mcc = sum(test_mccs) / len(test_mccs)
            if avg_test_mcc > best_avg_test_mcc:
                best_avg_test_mcc = avg_test_mcc
    return best_avg_test_mcc
```

## Common pitfalls

- The dataset is unbalanced, so accuracy is misleading; MCC must be used as the primary metric.
- Any configuration or seed yielding a negative MCC on the development set must be discarded entirely.
- Final scores must be averaged across 20 different random seeds per configuration to mitigate randomness.

## Evidence (verbatim from paper)

> In addition, the language models are trained using three different learning rates (5e-5, 3e-5, and 2e-5) and we evaluate models which achieved the highest Matthews Correlation Coefficient (MCC; Matthews (1975)) on the development data. This evaluation metric is an evaluation metric suitable for unbalanced binary classifiers also used in Warstadt et al. (2019). For each configuration, we trained 20 models with different random seeds to mitigate the effect of randomness. The score for each language model is calculated as the average across 20 different random seeds, but we ignore those results where the models achieved less than zero MCC score on the development set, as in Warstadt and Bowman (2020).

## Citation

```bibtex
@misc{someya2023jcola,
  title={JCoLA: Japanese Corpus of Linguistic Acceptability},
  author={Someya et al. (2023)},
  year={2023},
  note={arXiv:2309.12676}
}
```

- arXiv: 2309.12676