Clue Eval

---
name: clue-eval
description: Evaluates Chinese language understanding across nine diverse tasks, including text classification, natural language inference, semantic similarity, and machine reading comprehension. It probes a model's ability to handle Chinese-specific linguistic phenomena, whole-word masking, and token-level vs. global understanding through a standardized fine-tuning pipeline. Use when the user wants to benchmark on CLUE, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.05986
  bibtex_key: xu2020clue
  confidence: high
---

# clue-eval

> CLUE: A Chinese Language Understanding Evaluation Benchmark — Xu et al. (2020) (arXiv:2004.05986, 2020)

## What this evaluates

Evaluates Chinese language understanding across nine diverse tasks, including text classification, natural language inference, semantic similarity, and machine reading comprehension. It probes a model's ability to handle Chinese-specific linguistic phenomena, whole-word masking, and token-level vs. global understanding through a standardized fine-tuning pipeline.

## Datasets

- **CLUE** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted class labels out of the total number of instances. Computed as the number of matches divided by the total number of test samples.
- `Exact Match (EM)` — range: [0, 1]
  - A binary score indicating whether the predicted answer span exactly matches the gold answer span character-for-character. Used exclusively for the CMRC 2018 machine reading comprehension task.

## Input / output format

**Input**: Single sentences for classification tasks, sentence pairs for NLI/semantic similarity, and context-question pairs for machine reading comprehension (span extraction or multiple-choice).

**Output**: Class labels for classification/NLI tasks, start and end token indices for span-extraction MRC, or a selected option index for multiple-choice MRC.

## Scoring recipe

```python
def compute_metric(predictions, golds, task_name):
    if task_name == 'CMRC':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    else:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
```

## Common pitfalls

- Using English-optimized subword tokenizers (e.g., SentencePiece) on Chinese text severely degrades performance on token-level tasks like span extraction.
- Assuming human performance is a reliable upper bound; some tasks (e.g., TNEWS, C3) are actually harder for humans than for machines due to ambiguous labels or complex reasoning.
- Averaging scores across all tasks without considering varying class distributions or task difficulties can mask significant performance disparities.

## Evidence (verbatim from paper)

> We report EM for CMRC 2018 and accuracy for all other tasks.

## Citation

```bibtex
@misc{xu2020clue,
  title={CLUE: A Chinese Language Understanding Evaluation Benchmark},
  author={Xu et al. (2020)},
  year={2020},
  note={arXiv:2004.05986}
}
```

- arXiv: 2004.05986