Cuge Eval

---
name: cuge-eval
description: Evaluates Chinese language understanding and generation capabilities across a hierarchical framework. It probes discourse comprehension, conversational interaction, mathematical reasoning, and multilingual tasks using a multi-level scoring strategy that normalizes model performance against a fixed baseline. Use when the user wants to benchmark on CUGE (lite version), or asks about evaluating this task. Reports normalized capability performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.13610
  bibtex_key: yao2021cuge
  confidence: medium
---

# cuge-eval

> CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark — Yao et al. (2021) (arXiv:2112.13610, 2021)

## What this evaluates

Evaluates Chinese language understanding and generation capabilities across a hierarchical framework. It probes discourse comprehension, conversational interaction, mathematical reasoning, and multilingual tasks using a multi-level scoring strategy that normalizes model performance against a fixed baseline.

## Datasets

- **CUGE (lite version)** — total ?; splits: test (-1)

## Metrics

- `normalized capability performance` **(primary)** — range: [0, 1]
  - Raw model performance (e.g., accuracy) normalized against the mT5-Small baseline score on the same task. The exact normalization function is not specified, but it scales model results relative to the 300M-parameter baseline to enable cross-capability comparison.

## Input / output format

**Input**: Chinese language prompts/tasks spanning understanding and generation capabilities (e.g., discourse, conversation, math, summarization).

**Output**: Model-generated responses or predicted labels for each Chinese NLP task instance.

## Scoring recipe

```python
def compute_normalized_score(model_preds, baseline_preds, gold):
    model_acc = accuracy(model_preds, gold)
    baseline_acc = accuracy(baseline_preds, gold)
    # Normalize relative to baseline performance
    normalized = (model_acc - baseline_acc) / (1.0 - baseline_acc)
    return normalized
```

## Common pitfalls

- Uses a 'lite version' of CUGE for evaluation rather than the full benchmark suite.
- Reports normalized scores relative to mT5-Small, which can mask absolute performance gaps or ceiling effects.
- Aggregates metrics across capabilities, tasks, and datasets, potentially hiding weaknesses in specific sub-tasks.

## Evidence (verbatim from paper)

> In our experiments, we select mT5-Small*Xue et al. ([2020](#bib.bib24 ""))*, a representative pre-trained language model with $300$M parameters as our standard baseline model to normalize the performance of models under evaluation. ... We report the experimental results in Table[2](#S3.T2 "Table 2 ‣ NCLS-EnZh ‣ 3.6.2 Cross-lingual Text Summarization ‣ 3.6 Multilingual Capability ‣ 3 Benchmark Framework ‣ CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark") and visualize the normalized capability performance in Figure[3](#S5.F3 "Figure 3 ‣ 5 Using CUGE ‣ CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark")

## Citation

```bibtex
@misc{yao2021cuge,
  title={CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark},
  author={Yao et al. (2021)},
  year={2021},
  note={arXiv:2112.13610}
}
```

- arXiv: 2112.13610