Chinese Llm Bench Eval

---
name: chinese-llm-bench-eval
description: Evaluates Chinese large language models' world knowledge, academic understanding, and multi-dimensional alignment after pretraining or instruction fine-tuning. It probes the model's ability to follow instructions, reason across domains, and maintain safety and helpfulness standards in Chinese. Use when the user wants to benchmark on C-Eval, CMMLU, Alignbench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.08197
  bibtex_key: yu2025opencsg
  confidence: high
---

# chinese-llm-bench-eval

> OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training — Yu et al. (2025) (arXiv:2501.08197, 2025)

## What this evaluates

Evaluates Chinese large language models' world knowledge, academic understanding, and multi-dimensional alignment after pretraining or instruction fine-tuning. It probes the model's ability to follow instructions, reason across domains, and maintain safety and helpfulness standards in Chinese.

## Datasets

- **C-Eval** — total ?; splits: test (-1)
- **CMMLU** — total ?; splits: test (-1)
- **Alignbench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions or aligned responses across the benchmark suite. Calculated as (correct predictions / total instances) * 100.

## Input / output format

**Input**: For C-Eval and CMMLU: 5-shot prompting with question and options. For Alignbench: instruction prompts for multi-turn dialogue and task evaluation.

**Output**: Deterministic text generation (temperature=0). Expected to output the correct option letter/answer for C-Eval/CMMLU, or aligned text for Alignbench.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Using temperature > 0 for deterministic multiple-choice benchmarks like C-Eval/CMMLU introduces unnecessary variance and breaks reproducibility.
- Failing to strip markdown tokens during preprocessing causes models to overfit to formatting rather than semantics, skewing benchmark scores.
- Evaluating only on accuracy without multi-dimensional alignment metrics (e.g., safety, helpfulness) misses instruction-tuning quality.

## Evidence (verbatim from paper)

> We assessed model performance on the C-Eval (Huang et al., 2023) and CMMLU (Li et al., 2023) benchmarks, two authoritative suites for evaluating Chinese NLP models' world-knowledge and understanding abilities. During evaluation, we set the temperature to 0 (for deterministic outputs) and used a 5-shot prompting strategy to provide limited contextual examples. As illustrated in Figure 5, the Fineweb-Edu-Chinese model experienced a sharp accuracy increase around 45k steps, surpassing the baseline by a significant margin.

## Citation

```bibtex
@misc{yu2025opencsg,
  title={OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2501.08197}
}
```

- arXiv: 2501.08197