Ceval Eval

---
name: ceval-eval
description: Evaluates Chinese foundation models' domain knowledge and reasoning capabilities across 52 academic disciplines and four difficulty levels using multiple-choice questions. It probes the models' ability to follow instructions, perform in-context learning, and generate chain-of-thought reasoning in a Chinese language context. Use when the user wants to benchmark on C-EVAL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.08322
  bibtex_key: huang2023ceval
  confidence: high
---

# ceval-eval

> C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models — Huang et al. (2023) (arXiv:2305.08322, 2023)

## What this evaluates

Evaluates Chinese foundation models' domain knowledge and reasoning capabilities across 52 academic disciplines and four difficulty levels using multiple-choice questions. It probes the models' ability to follow instructions, perform in-context learning, and generate chain-of-thought reasoning in a Chinese language context.

## Datasets

- **C-EVAL** — total 13948; splits: val (1346), test (-1); repo https://github.com/hkust-nlp/ceval

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as the number of questions where the extracted answer matches the ground truth option divided by the total number of questions, multiplied by 100.

## Input / output format

**Input**: A multiple-choice question in Chinese with four options (A, B, C, D). For few-shot settings, the input includes the question plus up to five exemplars from the development split.

**Output**: Free-form text generation. The final answer choice is extracted from the model's response using regular expressions to match the option letter.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if pred == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Chain-of-thought prompting often degrades performance on subjects that are not reasoning-intensive.
- Five-shot exemplars can exceed the context window of smaller models, requiring dynamic reduction of demonstrations.
- Instruction-tuned models may suffer accuracy drops in few-shot settings if not explicitly trained on in-context examples.
- Test split labels are not publicly released, so developers must rely on the validation split for development.

## Evidence (verbatim from paper)

> We report the average accuracy, while a detailed breakdown of accuracy per subject is provided in Appendix F. GPT-4 is the only model that exceeds 60% average accuracy, highlighting the challenge presented by C-EVAL.

## Citation

```bibtex
@misc{huang2023ceval,
  title={C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2305.08322}
}
```

- arXiv: 2305.08322