Chinese Llm Benchmarks Eval

---
name: chinese-llm-benchmarks-eval
description: Evaluates the knowledge, reasoning, instruction-following, and safety alignment capabilities of Chinese instruction-tuned LLMs across academic, professional, open-ended, and safety-critical domains. Use when the user wants to benchmark on C-Eval, CMMLU, BELLE-EVAL, SafetyBench, or asks about evaluating this task. Reports log-likelihood.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.18058
  bibtex_key: bai2024coigcqia
  confidence: high
---

# chinese-llm-benchmarks-eval

> COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning — Bai et al. (2024) (arXiv:2403.18058, 2024)

## What this evaluates

Evaluates the knowledge, reasoning, instruction-following, and safety alignment capabilities of Chinese instruction-tuned LLMs across academic, professional, open-ended, and safety-critical domains.

## Datasets

- **C-Eval** — total 13948; splits: test (13948)
- **CMMLU** — total ?; splits: test (-1)
- **BELLE-EVAL** — total ?; splits: test (-1)
- **SafetyBench** — total 11435; splits: test (11435)

## Metrics

- `log-likelihood` **(primary)** — range: [0, 1]
  - For multiple-choice questions, the model selects the option with the highest log-likelihood. Accuracy is the fraction of correct predictions.
- `model-based evaluation score` — range: [0, 1]
  - An LLM-as-judge or automated scoring method used to assess open-ended instruction-following responses on BELLE-EVAL.

## Input / output format

**Input**: Multiple-choice questions with options, or open-ended instruction prompts. SafetyBench uses a few-shot setting with example prompts.

**Output**: For MCQs: the selected answer option letter/text. For open-ended: a generated text response.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'mcq':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    elif task_type == 'open_ended':
        return llm_judge_score(predictions, golds)
```

## Common pitfalls

- Using greedy decoding instead of log-likelihood selection for MCQs, which contradicts the paper's protocol and hurts C-Eval/CMMLU scores.
- BELLE-EVAL requires model-based evaluation rather than exact string matching, so standard NLP metrics will fail.
- SafetyBench is evaluated in a few-shot setting, not zero-shot, which must be replicated for fair comparison.

## Evidence (verbatim from paper)

> We choosing the answer option with the highest log-likelihood as the final prediction of the model. ... We employ sampling generation for generating responses to instructions and use a model-based evaluation method.

## Citation

```bibtex
@misc{bai2024coigcqia,
  title={COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2403.18058}
}
```

- arXiv: 2403.18058