Cci30 Hq Eval

---
name: cci30-hq-eval
description: Evaluates the effectiveness of a high-quality Chinese pre-training corpus by training a 0.5B language model from scratch and measuring its zero-shot generalization across standard English and Chinese knowledge benchmarks. The protocol also includes a comparative analysis of data quality classifiers using macro F1-score on a held-out test set. Use when the user wants to benchmark on Standard Benchmarks (ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, SIQA, CEval, CMMLU), or asks about evaluating this task. Reports Average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.18505
  bibtex_key: wang2024cci30hq
  confidence: high
---

# cci30-hq-eval

> CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models — Wang et al. (2024) (arXiv:2410.18505, 2024)

## What this evaluates

Evaluates the effectiveness of a high-quality Chinese pre-training corpus by training a 0.5B language model from scratch and measuring its zero-shot generalization across standard English and Chinese knowledge benchmarks. The protocol also includes a comparative analysis of data quality classifiers using macro F1-score on a held-out test set.

## Datasets

- **Standard Benchmarks (ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, SIQA, CEval, CMMLU)** — total ?; splits: test (-1)

## Metrics

- `Average` **(primary)** — range: [0, 1]
  - Arithmetic mean of zero-shot accuracy scores across 10 standard benchmarks (ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, SIQA, CEval, CMMLU).

## Input / output format

**Input**: Zero-shot natural language prompts corresponding to each benchmark task (multiple-choice or open-ended QA).

**Output**: Model-generated text or selected option corresponding to the ground truth answer.

## Scoring recipe

```python
def compute_average_accuracy(predictions, golds, benchmarks):
    task_scores = []
    for task in benchmarks:
        acc = sum(1 for p, g in zip(predictions[task], golds[task]) if p == g) / len(golds[task])
        task_scores.append(acc)
    return sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Benchmarks are evaluated in zero-shot mode, unlike standard few-shot protocols for tasks like MMLU.
- The 'Average' metric treats all 10 benchmarks equally despite varying difficulty and domain coverage.
- Classifier evaluation uses a fixed 0-5 score threshold of 3.0, which may not align with optimal precision-recall trade-offs across different data distributions.

## Evidence (verbatim from paper)

> Evaluation metrics include: Average_Chinese : Average score of Chinese metrics, including CEval and CMMLU. Average_English : Average score across standard English metrics such as ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA and SIQA. Average: Combined average score of all evaluation metrics above.

## Citation

```bibtex
@misc{wang2024cci30hq,
  title={CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2410.18505}
}
```

- arXiv: 2410.18505