Tiered Data Management Eval

---
name: tiered-data-management-eval
description: Evaluates the impact of tiered data management (L1–L3) on model performance across general knowledge, reasoning, math, and code domains. It compares models trained on different data quality tiers and different training strategies (mix vs. tiered) to validate data curation and scheduling efficacy. Use when the user wants to benchmark on OpenCompass Benchmarks, or asks about evaluating this task. Reports Average benchmark scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09003
  bibtex_key: wang2026tiereddatamanagement
  confidence: high
---

# tiered-data-management-eval

> Data Science and Technology Towards AGI Part I: Tiered Data Management — Wang et al. (2026) (arXiv:2602.09003, 2026)

## What this evaluates

Evaluates the impact of tiered data management (L1–L3) on model performance across general knowledge, reasoning, math, and code domains. It compares models trained on different data quality tiers and different training strategies (mix vs. tiered) to validate data curation and scheduling efficacy.

## Datasets

- **OpenCompass Benchmarks** — total ?; splits: test (-1); repo https://github.com/open-compass/opencompass

## Metrics

- `Average benchmark scores` **(primary)** — range: percent
  - Arithmetic mean of per-benchmark accuracy scores across a specified domain (English, Chinese, Math, Code) or overall.
- `Per-benchmark accuracy` — range: percent
  - Standard accuracy: fraction of correctly answered multiple-choice or generation tasks. For PPL benchmarks, accuracy is derived from log-likelihood over answer choices.

## Input / output format

**Input**: Few-shot prompts (0 to 8 shots) containing questions, code problems, or math reasoning tasks.

**Output**: Model-generated text, code, or selected answer choices.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if extract_final_answer(pred) == gold:
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- PPL-based benchmarks (MMLU, ARC, etc.) evaluate log-likelihood over predefined answer choices, not generation probability.
- Generation benchmarks (BBH, MATH, GSM8K, MBPP, HumanEval) require robust answer extraction or code execution, which can fail due to minor formatting differences.
- The paper reports domain-specific averages (English, Chinese, Math, Code) and an overall average; mixing these up leads to incorrect comparisons.

## Evidence (verbatim from paper)

> Across all four domains, downstream performance improves steadily from L1 to L3, demonstrating that data quality increases with each data tier. Specifically, average benchmark scores rise from 52.26 percentage points (pp) to 53.96pp in English (+1.70 pp), 49.44pp to 51.48pp in Chinese (+2.04 pp), 23.78pp to 30.84pp in Math (+7.06 pp), and 34.49pp to 36.28pp in Code (+1.79 pp).

## Citation

```bibtex
@misc{wang2026tiereddatamanagement,
  title={Data Science and Technology Towards AGI Part I: Tiered Data Management},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.09003}
}
```

- arXiv: 2602.09003