Skywork Benchmark Eval

---
name: skywork-benchmark-eval
description: Evaluates bilingual foundation models on general knowledge, Chinese domain-specific reasoning, mathematical problem-solving, and language modeling capabilities using standardized benchmarks and custom held-out text corpora. Use when the user wants to benchmark on MMLU, CEVAL, CMMLU, GSM8K, Custom Chinese LM Testset, or asks about evaluating this task. Reports 5-shot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.19341
  bibtex_key: wei2023skywork
  confidence: high
---

# skywork-benchmark-eval

> Skywork: A More Open Bilingual Foundation Model — Wei et al. (2023) (arXiv:2310.19341, 2023)

## What this evaluates

Evaluates bilingual foundation models on general knowledge, Chinese domain-specific reasoning, mathematical problem-solving, and language modeling capabilities using standardized benchmarks and custom held-out text corpora.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **CEVAL** — total 13948; splits: test (13948)
- **CMMLU** — total ?; splits: test (-1)
- **GSM8K** — total 8500; splits: test (8500)
- **Custom Chinese LM Testset** — total ?; splits: test (-1)

## Metrics

- `5-shot accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions across MMLU, CEVAL, and CMMLU. Computed as (number of correct predictions / total instances) * 100. Evaluated with 5 few-shot examples per prompt.
- `8-shot accuracy` — range: percent
  - Percentage of correctly solved grade-school math word problems in GSM8K. Computed as (number of correct predictions / total instances) * 100. Evaluated with 8 few-shot examples per prompt.
- `perplexity` — range: other
  - Exponential of the average negative log-likelihood of the ground truth tokens over the custom Chinese language modeling testset. Lower values indicate better language modeling capability.

## Input / output format

**Input**: Multiple-choice questions or math word problems accompanied by 5 or 8 few-shot examples for accuracy benchmarks; unlabeled natural text documents published after September 1, 2023 for language modeling.

**Output**: Model's predicted answer option or generated text sequence.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

def compute_perplexity(model, testset):
    total_log_prob = 0
    total_tokens = 0
    for doc in testset:
        log_probs = model.log_prob(doc)
        total_log_prob += sum(log_probs)
        total_tokens += len(doc)
    return math.exp(-total_log_prob / total_tokens)
```

## Common pitfalls

- Static benchmarks are prone to data contamination over time; the authors mitigate this for their custom LM testset by enforcing a strict post-training cutoff date (Sept 1, 2023).
- Few-shot settings differ across benchmarks (5-shot for knowledge, 8-shot for GSM8K), requiring consistent prompt formatting to avoid unfair comparisons.
- Perplexity evaluation is restricted to Chinese domains only, making cross-lingual language modeling comparisons impossible.

## Evidence (verbatim from paper)

> The metrics for CEVAL, CMMLU and MMLU are 5-shot accuracy, while for GSM8K it is 8-shot accuracy. Higher numbers indicate better performance.

## Citation

```bibtex
@misc{wei2023skywork,
  title={Skywork: A More Open Bilingual Foundation Model},
  author={Wei et al. (2023)},
  year={2023},
  note={arXiv:2310.19341}
}
```

- arXiv: 2310.19341