Glm 130b Eval

---
name: glm-130b-eval
description: This evaluation protocol assesses the zero-shot and few-shot capabilities of large bilingual language models across diverse English and Chinese benchmarks. It probes language modeling, multi-choice question answering, reasoning, commonsense, and cross-lingual transfer abilities. Use when the user wants to benchmark on LAMBADA, Pile, MMLU, BIG-bench-lite, CLUE, FewCLUE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.02414
  bibtex_key: zeng2022glm130b
  confidence: high
---

# glm-130b-eval

> GLM-130B: An Open Bilingual Pre-trained Model — Aohan Zeng et al. (arXiv:2210.02414, 2022)

## What this evaluates

This evaluation protocol assesses the zero-shot and few-shot capabilities of large bilingual language models across diverse English and Chinese benchmarks. It probes language modeling, multi-choice question answering, reasoning, commonsense, and cross-lingual transfer abilities.

## Datasets

- **LAMBADA** — total ?; splits: test (-1)
- **Pile** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **BIG-bench-lite** — total ?; splits: test (-1)
- **CLUE** — total ?; splits: dev (-1)
- **FewCLUE** — total ?; splits: dev (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or next words compared to ground truth. Calculated as the mean of binary matches between normalized predictions and gold answers across all instances.
- `BPB` — range: other
  - Bits Per Byte, computed as the negative log-likelihood of the test set normalized by the total number of bytes. Used for language modeling evaluation on the Pile.

## Input / output format

**Input**: Text prompts formatted for zero-shot or few-shot evaluation. For multiple-choice tasks, the prompt includes the question and options, optionally preceded by in-context examples. For language modeling, the prompt is a text sequence ending with a blank or partial sentence.

**Output**: Predicted next token(s) or selected option label. For multiple-choice QA, the model outputs the chosen answer string or letter. For language modeling, it generates the continuation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return correct / len(gold_labels)

def compute_bpb(log_probs, byte_counts):
    total_bits = sum(log_probs * byte_counts)
    return total_bits / sum(byte_counts)
```

## Common pitfalls

- Data contamination: Datasets must be filtered for 13-gram overlap with training corpora and excluded if released after training data crawling.
- Zero-shot definition: The paper uses a specific criterion involving unseen class labels or obvious domain transfer, which may differ from standard community usage.
- Few-shot bias: Models trained with MIP are biased toward zero-shot performance and may show weaker few-shot scaling compared to autoregressive models.

## Evidence (verbatim from paper)

> LAMBADA (Paperno et al., 2016) is a dataset to test the last word language modeling capability. The results previously shown in Figure 2 suggest GLM-130B achieves a zero-shot accuracy of 80.2 with its bidirectional attention, setting up a new record on LAMBADA.

## Citation

```bibtex
@misc{zeng2022glm130b,
  title={GLM-130B: An Open Bilingual Pre-trained Model},
  author={Aohan Zeng et al.},
  year={2022},
  note={arXiv:2210.02414}
}
```

- arXiv: 2210.02414