Llm Pretrain Finetune Eval

---
name: llm-pretrain-finetune-eval
description: Evaluates memory-efficient gradient compression optimizers against full-rank baselines during LLM pre-training and fine-tuning, measuring final model quality, convergence speed, memory footprint, and training throughput. Use when the user wants to benchmark on C4, MMLU, GLUE, or asks about evaluating this task. Reports Validation PPL, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.07237
  bibtex_key: wen2025wavelet
  confidence: high
---

# llm-pretrain-finetune-eval

> Wavelet Meets Adam: Compressing Gradients for Memory-Efficient Training — Wen et al. (2025) (arXiv:2501.07237, 2025)

## What this evaluates

Evaluates memory-efficient gradient compression optimizers against full-rank baselines during LLM pre-training and fine-tuning, measuring final model quality, convergence speed, memory footprint, and training throughput.

## Datasets

- **C4** — total ?; splits: val (-1)
- **MMLU** — total ?; splits: test (-1); HF `cais/mmlu`
- **GLUE** — total ?; splits: test (-1); HF `glue`

## Metrics

- `Validation PPL` **(primary)** — range: other
  - Perplexity computed on the validation set during pre-training; lower values indicate better language modeling performance.
- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers on MMLU and GLUE tasks; averaged across sub-tasks for MMLU.
- `Memory Usage` — range: other
  - Estimated peak optimizer memory footprint in gigabytes (GB) during training.
- `Training Throughput` — range: other
  - Number of training tokens processed per second per GPU.

## Input / output format

**Input**: Tokenized text sequences for pre-training; multiple-choice question-answer pairs for MMLU; task-specific prompts for GLUE.

**Output**: Next-token probability distributions for pre-training; discrete class labels for MMLU/GLUE fine-tuning.

## Scoring recipe

```python
PPL: exp(-mean(log_softmax(logits)))
Accuracy: sum(predictions == gold_labels) / total_samples * 100
Memory: peak GPU memory allocated for optimizer states
Throughput: total_tokens / (training_time * num_gpus)
```

## Common pitfalls

- Memory estimates are approximations and depend heavily on framework implementation and optimizer state tracking.
- Throughput measurements are hardware-specific (RTX 3090 used) and may not generalize across GPU architectures.
- Fair comparison requires sweeping learning rates for each optimizer due to differing sensitivities to hyperparameters.

## Evidence (verbatim from paper)

> The final validation perplexity (PPL) and estimated memory usage for all methods are summarized in Table [1]. Experimental results show that GWT consistently achieves lower validation PPL while reducing memory usage compared to other memory-efficient baselines.

## Citation

```bibtex
@misc{wen2025wavelet,
  title={Wavelet Meets Adam: Compressing Gradients for Memory-Efficient Training},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2501.07237}
}
```

- arXiv: 2501.07237