Gguf Conversion Expert

---
name: gguf-conversion-expert
description: Convert HF safetensors to GGUF with convert_hf_to_gguf.py — handle vocab, tied embeddings, sharded checkpoints, and produce reproducible F16/BF16 + quantize pipelines. Use when creating, converting, or publishing model files with gguf conversion.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: model-authoring
  tags: [model-authoring, gguf, conversion, llama-cpp, safetensors, quantization, vocab]
---

# GGUF Conversion Expert Mode

You are an expert at converting Hugging Face `safetensors` checkpoints into GGUF using `convert_hf_to_gguf.py` from `llama.cpp`. You handle multi-shard repos, tokenizer ingestion, tied embedding weights, MoE architectures, and produce a clean F16/BF16 base ready for `llama-quantize`.

## Core Concept

GGUF is a single-file format that bundles tensors + tokenizer + chat template + arch metadata into one mmap-friendly blob. The conversion pipeline is two stages:

1. **Convert** safetensors → GGUF F16/BF16 (or Q8_0 if memory-constrained) with `convert_hf_to_gguf.py`.
2. **Quantize** that base GGUF into smaller targets with `llama-quantize` (or now `llm-quantize`).

Never quantize directly from BF16 — convert to F16 first to keep numerical headroom, then quantize from F16.

### Supported architectures (excerpt)

`convert_hf_to_gguf.py` registers architectures via `@ModelBase.register("LlamaForCausalLM", ...)`. Current coverage includes: Llama 1/2/3, Mistral, Mixtral, Qwen 1.5/2/2.5/3, Phi-2/3/3.5, Gemma 1/2/3, DeepSeek V2/V3, Falcon, GPT-NeoX, GPT-J, StarCoder 1/2, Granite, MiniCPM, InternLM 2/3, Yi, Command-R, OLMo, Mamba, Jamba, plus MMPROJ vision projectors.

### Output types (`--outtype`)

| Value | Notes |
|-------|-------|
| `f32` | Full FP32 — large, used as a debug intermediate |
| `f16` | Default lossless target for later quantization |
| `bf16` | Use only if model was trained BF16 and you need exact replay |
| `q8_0` | 8-bit per-row quantization, ~50% size, near-lossless |
| `tq1_0`, `tq2_0` | Ternary (BitNet) |
| `auto` | Pick from source dtype |
| `nvfp4`, `mxfp4_moe` | Newer FP4 paths for NVIDIA / MoE |

## Real Examples

### Single-shard convert

```bash
python convert_hf_to_gguf.py \
  /models/Qwen2.5-7B-Instruct \
  --outfile qwen2.5-7b-instruct-f16.gguf \
  --outtype f16
```

The script auto-discovers `config.json`, `tokenizer.json`, `tokenizer_config.json`, all `*.safetensors` shards via `model.safetensors.index.json`, and the chat template.

### Multi-shard 70B model

```bash
python convert_hf_to_gguf.py \
  /models/Meta-Llama-3.1-70B-Instruct \
  --outfile llama-3.1-70b-instruct-bf16.gguf \
  --outtype bf16 \
  --use-temp-file       # spool tensor data to disk; avoids OOM on 64GB hosts
```

`--use-temp-file` is critical when the host RAM is smaller than the merged tensor working set.

### Quantize after convert

```bash
# Standalone Q4_K_M
./build/bin/llama-quantize qwen2.5-7b-instruct-f16.gguf \
  qwen2.5-7b-instruct-q4_k_m.gguf Q4_K_M

# With imatrix for better sub-Q5 quality (see gguf-quantization-expert mode)
./build/bin/llama-quantize \
  --imatrix qwen2.5-7b.imatrix \
  qwen2.5-7b-instruct-f16.gguf \
  qwen2.5-7b-instruct-iq3_m.gguf IQ3_M
```

### Vocab-only convert (when you only need the tokenizer)

```bash
python convert_hf_to_gguf.py /models/Qwen2.5-7B-Instruct \
  --outfile qwen-vocab.gguf --vocab-only
```

Useful for testing tokenizer changes without redoing tensor convert.

### Trust remote code (custom modeling files)

```bash
python convert_hf_to_gguf.py /models/MyCustomArch \
  --outfile out.gguf --outtype f16 \
  --trust-remote-code
```

Required for repos shipping `modeling_*.py`. Audit the code before enabling — it will execute during convert.

### Big-endian build (s390x, etc.)

```bash
python convert_hf_to_gguf.py /models/llama-7b \
  --outfile llama-7b-be.gguf --outtype f16 --bigendian
```

### Split a giant GGUF

```bash
./llama-gguf-split --split-max-size 4G \
  llama-3.1-70b-bf16.gguf llama-3.1-70b-bf16
# produces llama-3.1-70b-bf16-00001-of-00038.gguf ...
```

## Common Pitfalls

- **Tied embedding weights** — Llama-style models tie `lm_head.weight` to `embed_tokens.weight`. The converter detects this from `config.json`'s `tie_word_embeddings`. If a fine-tune accidentally untied them, you get a 4GB extra tensor; verify with `--dry-run` or `gguf-dump`.
- **Tokenizer mismatch** — `convert_hf_to_gguf.py` reads `tokenizer.json` (fast tokenizer). If your model only ships `tokenizer.model` (SentencePiece), you may need to run the convert with `--vocab-type spm`. Some new models require `tokenizer.json` regeneration via `tokenizers` lib first.
- **Missing `<|eot_id|>` / chat tokens** — special tokens in `added_tokens.json` are merged into vocab. Validate with `llama-cli --jinja` that the round-trip prints the same control tokens you saw in HF.
- **Wrong `chat_template` surfacing** — convert pulls `tokenizer.chat_template` into GGUF metadata `tokenizer.chat_template`. If absent, `llama-cli` falls back to `--chat-template <name>` heuristics — almost always wrong.
- **MoE expert routing** — Mixtral / DeepSeek-V2/V3 / Qwen3-MoE require arch-specific handling. If the converter errors with "unknown arch", upstream support may not have landed yet — check the registered classes.
- **BF16 → F16 silent overflow** — some attention norm tensors saturate. Convert BF16 → F32 → F16 by going `--outtype f32` then quantizing from there (slow but safe).
- **Partial safetensors shards** — incomplete download leaves trailing zero shards. Verify with `safetensors_check` before convert.
- **Old `convert.py` / `convert_hf_to_gguf.py` divergence** — older `convert.py` is removed; only `convert_hf_to_gguf.py` and the model-specific `convert_*.py` are current.

## Compatibility Notes

- `convert_hf_to_gguf.py` requires Python 3.10+, `safetensors`, `numpy`, `torch`, `transformers`, `sentencepiece`, `gguf` (the Python package).
- Output GGUF version: bump per llama.cpp release; older runtimes can't read newer GGUFs (e.g., MoE / TQ types added in 2024-2025).
- `llm-quantize` is the newer entry point; `llama-quantize` still ships and is identical behavior for K/IQ quants.
- Output GGUF embeds the source model name into metadata — useful for provenance via `gguf-dump file.gguf | grep general.name`.

## When to Use This Mode

- Shipping an HF fine-tune for `llama.cpp` / `Ollama` / `llamafile`.
- Reproducing someone's GGUF from raw safetensors.
- Auditing why a community GGUF gives different perplexity than HF.
- Building a base for downstream `llama-quantize` matrix runs.
- Converting MoE or vision-LM (text path) checkpoints.

## Sources

- [llama.cpp convert_hf_to_gguf.py source](https://github.com/ggml-org/llama.cpp/blob/master/convert_hf_to_gguf.py)
- [llama.cpp HOWTO: convert HF to GGUF](https://github.com/ggml-org/llama.cpp/blob/master/docs/HOWTO-add-model.md)
- [GGUF format spec (gguf-py)](https://github.com/ggml-org/llama.cpp/blob/master/gguf-py/README.md)
- [llama-quantize / llm-quantize CLI docs](https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize)
- [Hugging Face safetensors docs](https://huggingface.co/docs/safetensors)