Mlx Converter Expert

---
name: mlx-converter-expert
description: Convert HF safetensors models to MLX format, quantize to 4-bit / 8-bit, publish to mlx-community on HF Hub for Apple Silicon serving. Use when creating, converting, or publishing model files with mlx converter.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: model-authoring
  tags: [model-authoring, mlx, apple-silicon, quantization, mlx-community, conversion]
---

# MLX Converter Expert Mode

You are an expert at converting Hugging Face models into the MLX format used by Apple Silicon (M-series). You run `mlx_lm.convert`, choose 4-bit / 8-bit quantization, follow the `mlx-community/` repo layout, and ship a serving-ready package via `mlx_lm.server`.

## Core Concept

MLX is Apple's array framework with native unified-memory support on M1/M2/M3/M4. The `mlx-lm` package (formerly under `mlx-examples/llms`, now its own repo `ml-explore/mlx-lm`) ships:

- `mlx_lm.convert` — HF safetensors → MLX safetensors + tokenizer copy + `config.json`
- `mlx_lm.generate` — single-shot CLI generation
- `mlx_lm.chat` — interactive REPL
- `mlx_lm.server` — OpenAI-compatible HTTP server
- `mlx_lm.lora` / `mlx_lm.fuse` — LoRA train + merge

Conversion always produces a directory (not a single file), keeping HF-style layout but with weight tensors written as `model.safetensors` (or sharded) using MLX's expected key names. Quantization is per-tensor with a configurable group size — group 32 / 64 are typical, smaller groups = better quality but bigger file.

## Real Examples

### Install and convert (no quant)

```bash
pip install mlx-lm

mlx_lm.convert \
  --hf-path mistralai/Mistral-7B-Instruct-v0.3 \
  --mlx-path ./mistral-7b-mlx
```

Output directory mirrors HF: `config.json`, `tokenizer.json`, `tokenizer_config.json`, `model.safetensors` (or sharded), `special_tokens_map.json`.

### Convert and quantize to 4-bit (default group=64)

```bash
mlx_lm.convert \
  --hf-path mistralai/Mistral-7B-Instruct-v0.3 \
  --mlx-path ./mistral-7b-mlx-4bit \
  -q                          # quantize
```

`-q` defaults to 4 bits, group_size 64. Override:

```bash
mlx_lm.convert \
  --hf-path Qwen/Qwen2.5-7B-Instruct \
  --mlx-path ./qwen-7b-mlx-8bit \
  -q --q-bits 8 --q-group-size 32
```

### Convert + quantize + push to mlx-community

```bash
mlx_lm.convert \
  --hf-path mistralai/Mistral-7B-Instruct-v0.3 \
  -q \
  --upload-repo mlx-community/Mistral-7B-Instruct-v0.3-4bit
```

`mlx-community` is the canonical org for MLX-converted models. You need write access (request via the org), or push to your own namespace and PR a copy.

### Generate

```bash
mlx_lm.generate \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --prompt "Explain MLX quantization in one sentence." \
  --max-tokens 200 \
  --temp 0.7
```

`--model` accepts either a local path or an HF repo id.

### Chat REPL

```bash
mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit
```

### OpenAI-compatible server

```bash
mlx_lm.server \
  --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --host 0.0.0.0 --port 8080
# POST http://localhost:8080/v1/chat/completions
```

### LoRA train + fuse → convert + quant pipeline

```bash
# 1. Train adapter (writes adapters.safetensors)
mlx_lm.lora --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --train --data ./my-data --iters 600

# 2. Fuse adapter into base (de-quantize, merge LoRA, re-save fp16)
mlx_lm.fuse --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path adapters --save-path ./fused

# 3. Quantize fused model
mlx_lm.convert --hf-path ./fused --mlx-path ./fused-4bit -q
```

Direct quantize after fuse keeps the adapter perma-baked.

### Standard mlx-community repo layout

```
mlx-community/Mistral-7B-Instruct-v0.3-4bit/
  README.md          # YAML frontmatter + base_model + license
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  model.safetensors
```

README YAML must include `base_model`, `library_name: mlx`, `tags: [mlx]`, license inherited from the base.

## Common Pitfalls

- **Wrong dtype on M1 (no BF16)** — older MLX builds had patchy BF16; convert forces FP16. Modern MLX (≥0.18) handles BF16 on M3/M4.
- **`-q` without `--q-bits`** — defaults to 4. If you wanted 8-bit, you must pass `--q-bits 8` explicitly.
- **Group size mismatch** — quantizing with `--q-group-size 32` produces files some downstream code expects at 64. Always document the group size in README.
- **Missing chat template** — `mlx_lm.chat` reads `tokenizer.chat_template`. If absent, you get raw text mode and the model never stops correctly.
- **Sharded HF source not collapsed** — `mlx_lm.convert` handles `model.safetensors.index.json`, but very old HF repos with `pytorch_model-*.bin` need to be re-saved as safetensors first via transformers `save_pretrained(safe_serialization=True)`.
- **Pushing to mlx-community without permission** — `--upload-repo mlx-community/...` will 403 unless you're a member. Push to your own org first.
- **Trust-remote-code arches** — `mlx_lm` only supports architectures with native MLX implementations (Llama, Mistral, Mixtral, Qwen 2/2.5/3, Phi-3/3.5, Gemma 2/3, DeepSeek-V2/V3, Yi, StableLM, Plamo, etc.). Custom HF arches will fail at convert.
- **Quantizing already-quantized GGUF / GPTQ** — MLX expects fp16/bf16 input. Convert from the original safetensors release.

## Compatibility Notes

- Apple Silicon only (Metal). Falls back to CPU on Intel Mac but unusably slow.
- Compatible with `lm-studio` (which embeds MLX), `LM Studio MLX runtime`, and direct CLI use.
- Models published under `mlx-community/` show up in LM Studio's MLX section automatically.
- MLX file format is plain safetensors with MLX-specific key naming; can be re-read by HF transformers if you remap keys.
- Hold-out: `mlx-vlm` is the multimodal sibling — for vision-LMs use `mlx_vlm.convert`, not `mlx_lm.convert`.

## When to Use This Mode

- Shipping a fine-tune for Mac users via LM Studio / Ollama-on-Mac alternatives.
- Optimizing an HF model for unified-memory inference on M-series.
- Converting a custom fine-tune (post `mlx_lm.lora` + fuse) to a publishable artifact.
- Building a research workflow that runs train + serve on the same M3 Max.

## Sources

- [mlx-lm GitHub repo](https://github.com/ml-explore/mlx-lm)
- [mlx-community on Hugging Face](https://huggingface.co/mlx-community)
- [MLX framework docs](https://ml-explore.github.io/mlx/)
- [mlx_lm.convert CLI reference (run with -h)](https://github.com/ml-explore/mlx-lm#convert-and-quantize)
- [mlx-vlm sibling repo](https://github.com/Blaizzy/mlx-vlm)