Installs into .claude/skills of the current project.
Are you the author of Lora Adapter Publish Expert?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/anubhavg-icpl-lora-adapter-publish-expert)
---
name: lora-adapter-publish-expert
description: Package and publish LoRA adapters — HF Hub layout, vLLM dynamic loading, llama.cpp LoRA GGUF, Ollama ADAPTER directive, Replicate Cog. Use when creating, converting, or publishing model files with lora adapter publish.
license: CC-BY-NC-SA-4.0
metadata:
risk: unknown
source: community
kind: mode
category: model-authoring
tags: [model-authoring, lora, peft, adapter, vllm, llama-cpp, ollama, replicate, cog]
---
# LoRA Adapter Publish Expert Mode
You are an expert at packaging LoRA / QLoRA adapters for distribution and serving across stacks. You write a clean `adapter_config.json` + `adapter_model.safetensors` pair, push to HF Hub, wire it into vLLM dynamic LoRA, convert it to llama.cpp GGUF LoRA, attach via Ollama `ADAPTER` directive, and ship a Replicate Cog with adapter swap.
## Core Concept
A LoRA adapter is two-row matrices `A` and `B` injected into target linear layers (`q_proj`, `k_proj`, etc.) such that the effective weight becomes `W + (B @ A) * (alpha / r)`. The adapter is small (1-100 MB) compared to the base, and multiple adapters can be hot-swapped at serve time without reloading the base.
### Canonical adapter directory
```
my-lora/
adapter_config.json
adapter_model.safetensors
README.md # model card
tokenizer.json # only if you added tokens
tokenizer_config.json # only if you added tokens
special_tokens_map.json # only if you added tokens
```
### `adapter_config.json` (PEFT current schema)
```json
{
"peft_type": "LORA",
"task_type": "CAUSAL_LM",
"base_model_name_or_path":"meta-llama/Meta-Llama-3.1-8B-Instruct",
"r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"bias": "none",
"target_modules": ["q_proj","k_proj","v_proj","o_proj"],
"modules_to_save": null,
"fan_in_fan_out": false,
"use_rslora": false,
"use_dora": false,
"init_lora_weights": true,
"inference_mode": true
}
```
`base_model_name_or_path` must be the *base*, not another fine-tune, or downstream `from_pretrained` resolves the wrong weights.
### `adapter_model.safetensors` key naming
```
base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight
base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight
... (per layer × per target_module)
```
## Real Examples
### Save with PEFT after training
```python
from peft import PeftModel
peft_model.save_pretrained("./my-lora") # writes both files
peft_model.push_to_hub("yourname/my-lora")
```
### Push to HF Hub manually
```python
from huggingface_hub import HfApi
api = HfApi()
api.create_repo("yourname/my-lora", repo_type="model")
api.upload_folder(folder_path="./my-lora", repo_id="yourname/my-lora")
```
### Load + use with transformers
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
model = PeftModel.from_pretrained(base, "yourname/my-lora")
```
### vLLM static (server start) LoRA
```bash
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--enable-lora \
--max-loras 4 \
--max-lora-rank 64 \
--lora-modules my-lora=yourname/my-lora other=yourname/other-lora
```
```python
from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
c.completions.create(model="my-lora", prompt="hi") # routes through adapter
```
### vLLM dynamic LoRA (load/unload at runtime)
```bash
VLLM_ALLOW_RUNTIME_LORA_UPDATING=true vllm serve meta-llama/... --enable-lora
```
```bash
# Load
curl -X POST http://localhost:8000/v1/load_lora_adapter \
-H "Content-Type: application/json" \
-d '{"lora_name":"my-lora","lora_path":"yourname/my-lora"}'
# Use
curl http://localhost:8000/v1/completions -d \
'{"model":"my-lora","prompt":"hi"}'
# Unload
curl -X POST http://localhost:8000/v1/unload_lora_adapter \
-d '{"lora_name":"my-lora"}'
```
### Convert PEFT LoRA → llama.cpp GGUF LoRA
```bash
python convert_lora_to_gguf.py \
--base meta-llama/Meta-Llama-3.1-8B-Instruct \
--outfile my-lora.gguf \
./my-lora
```
Then in llama-cli:
```bash
./llama-cli -m base-q4_K_M.gguf --lora my-lora.gguf -p "hi"
# or with scaling
./llama-cli -m base.gguf --lora-scaled my-lora.gguf 0.8 -p "hi"
```
### Ollama ADAPTER directive
```
FROM llama3.1:8b-instruct-q4_K_M
ADAPTER ./my-lora # safetensors directory OR
ADAPTER ./my-lora.gguf # converted gguf adapter
SYSTEM "You are a SQL expert."
```
```bash
ollama create yourname/llama-sql -f Modelfile
ollama push yourname/llama-sql
```
Ollama 0.3+ accepts safetensors LoRA directories; older versions need GGUF conversion.
### Replicate Cog with adapter swap
`cog.yaml`:
```yaml
build:
gpu: true
python_version: "3.11"
python_packages:
- peft==0.13.0
- transformers==4.46.0
- torch==2.4.0
predict: "predict.py:Predictor"
```
`predict.py`:
```python
from cog import BasePredictor, Input, Path
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
class Predictor(BasePredictor):
def setup(self):
self.tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
self.base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3.1-8B-Instruct", device_map="auto", torch_dtype="auto")
def predict(self,
prompt: str = Input(description="user prompt"),
lora_repo: str = Input(default="yourname/my-lora")) -> str:
model = PeftModel.from_pretrained(self.base, lora_repo)
ids = self.tok(prompt, return_tensors="pt").to(self.base.device)
out = model.generate(**ids, max_new_tokens=256)
return self.tok.decode(out[0], skip_special_tokens=True)
```
```bash
cog push r8.im/yourname/llama-with-lora
```
## Common Pitfalls
- **Base mismatch** — `adapter_config.base_model_name_or_path` pointing to a quantized or merged base produces silent garbage. Pin the original release.
- **Wrong `target_modules`** — adapter trained on `q_proj,k_proj` will not load on a base whose modules are named `Wq,Wk` (some custom arches). Inspect with `model.named_modules()`.
- **Missing tokenizer** — if you added new tokens during training, you must publish the tokenizer too or downstream `from_pretrained` loads the base tokenizer and your special tokens are unknown.
- **Rank > `--max-lora-rank`** — vLLM rejects with rank-too-high error. Match `--max-lora-rank` to your max trained `r`.
- **DoRA / RsLoRA on engines that don't support them** — llama.cpp's LoRA loader supports plain LoRA only. Convert DoRA to merged weights first via `merge_and_unload()`.
- **GGUF LoRA on different quant** — applying a fp16-trained LoRA on a Q2_K base is technically allowed but quality collapses. Test perplexity before publishing.
- **Multiple adapters with different `r`** — vLLM allows it via `--max-lora-rank` set to the max; but mixing `r=8` and `r=64` in the same batch wastes VRAM.
- **`adapter_model.bin` instead of `.safetensors`** — old PEFT versions wrote pickle; bump PEFT ≥0.12 and re-save.
## Compatibility Notes
- Hugging Face Hub renders LoRA adapters with a "LoRA" tag automatically when `adapter_config.json` is present.
- vLLM dynamic LoRA needs `VLLM_ALLOW_RUNTIME_LORA_UPDATING=true`.
- llama.cpp adapter file is a GGUF with arch=`adapter`, separate from the base GGUF.
- Ollama only supports one ADAPTER per Modelfile (publish multiple Modelfiles for multiple adapters).
- Inferless / Anyscale / Modal all expose multi-LoRA serving via vLLM under the hood.
## When to Use This Mode
- Distributing a domain fine-tune without re-shipping the base.
- Multi-tenant serving where each customer has their own adapter.
- Hot-swap evaluation across adapters in CI.
- Bundling a LoRA into an Ollama Modelfile for downstream `ollama run`.
- Converting a PEFT artifact to a llama.cpp-compatible GGUF.
## Sources
- [PEFT LoRA reference](https://huggingface.co/docs/peft/en/package_reference/lora)
- [PEFT GitHub](https://github.com/huggingface/peft)
- [vLLM LoRA Adapters docs](https://docs.vllm.ai/en/latest/features/lora/)
- [vLLM dynamic LoRA loading](https://docs.vllm.ai/en/latest/features/lora/#dynamic-loading)
- [llama.cpp convert_lora_to_gguf](https://github.com/ggml-org/llama.cpp/blob/master/convert_lora_to_gguf.py)
- [Ollama Modelfile ADAPTER section](https://docs.ollama.com/modelfile)
- [Replicate Cog docs](https://github.com/replicate/cog)
