Fine Tuning

---
name: fine-tuning
description: "Prepare FlagEmbedding fine-tuning data and safe training commands
  for embedders and rerankers."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Fine-Tuning

Use this sub-skill when the task is to prepare or validate FlagEmbedding fine-tuning data, choose the correct training module for an embedder or reranker, or build a bounded `torchrun` command without launching training automatically.

Read first:

- `references/data-formats.md` for JSONL schemas, hard negatives, teacher scores, `prompt`, and embedder ICL `type` handling.
- `references/training-commands.md` for module entry points, command patterns, DeepSpeed config shapes, LoRA flags, flash-attn caveats, and bge-m3 unified fine-tuning.
- `references/troubleshooting.md` when JSONL validation, optional finetune extras, CUDA, DeepSpeed, flash-attn, cache/token, OOM, or distributed launch issues appear.

Bundled helpers:

- `scripts/validate_train_jsonl.py` validates embedder or reranker JSONL before training. Use `--knowledge-distillation` only when score fields are intended for KD.
- `scripts/split_jsonl_by_text_length.py` splits JSONL by safe character or token-estimate length without downloading tokenizers.

Route away:

- Inference-only model loading, encoding, reranking, and score computation belongs to sibling `inference`.
- Post-training retrieval, reranking, MTEB, BEIR, or custom benchmark work belongs to sibling `evaluation`.
- Broad package installation, import failures, and backend setup outside fine-tuning belongs to the root troubleshooting reference.

Operational constraints:

- Do not run full training unless the user explicitly asks for it and model, data, devices, caches, output directory, and runtime budget are clear.
- Treat hard-negative mining and teacher-score generation as model-loading workflows with network/cache/device dependencies unless every model artifact is already local.
- The optional `FlagEmbedding[finetune]` extra adds DeepSpeed and flash-attn, but full GPU training and flash-attn compatibility were not verified by this sub-skill.