Hf Inference

---
name: hf-inference
description: "Routes Chinese-LLaMA-Alpaca-2 local generation, chat, and
  speculative-sampling workflows."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# hf-inference

Use this sub-skill for direct Transformers-based inference, interactive chat, speculative sampling, and prompt-template handling.

## Use it when

- the user mentions `inference_hf.py`, `gradio_demo.py`, `speculative_sample.py`, or the attention patch helpers
- the task is about `--interactive`, `--with_prompt`, `--negative_prompt`, `--guidance_scale`, or `--speculative_sampling`
- the task needs CPU-only inference, 4-bit/8-bit loading, NTK scaling, or FlashAttention acceleration
- the user wants a chat UI rather than an HTTP server

## Workflow

1. Read `references/workflows.md` for the supported CLI flag combinations.
2. Check `assets/prompts/` and `assets/tokenizer/` if the task is about prompt wrapping or tokenizer compatibility.
3. Use `scripts/inference/inference_hf.py` for single-turn or file-based generation.
4. Use `scripts/inference/gradio_demo.py` for multi-turn chat or a browser UI.
5. Read `references/troubleshooting.md` if a run fails on tokenizer paths, draft models, optional acceleration helpers, or CPU/GPU constraints.

## Bundled runtime files

- `scripts/inference/inference_hf.py`
- `scripts/inference/gradio_demo.py`
- `scripts/inference/speculative_sample.py`
- `scripts/inference/flash_attn_patch_for_inference.py`
- `scripts/attn_and_long_ctx_patches.py`
- root asset `../../assets/prompts/alpaca-2.txt`
- root asset `../../assets/prompts/alpaca-2-long.txt`
- root asset `../../assets/tokenizer/`

## What to read first

- `references/workflows.md` for the supported CLI modes and prompt behavior
- `references/troubleshooting.md` for optional acceleration and model-loading failures
- `../../references/prompt-and-tokenizer.md` when the task is specifically about prompt text or tokenizer files

## Routing notes

- Use this sub-skill for local generation and chat UX.
- Use the API-serving sub-skill when the task becomes an HTTP server instead of a direct generation CLI.
- The optional vLLM code path inside the inference CLI is still an inference concern, but the dedicated server sub-skill is usually the better choice when the user wants deployment rather than local generation.