Inference Serving Topology

---
name: inference-serving-topology
description: |
  LLM/model inference serving architecture: the engine → serving → orchestration
  layering (vLLM/SGLang/TensorRT-LLM, Triton, KServe/Ray Serve), KV-cache &
  continuous batching, prefill-decode disaggregation, and scaling. Architect-level
  topology, not model training.

  USE WHEN: designing model/LLM serving infra, "vLLM", "SGLang", "TensorRT-LLM",
  "Triton", "KServe", "Ray Serve", "continuous batching", "KV cache", "prefill
  decode", "TTFT", multi-GPU/multi-model serving, inference autoscaling.

  DO NOT USE FOR: on-device (use `edge-inference`); provider routing (use
  `model-gateway-routing`); RAG app logic (use rag/rag-frameworks skills).
allowed-tools: Read, Grep, Glob
---
# Inference Serving Topology

## The three layers (name which you're designing)
1. **Engine** — executes the model on accelerators: **vLLM**, **SGLang**,
   **TensorRT-LLM**. Owns paged **KV-cache**, **continuous (in-flight) batching**,
   quantization. This is where throughput/latency is won.
2. **Serving** — request routing, batching policy, API contract, metrics, rate
   limiting: **Triton** (production shell around an engine), **KServe**,
   **LiteLLM/Envoy AI Gateway**.
3. **Orchestration** — scaling, health, placement: **Kubernetes + KEDA**,
   **Ray Serve**, llm-d, GKE Inference Gateway.

A common 2026 pairing: **vLLM as the token engine + Triton as the production
shell**; **Ray Serve** when you need multi-GPU/multi-node distributed strategies.

## Levers that decide the topology
- **KV-cache** is the memory bottleneck for LLMs → paged KV (vLLM), cache reuse,
  quantized KV. Drives max batch / context.
- **Continuous batching** (vs static) is mandatory for throughput.
- **Prefill–decode disaggregation**: split the compute-bound prefill from the
  memory-bound decode onto different pools → better utilization at scale.
- **Parallelism**: tensor / pipeline / expert (MoE) / data-parallel attention —
  chosen by model size vs GPU memory.
- **Targets**: state TTFT (time-to-first-token, low hundreds of ms) and
  inter-token latency (tens of ms) goals — they drive batching/parallelism.

## Scale ladder
Single GPU + vLLM → multi-GPU one node → Ray Serve/KServe multi-node + KEDA
autoscaling → disaggregated prefill/decode + multi-region. Don't jump tiers
without a load/latency reason.

## When to recommend what
- One model, moderate load → single-node vLLM (+ Triton for ops).
- Many models / platform team → Triton + KServe on k8s.
- Large model / high concurrency / SLO-driven → Ray Serve, disaggregation,
  tensor/expert parallelism.