Model Runtime

---
name: model-runtime
description: "Route inference-models users through AutoModel, backend selection,
  model/package negotiation, local packages, environment inspection, and runtime
  troubleshooting."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# model-runtime

Use this sub-skill when the user needs to inspect, install, select, or debug `inference-models` at runtime.

## Route here when

- Choosing a backend, quantization, batch size, or device for `AutoModel.from_pretrained(...)`.
- Inspecting available packages with `AutoModel.describe_model(...)` or `AutoModel.describe_model_package(...)`.
- Checking hardware and dependency availability with `AutoModel.describe_compute_environment()`.
- Loading a Roboflow model, a cached package, a local package directory, or a local checkpoint.
- Debugging backend extras, offline mode, cache reuse, package negotiation, or runtime load failures.
- Understanding when a package is rejected because of trust, device, batch size, quantization, or model-feature constraints.

## Do not handle here

- CLI command surfaces, server lifecycle, or workflow CLI commands.
- WebRTC streaming or any task whose primary surface is a model-id stream.
- Training, export, or repository-maintenance work.

## Read order

1. Read `references/api-reference.md` for the public API and load flow.
2. Read `references/backends.md` for installation and backend-selection guidance.
3. Read `references/troubleshooting.md` when the user reports a failure.
4. Use `scripts/describe_compute_environment.py` for a deterministic environment probe when the runtime is unclear.

## Operating rules

- Start with `AutoModel.describe_compute_environment()` or the bundled probe whenever the user asks what this machine can run.
- Prefer `AutoModel.describe_model(model_id)` before forcing `backend`, `quantization`, `batch_size`, or `model_package_id`.
- Prefer `AutoModel.describe_model_package(model_id, package_id)` before forcing a specific package or diagnosing a mismatch.
- Treat `AutoModel.from_pretrained(...)` as the primary entry point.
- Use `weights_provider="local"` only for local directories or checkpoint files.
- Treat a path-like `model_id_or_path` as local storage only when the user actually intends a local load.
- Keep `allow_local_code_packages=True` only for trusted custom packages that ship `model_module` and `model_class`.
- Keep `allow_untrusted_packages=False` unless the user explicitly accepts the trust trade-off.
- For checkpoint files, require `model_type` and a supported `task_type` / `backend` combination.
- Keep `allow_loading_dependency_models=True` unless the user intentionally wants to block nested dependency loads.
- When negotiation fails, report the rejected constraints, installed extras, and runtime facts rather than guessing.

## Current selection stance

- Current package ranking priority in the code path is: `trt` > `onnx` > `torch` > `hugging-face` > `torch-script` > `ultralytics` > `custom`.
- Ranking still depends on trust, batch size, quantization, runtime/device compatibility, model features, and package metadata.
- If a user forces a backend, the loader still filters for compatibility before load.

## What this sub-skill should answer

- What backend or extra should I install for this model on this machine?
- Why did `AutoModel` choose a different package than I expected?
- How do I load a local package, a cached package, or a direct checkpoint safely?
- Which env vars or cache settings are relevant to this runtime failure?