Inference And Models

---
name: inference-and-models
description: "Marqo model registry, inference request shaping, preprocessing,
  model backends, cache behavior, and Triton/model-management troubleshooting."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# inference-and-models

Use this sub-skill for Marqo model-registry facts, inference request shaping, preprocessing, model backends, cache behavior, and Triton/model-management troubleshooting.

## Route here when the task involves
- translating a registry model name into runtime `modelProperties`
- deciding between random, HF, and OpenCLIP pipelines
- building or repairing `/vectorise` requests
- inspecting or clearing loaded models
- probing CUDA, Triton, or download/cache readiness
- diagnosing model load, unload, auth, or preprocessing failures

## Route away when the task is about
- search query construction or ranking fusion
- index schema or Vespa layout
- service startup, compose, or container orchestration

## Bundled references
- `references/model-registry.md`
- `references/inference-api.md`
- `references/model-services.md`
- `references/troubleshooting.md`
- `scripts/check_model_backends.py`

## Working rules
1. Prefer `random/*` for no-download smoke checks and shape debugging.
2. For direct `/vectorise`, always send a complete `embeddingModelConfig.modelProperties` object; the direct inference route does not resolve registry names for you.
3. Keep `/vectorise` requests and responses on `application/msgpack`.
4. Use the bundled backend probe before touching live Triton or model-download paths.
5. When a model is already loaded, inspect the full cache key returned by `/models` before trying `DELETE /models`.
6. Never trigger model downloads or Triton calls just to inspect support; use the bundled script and the `random` family first.

## Fast selection guide
| Need | Start with |
| --- | --- |
| No-download smoke test | `random/small` or `random/medium` |
| Text embeddings | `hf/*` |
| Text + image embeddings | `open_clip/*` or `hf-hub:*` |
| Load/unload or Triton errors | `references/model-services.md` |
| Bad request or preprocessing validation | `references/inference-api.md` + `references/troubleshooting.md` |
| Backend/cuda sanity check | `scripts/check_model_backends.py` |

## Typical flow
1. Identify the model family and whether the request is meant for the direct inference service or the Marqo API client.
2. Use `references/model-registry.md` to obtain the correct property shape and any family-specific fields.
3. Use `references/inference-api.md` to build the request body and expected response shape.
4. Use `references/model-services.md` for load/unload, Triton, and cache-key behavior.
5. Use `references/troubleshooting.md` and the probe script when a backend or auth dependency fails.

## Output expectations
- This sub-skill should help produce corrected request payloads, cache-key-aware diagnostics, and backend triage steps.
- It should not broaden into search or index advice.
- It should not assume external downloads are safe unless the user explicitly wants that path.