Deployment And Distributed

---
name: deployment-and-distributed
description: "Use this sub-skill for Torch-TensorRT deployment artifacts, Triton
  serving, C++ or AOTInductor execution, ExecuTorch export, TensorRT-RTX
  platforms, DLA/Jetson, Windows cross-compile, and distributed inference."
metadata:
  disco-role: operating
disable-model-invocation: true
license: BSD 3-Clause
---

# Torch-TensorRT Deployment and Distributed Inference

Use this sub-skill when the question is not just "how do I compile?" but "where and how will the compiled result run?"

## Choose the deployment target first

| Target | Typical artifact/API | Read |
| --- | --- | --- |
| Python deployment | `.ep`, `torch.export.load(...).module()`, `torch_tensorrt.load(...)` | `references/artifact-deployment-matrix.md` |
| C++/libtorch | `.ts`, `torch::jit::load`, Torch-TensorRT runtime libraries | `references/cpp-aoti-executorch.md` |
| AOTInductor | `.pt2`, `torch._inductor.aoti_load_package`, C++ AOTI loader | `references/cpp-aoti-executorch.md` |
| Triton serving | model repository, `config.pbtxt`, client requests | `references/triton-and-serving.md`; `scripts/generate_triton_config.py --help` |
| Raw TensorRT runtime | `.engine` bytes | `../compilation-and-export/references/serialization-and-engines.md` |
| ExecuTorch | `.pte` with ExecuTorch runtime | `references/cpp-aoti-executorch.md` |
| Distributed inference | `torchtrtrun`, `distributed_context`, NCCL/TRT collectives | `references/distributed-inference.md`; `scripts/torchtrtrun_env_probe.py --help` |
| Platform-specific deployment | TensorRT-RTX, DLA/Jetson, Windows x86-64 cross-compile, Windows ARM64 build | `references/platforms-and-rtx.md` |

## Deployment workflow

1. Identify the target process: Python, C++, Triton server, raw TensorRT runtime, ExecuTorch, Windows, Jetson/DLA, or multi-GPU/distributed.
2. Verify the needed feature gates and runtime libraries on both build and target machines.
3. Compile with the right artifact in mind. Some choices must be made before save/export, such as `output_format`, `hardware_compatible`, dynamic shapes, and runtime support.
4. Save the artifact and test load/execute in the closest available target runtime.
5. Package supporting files: Triton model repository, C++ library path, AOTI loader, NCCL environment, runtime cache directories, or Windows/Jetson prerequisites.

## Common deployment decisions

- Use `.ep` for Python deployment where PyTorch remains available.
- Use `.ts` only when the Torch-TensorRT C++ runtime is available and the target uses libtorch/TorchScript.
- Use `.pt2` AOTInductor on Linux when the user wants a package runnable without importing Torch-TensorRT at inference time.
- Use raw `.engine` when the entire graph must be TensorRT-only and no PyTorch fallback is allowed.
- Use TensorRT-RTX for RTX desktop/laptop/workstation targets and RTX-specific runtime JIT behavior.
- Use DLA only on supported embedded platforms and only with FP16/INT8.
- Use `torchtrtrun` and `distributed_context` for distributed inference with TensorRT collectives/NCCL concerns.

## Guardrails

- Do not tell users to copy files from the original source checkout. Use the recipes and bundled scripts here.
- Do not run Triton servers, multi-rank jobs, Docker, network clients, or C++ builds unless the user explicitly asks and the environment is prepared.
- Do not promise a Python-only Torch-TensorRT wheel can support C++ runtime or TorchScript artifact execution.
- Do not treat a model artifact as portable across TensorRT versions, GPU compute capabilities, CUDA versions, or package flavors unless `hardware_compatible`/`version_compatible` and target testing support that claim.