Doca Gpi

---
license: Apache-2.0
name: doca-gpi
description: >
  Use this skill for hands-on DOCA GPI programming
  — wiring a GPU-Packet-Initiator context so a CUDA kernel drives
  RDMA queues directly from GPU memory without host CPU mediation.
  Covers picking GPI vs doca-gpunetio, the doca_gpi / domain /
  channel object model, the GPU-side handle handoff
  (doca_gpu_gpi_channel*), attaching GPU memory to a GPI domain,
  the domain and channel attribute objects, and debugging
  DOCA_ERROR_* from doca_gpi_* calls. Trigger even when the user
  does not explicitly mention "DOCA GPI" — implicit
  phrasings include "my CUDA kernel needs to post RDMA directly
  from GPU memory", "DOCA_ERROR_* from doca_gpi_gpu_channel_get",
  "how do I hand a GPU handle to my CUDA kernel", "how many
  channels can a GPI domain hold", or "GPU kernel driving RDMA
  without the host CPU on the path". Refuse and route elsewhere
  for the doca-gpunetio
  Send/Receive surface, the doca-rdma queue lifecycle, DPA-side
  initiation (doca-rdmi), or the CUDA programming model — those
  belong to other skills.
metadata:
  kind: library
compatibility: >
  Requires DOCA SDK installed at /opt/mellanox/doca on Linux (Ubuntu
  22.04/24.04 or RHEL/SLES) with a BlueField DPU or ConnectX NIC
  attached, plus an NVIDIA GPU with CUDA Toolkit installed
  (GPUDirect-style PCIe path between GPU and NIC). Reads the local
  install via `pkg-config doca-gpi` (co-requires doca-gpunetio,
  doca-dpa, doca-verbs) and inspects /opt/mellanox/doca/{lib,include,samples,applications}.
---

# DOCA GPI

**Where to start:** This skill assumes DOCA is already installed
and the user is doing **hands-on GPI work** on a host that has both
a BlueField / ConnectX device and an NVIDIA GPU reachable over
PCIe. Open [`TASKS.md`](TASKS.md) if the user wants to *do*
something (install / configure / build / modify / run / test /
debug / use); open [`CAPABILITIES.md`](CAPABILITIES.md) when the
question is *what can GPI express on this version* — the domain +
channel object model, the GPU-side handle handoff, the relationship
to doca-gpunetio and doca-verbs, the attribute objects, and the
safety overlay. If the user has not installed DOCA yet, route to
[`doca-setup`](../../doca-setup/SKILL.md) first.

## Example questions this skill answers well

The CLASSES of GPI questions this skill is built to answer, each
with one worked example. The agent should treat the *class* as the
load-bearing piece — the worked example is a single instance.

- **"Should I use `doca-gpi` or `doca-gpunetio` for this case?"** —
  worked example: *"my CUDA kernel needs to post RDMA writes
  directly to a remote DPU's memory — do I want the higher-level
  Send/Receive surface or the lower-level channel/queue surface?"*.
  Answered by the *channel-level vs Send/Receive-level* selection
  rule in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  surface-selection table.
- **"How do I bring up a GPI channel and connect it to a remote
  peer?"** — worked example: *"create the GPI, set domain + channel
  attribute sizing, create the channel, exchange endpoint
  connection info with the remote, connect the endpoint"*. Answered
  by the channel-object lifecycle in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the configure walk in
  [`TASKS.md ## configure`](TASKS.md#configure).
- **"What is the GPU-side handle and how do I hand it to my CUDA
  kernel?"** — worked example: *"`doca_gpi_gpu_channel_get`
  returns a `doca_gpu_gpi_channel*` — how do I get that into my
  CUDA kernel's argument list?"*. Answered by the GPU-handoff
  pattern in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the run-side wiring in [`TASKS.md ## run`](TASKS.md#run),
  cross-linked into
  [`doca-gpunetio`](../doca-gpunetio/SKILL.md) for the CUDA-side
  programming surface itself.
- **"What does my CUDA + GPU + DOCA version stack need to look
  like?"** — worked example: *"I have BlueField-3 + A100; which
  CUDA Toolkit and which DOCA version do I need?"*. Answered by
  the version-overlay in
  [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility)
  + the install-checks in [`TASKS.md ## install`](TASKS.md#install).
- **"How do I size the channels and work queues I want?"** —
  worked example: *"I want 64 channels in a domain, each with a
  1024-entry send queue; which setters express that?"*. Answered
  by the attribute-object sizing rule
  (`doca_gpi_domain_attr_set_num_channels`,
  `doca_gpi_channel_attr_set_sq_wqe_num`) in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the sizing step in
  [`TASKS.md ## configure`](TASKS.md#configure).
- **"What does this `DOCA_ERROR_*` from a `doca_gpi_*` call
  mean?"** — worked example: *"`DOCA_ERROR_*` from
  `doca_gpi_gpu_channel_get`"*. Answered by the GPI overlay
  on the cross-library taxonomy in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
  + the layered ladder in [`TASKS.md ## debug`](TASKS.md#debug)
  that escalates to [`doca-debug`](../../doca-debug/SKILL.md).

## Audience

This skill serves **external developers building GPU-resident DOCA
applications that need to drive RDMA queues directly from CUDA
kernels** — i.e., users whose accelerator-side code wants to post
RDMA work from GPU memory without round-tripping through the host
CPU. The canonical caller is a CUDA kernel that runs on an NVIDIA
GPU on the same host as a BlueField / ConnectX device, has
GPUDirect-style access to the DPU's RDMA queues through the
DOCA GPU-NetIO stack, and uses the GPI channel + queue handle to
drive RDMA initiation. This skill is *not* for NVIDIA developers
contributing to DOCA GPI itself, and it is not the right surface
for the higher-level Send/Receive Ethernet-shaped GPU NetIO API —
that belongs to [`doca-gpunetio`](../doca-gpunetio/SKILL.md).

## Language scope

DOCA GPI ships as a C library with the `pkg-config` module name
`doca-gpi`. The library's **host-side** surface (`doca_gpi_*`)
is C; the **GPU-side** surface — the `doca_gpu_gpi_channel*`
handle and the device-side calls a CUDA kernel uses against that
handle — is compiled with `nvcc` against the DOCA GPU NetIO
device-side header set documented in
[`doca-gpunetio`](../doca-gpunetio/SKILL.md). Other-language
consumers (Rust, Go, Python, …) consume the host-side `*.so`
through FFI; the skill's contribution in that case is to keep the
channel / queue lifecycle, the GPU-handle handoff, the version
discipline, and the safety overlay language-neutral, and to route
the agent to the public C ABI as the authoritative surface that
any wrapper will eventually call. The GPU-side surface is *not*
wrappable in another language — it is compiled and linked into
the CUDA binary itself.

## When to load this skill

Load this skill when the user is doing **hands-on DOCA GPI work**
on a host with both a BlueField / ConnectX device and an NVIDIA
GPU. Concretely:

- Deciding between `doca-gpi` (the lower-level channel/queue
  surface) and `doca-gpunetio` (the higher-level Send/Receive
  surface) for a new GPU-initiated RDMA workload.
- Creating a `doca_gpi` on a `doca_dev`, configuring it via the
  `doca_gpi_set_*` family (domain count, GID index, port) and
  sizing domains and channels through the
  `doca_gpi_domain_attr_*` / `doca_gpi_channel_attr_*` setters
  before `doca_gpi_start()`.
- Creating a channel with `doca_gpi_channel_create` and
  retrieving its GPU-side handle with `doca_gpi_gpu_channel_get`,
  then handing the GPU-side handle to a CUDA kernel.
- Exchanging endpoint connection info with a remote peer using
  `doca_gpi_channel_ep_conn_info_create` /
  `doca_gpi_channel_ep_connect` to establish the GPU-driven
  channel end-to-end.
- Attaching memory regions to a GPI domain with
  `doca_gpi_domain_attach_local_mmap` /
  `doca_gpi_domain_attach_remote_mmap` (each backed by a
  `doca_mmap` the application created).
- Debugging a `DOCA_ERROR_*` returned by a `doca_gpi_*` call
  and deciding whether the cause is a lifecycle ordering bug, a
  GPU datapath mis-assignment, a CUDA-version mismatch, or a
  layer below DOCA.

Do **not** load this skill for general DOCA orientation, install
of DOCA itself, host-CPU-initiated RDMA, or the higher-level GPU
NetIO Send/Receive Ethernet-shaped API. For those, use
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),
[`doca-setup`](../../doca-setup/SKILL.md),
[`doca-rdma`](../doca-rdma/SKILL.md), and
[`doca-gpunetio`](../doca-gpunetio/SKILL.md) respectively.
When one question spans the GPI channel lifecycle and CUDA-side
GPU NetIO behavior, load both skills: this skill owns GPI object
creation, connection, and teardown, while `doca-gpunetio` owns
kernel launch and device-side execution. If that boundary remains
ambiguous after reading both scopes, stop and ask which side is
failing instead of choosing one implicitly.

## What this skill provides

This is a **thin loader**. The body keeps only the orientation
needed to pick the right next file. The substantive GPI-specific
material lives in two companion files:

- `CAPABILITIES.md` — what GPI can express on this version: the
  `doca_gpi` / `doca_gpi_domain` / `doca_gpi_channel` object
  model, the GPU-side handle handoff, the relationship to
  [`doca-gpunetio`](../doca-gpunetio/SKILL.md) (which owns the
  GPU-side `doca_gpu_gpi_channel*` device surface) and to
  [`doca-verbs`](../doca-verbs/SKILL.md) and
  [`doca-dpa`](../doca-dpa/SKILL.md) (the transport and DPA layers
  GPI builds on), the domain and channel attribute objects, the
  maturity statement (every `doca_gpi_*` symbol is
  `DOCA_EXPERIMENTAL`), the GPI overlay on the cross-library
  `DOCA_ERROR_*` taxonomy, the observability surface (CUDA-side
  channel polling, mmap-attach exchange), and the safety policy
  that gates GPU-side RDMA initiation.
- `TASKS.md` — step-by-step workflows for the eight in-scope
  verbs: `install`, `configure`, `build`, `modify`, `run`,
  `test`, `debug`, `use`. Plus a `Deferred task verbs` block that
  points out-of-scope questions at the right next skill.

The skill assumes a host where DOCA is already installed at the
standard location, a CUDA Toolkit compatible with the installed
DOCA is present, and the user has the privileges their public
install profile expects. It does not cover installing DOCA — that
path goes through [`doca-setup`](../../doca-setup/SKILL.md).

## What this skill deliberately does not ship

This skill is **agent guidance**, not a samples or templates
bundle. To keep the boundary clean, it deliberately does not
contain — and pull requests should not add:

- **Pre-written DOCA GPI application source code, in any
  language.** The agent's job is to route the user to verified
  reference code (the shipped DOCA GPU-NetIO samples on the
  installed package set are the canonical worked examples for the
  GPU-side handoff) and to prescribe a minimum-diff modification
  via the universal modify-a-sample workflow in
  [`doca-programming-guide`](../../doca-programming-guide/SKILL.md).
  Because every GPI symbol is tagged `DOCA_EXPERIMENTAL`
  in the public header, the skill refuses to author GPI
  source from documentation prose.
- **Standalone build manifests** (`meson.build`, `CMakeLists.txt`,
  `Cargo.toml`, …) parked inside the skill. The agent constructs
  the build manifest *in the user's project directory* against
  the user's installed DOCA, where `pkg-config --modversion
  doca-gpi` is the source of truth.
- **CUDA kernel templates.** The CUDA-side surface is owned by
  [`doca-gpunetio`](../doca-gpunetio/SKILL.md); GPI's GPU-side
  handle is *consumed by* the CUDA programming model documented
  there. This skill names the GPI-specific handoff (the
  `doca_gpu_gpi_channel*` type, the channel-connect call) but
  does not author CUDA kernels.
- **A `samples/`, `bindings/`, or `reference/` subtree** of any
  kind. A mock or incomplete artifact in this skill's tree, even
  one labeled "reference", is misleading: users will read it as
  buildable.

## Loading order

1. Read this `SKILL.md` first to confirm the user's question is
   in scope.
2. **For the GPI object model, the GPU-side handoff pattern, the
   domain and channel attribute objects, the version compatibility
   rule, the error taxonomy, observability, and safety policy, see
   [CAPABILITIES.md](CAPABILITIES.md).**
3. **For step-by-step workflows — install, configure, build,
   modify, run, test, debug, use — see [TASKS.md](TASKS.md).**

Both companion files cross-link to each other and to
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)
whenever the right answer is "look it up in the public docs or
the installed package layout" rather than "GPI-specific
guidance".

## Related skills

- [`doca-gpunetio`](../doca-gpunetio/SKILL.md) — the GPU NetIO
  library that exposes Send/Receive-shaped Ethernet I/O for CUDA
  kernels and **owns the GPU-side device surface GPI hands off
  to**: the `doca_gpu_gpi_channel*` type and its device-side
  `.cuh` API live in doca-gpunetio, and `doca_gpi.h` includes
  `doca_gpunetio.h`. Both can coexist in the same application.
  The selection table in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  is the load-bearing decision aid.
- [`doca-verbs`](../doca-verbs/SKILL.md) and
  [`doca-dpa`](../doca-dpa/SKILL.md) — the transport and DPA
  layers GPI builds on. `dependencies/meson.build` lists
  `doca-dpa`, `doca-gpunetio`, and `doca-verbs` (plus the
  `libmlx5` / `libibverbs` externals); `doca_gpi_get_dpa` returns
  the GPI-owned `doca_dpa*` for tuning DPA attributes. The RDMA
  transport type, GID / port selection, and permission semantics
  live at the verbs layer; GPI consumes it rather than binding a
  `doca-rdma` queue.
- [`doca-rdmi`](../doca-rdmi/SKILL.md) — the sister DPA-side
  initiator surface. Both GPI and RDMI exist for "drive RDMA
  initiation from an accelerator without the host CPU on the
  data path"; GPI is the GPU case, RDMI is the DPA case.
- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md) — the
  routing table for every public DOCA documentation source and
  the on-disk layout of an installed DOCA package.
- [`doca-setup`](../../doca-setup/SKILL.md) — env preparation,
  install verification, and the *I have no install yet* path
  with the public NGC DOCA container.
- [`doca-programming-guide`](../../doca-programming-guide/SKILL.md) —
  general DOCA programming patterns shared by every library: the
  canonical `pkg-config` + meson build pattern, the universal
  modify-a-shipped-sample first-app workflow, the universal
  Core-context lifecycle, the cross-library `DOCA_ERROR_*`
  taxonomy. This skill layers GPI specifics on top.
- [`doca-debug`](../../doca-debug/SKILL.md) — the cross-cutting
  debug ladder (install / version / build / link / runtime /
  program / driver). GPI-specific debug overlays on top of it.
- [`doca-hardware-safety`](../../doca-hardware-safety/SKILL.md) —
  the bundle-wide hardware-safety meta-policy. The `## Safety
  policy` overlay in `CAPABILITIES.md` cross-links it.
- [`doca-version`](../../doca-version/SKILL.md) — the version
  detection / four-way match rule every per-artifact `##
  Version compatibility` anchor builds on. This skill quotes
  the GPI-specific overlay only (DOCA-side `.pc` PLUS the CUDA
  Toolkit axis).
- [`doca-structured-tools-contract`](../../doca-structured-tools-contract/SKILL.md) —
  the JSON-schema contracts for the agent-preferred structured
  helpers; the `## Command appendix` in `TASKS.md` defers to
  them before falling back to the manual chain.