Doca Gpunetio Ib Write Bw

---
license: Apache-2.0
name: doca-gpunetio-ib-write-bw
description: >
  Use this skill when the user is building, running, or interpreting
  the doca/tools/gpunetio_ib_write_bw client+server benchmark — a CUDA
  kernel on the client posts RDMA WRITE work requests through the
  doca-gpunetio device-side surface to measure sustained GPU-driven
  WRITE bandwidth on a GPU+IB-device pair. Trigger even when the user
  does not explicitly mention "doca-gpunetio-ib-write-bw" or
  "GPUNetIO" — typical implicit phrasings include "measure WRITE BW
  when the GPU posts the WRs", "BW swings between runs on the same
  flags", "is the NIC saturated or am I CPU-bound on the CUDA
  kernel", "meson compile fails for the GPUNetIO bw tool",
  "nvidia_peermem isn't picking up my GPU buffer", or "GPU-initiated
  WRITE throughput vs CPU-initiated perftest". Refuse and route
  elsewhere for general doca-gpunetio library work, DOCA install, the
  GPU-initiated WRITE latency analog, the CPU-initiated upstream
  perftest, or application-level end-to-end throughput — those belong
  to other skills.
metadata:
  kind: tool
compatibility: >
  Requires DOCA SDK on Linux with a BlueField DPU or ConnectX NIC,
  NVIDIA GPU, CUDA toolkit and nvcc, loaded `nvidia_peermem`, and an
  InfiniBand RNIC paired with the GPU. Uses `pkg-config` for
  doca-gpunetio, doca-rdma, and doca-common, plus the installed
  gpunetio_ib_write_bw sources. Run only on a trusted, non-shared IB
  fabric during the benchmark window.
---

# DOCA GPUNetIO ib_write_bw

**Where to start:** This is a tool skill for the GPUNetIO-
flavored `ib_write_bw` benchmark shipped under
`doca/tools/gpunetio_ib_write_bw/` (a client + server pair,
built from source against the installed DOCA via `meson`).
It measures sustained RDMA WRITE bandwidth when the WRs are
posted **from a CUDA kernel through the doca-gpunetio
device-side surface**, with the GPU on the data path. Open
[`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure) for the GPU-NIC
pairing precondition and the build pattern; jump to
[`## run`](TASKS.md#run) for the smoke-before-bulk flow.
Open [`CAPABILITIES.md`](CAPABILITIES.md) when the question
is *what this tool actually measures*, *how the result
decomposes (GPU occupancy vs NIC issue rate vs link
saturation)*, or *how the result reads against the GPI
sister tool and the upstream CPU-initiated `perftest`
`ib_write_bw`*. If DOCA is not installed yet, route to
[`doca-setup`](../../doca-setup/SKILL.md) first; if the
user is still deciding between the GPI and GPUNetIO
programming surfaces, the picture in
[`../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes)
and
[`../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes)
is the first stop.

## Example questions this skill answers well

The CLASSES of `doca-gpunetio-ib-write-bw` questions this
skill is built to answer, each with one worked example. The
class is the load-bearing piece; the worked example is one
instance.

- **"What sustained RDMA-WRITE bandwidth can the GPUNetIO
  path deliver on this GPU-NIC pair?"** — worked example:
  *"measure sustained WRITE BW between two hosts with an
  H100 + ConnectX-7 on each side"*. Answered by the
  GPU-NIC pairing precondition in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the bring-up flow in
  [`TASKS.md ## configure`](TASKS.md#configure) +
  [`TASKS.md ## run`](TASKS.md#run). The same shape
  answers *"measure GPUNetIO-driven WRITE BW between a
  host GPU and a BlueField DPU"*.
- **"Where is the bottleneck — GPU compute occupancy, NIC
  issue rate, or link saturation?"** — worked example:
  *"I see 120 Gbit/s on a 200 Gbit/s link; is the NIC
  saturated, am I CPU-bound on the client, or is the CUDA
  kernel not driving enough WRs in flight?"*. Answered by
  the throughput-decomposition rules in
  [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)
  + the eval-loop overlay in
  [`TASKS.md ## test`](TASKS.md#test).
- **"How does the result differ from the classic CPU-
  initiated `perftest` `ib_write_bw`?"** — worked example:
  *"my team has a CPU-initiated WRITE BW number on this
  same NIC; should I expect the GPUNetIO number to match
  or be different?"*. Answered by the *"GPU-initiated
  path adds (or removes) overhead vs the CPU-initiated
  path"* rule in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).
- **"Is the doca-gpunetio path the right surface for my
  sustained-throughput workload class?"** — worked example:
  *"my application streams sensor data from GPU memory at
  line rate to a remote consumer"*. Answered by the
  *"when GPUNetIO is the right surface vs GPI vs CPU-
  initiated"* rule in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the use-side decision in [`TASKS.md ## use`](TASKS.md#use).
- **"My BW number swings between runs. What do I check
  before quoting it?"** — worked example: *"three runs at
  the same flags gave 145, 187, and 160 Gbit/s; is the
  benchmark noisy or is my platform inconsistent?"*.
  Answered by the measurement-soundness rules in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
  layer 5 + the steady-state guidance in
  [`TASKS.md ## test`](TASKS.md#test).
- **"What version of DOCA + CUDA Toolkit do I need for this
  binary to build and run?"** — worked example: *"my
  install has DOCA at one semver and CUDA at another; will
  the ToT-shipped `gpunetio_ib_write_bw` even link?"*.
  Answered by the version overlay in
  [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility)
  which cross-links the canonical detection chain in
  [`doca-version`](../../doca-version/SKILL.md).

## Audience

This skill serves **external developers and performance
engineers who need a reproducible measurement of sustained
RDMA WRITE bandwidth when the WRs are posted from a CUDA
kernel through doca-gpunetio**, on the user's actual install
and GPU-NIC pair. Concretely:

- A developer comparing the GPUNetIO path against the GPI
  path or the host-initiated `perftest`-style path before
  committing an application design to one of them.
- A platform operator validating a tuning change (NUMA
  pinning, GPU PCIe placement, IB device choice, GID
  index, NIC firmware burn) by re-running this benchmark
  against the new state.
- An SRE / performance engineer producing a *"this is the
  GPUNetIO-driven WRITE BW on this GPU-NIC pair today"*
  artifact downstream consumers can cite.
- An AI agent answering *"is the doca-gpunetio path a win
  for my sustained-throughput workload class"* honestly —
  with a measured number, the build + invocation that
  produced it, and the GPU + NIC + DOCA version that
  scopes it — rather than guessing from datasheet
  headlines.

It is **not** for users debugging the `doca-gpunetio`
library itself (route to
[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md)),
and **not** a substitute for the `perftest` upstream
`ib_write_bw` (which measures CPU-initiated WRITE BW).

## Language scope

The `doca-gpunetio-ib-write-bw` tool is shipped as **C plus
a CUDA `.cu` translation unit** under
`doca/tools/gpunetio_ib_write_bw/`, split into a `client/`
subtree and a `server/` subtree. The verified surface (per
`client/{main.c,common.h,common.c,kernel.cu,perftest.c}` and
`server/{main.c,common.h,common.c,perftest.c}`): host-side
build via `meson` against the installed DOCA `pkg-config`
modules (`doca-gpunetio`, `doca-rdma`, `doca-common`); the
device-side build via `nvcc` against the DOCA GPU NetIO
device-side header set; the OOB descriptor exchange via a
TCP socket between client and server. There is no Python /
Rust / Go binding — the tool is a pair of CLI binaries.
The skill's job is to keep the operator-side workflow
language-neutral; the device-side CUDA surface is not
wrappable in another language.

## When to load this skill

Load this skill when the user is — or the agent needs to —
build and run the `gpunetio_ib_write_bw` client + server on
real hosts with DOCA installed plus a CUDA Toolkit matched
to the DOCA install, and a GPU + IB device pair on the
host's PCIe topology. Concretely:

- Measuring sustained kernel-initiated RDMA WRITE
  bandwidth between two hosts (or a host and a BlueField
  DPU) with the GPUNetIO surface.
- Deciding whether the GPUNetIO path is the right runtime
  surface for a class of workload vs the GPI programming
  surface (the [`doca-gpi`](../../libs/doca-gpi/SKILL.md)
  library — `doca/tools/` ships no GPI benchmark binary) or
  the classic CPU-initiated `perftest` path.
- Capturing a documented baseline (build + invocation +
  DOCA version + GPU + NIC + as-deployed environment +
  numbers) for later regression hunts.
- Diagnosing a build / link / run failure that surfaces
  the GPUNetIO + RDMA bring-up sequence under this tool's
  shipped scaffolding.

Do **not** load this skill for general DOCA orientation,
library API work, or installation. For those, use
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),
[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md),
or [`doca-setup`](../../doca-setup/SKILL.md). Do not load
it for *application-level* end-to-end throughput either —
this benchmark measures the WR-submission path through
GPUNetIO, not the user's full pipeline.

## What this skill provides

This is a **thin loader**. Substantive material lives in
two companion files:

- `CAPABILITIES.md` — what the tool measures (the
  sustained-WRITE-BW primitive driven by a client-side
  CUDA kernel through doca-gpunetio), the
  runtime-surface selection rule (GPUNetIO vs GPI vs
  CPU-initiated), the GPU-NIC pairing precondition, the
  throughput-decomposition guide (GPU compute occupancy
  vs NIC issue rate vs link saturation), the version
  overlay (DOCA `.pc` PLUS CUDA Toolkit), the layered
  error taxonomy (config-syntax / build-time / GPU-NIC-
  pairing / GPUNetIO-lifecycle / RDMA-connection /
  measurement-soundness / version / cross-cutting), the
  observability surface (stdout report, DOCA log levels,
  OOB-socket exchange), and the safety overlay (the
  *"GPU-side handle is a credential"* rule from
  doca-gpunetio; the cross-cutting hardware-safety
  meta-policy).
- `TASKS.md` — step-by-step workflows for the in-scope
  task verbs: `install` (preconditions — DOCA install,
  CUDA Toolkit, GPU + NIC pair, OOB connectivity),
  `configure` (build-tree under
  `doca/tools/gpunetio_ib_write_bw/` and the `meson`
  build wrapping the shipped DOCA), `build` (the
  `meson setup` + `meson compile` pattern from the
  public DOCA build documentation), `modify` (do not
  patch the shipped tool source; modify the invocation
  and the surrounding environment instead), `run` (smoke-
  before-bulk; client + server bring-up order; reading
  the per-iteration report), `test` (the eval loop —
  steady-state, NUMA placement, NIC saturation cross-
  check), `debug` (walk the error taxonomy layer by
  layer), `use` (how a BW result feeds a class-of-
  workload decision), plus a `Deferred task verbs`
  block routing out-of-scope questions.

The skill assumes a host where DOCA is already installed,
a CUDA Toolkit matched to the install is present, and the
operator has whatever privileges the public install profile
expects for binding a `doca_dev`, a `doca_gpu`, and an OOB
TCP socket.

## What this skill deliberately does not ship

This skill is **agent guidance**, not a samples or scripts
bundle. To keep the boundary clean, it deliberately does
not contain — and pull requests should not add:

- **Specific flag strings or expected throughput numbers**
  beyond what the tool's shipped `--help` and `main.c` ARGP
  registration establish. The flag surface is small
  (device name, GPU PCIe address, GID index, server IP on
  the client side); the agent re-reads the binary's
  `--help` on the installed version before quoting flag
  strings. Throughput numbers are device-, firmware-,
  version-, and topology-specific.
- **Pre-written DOCA GPUNetIO or CUDA kernel source code**
  that would compete with the shipped tool tree. The
  shipped `client/{main.c,kernel.cu,perftest.c,common.{c,h}}`
  and `server/{main.c,perftest.c,common.{c,h}}` files are
  the verified worked example; the agent's job is to
  route the user there and prescribe minimum-diff
  modification per the universal modify-a-sample workflow
  in
  [`doca-programming-guide`](../../doca-programming-guide/SKILL.md).
- **Wrappers, parsers, or scripts** in any language that
  consume the tool's stdout. The output format is small
  and documented in
  [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability);
  if the user wants to script against it, the right
  answer is *"read the live source, write the parser
  against your installed binary"*.
- **A `samples/`, `bindings/`, or `reference/` subtree.**
  This is a thin loader for a shipped tool tree;
  substantive material lives in the source tree and in
  the GPUNetIO library docs.

## Loading order

1. Read this `SKILL.md` first to confirm the user's
   question is in scope (the user actually wants to
   measure sustained kernel-initiated WRITE BW through
   GPUNetIO, not learn GPUNetIO as a library or do a
   CPU-initiated measurement).
2. **For what the tool measures, the surface-selection
   rule against the GPI sister tool and the CPU-initiated
   `perftest`, the throughput-decomposition guide, the
   version overlay, the error taxonomy, the observability
   surface, and the safety overlay, see
   [CAPABILITIES.md](CAPABILITIES.md).**
3. **For step-by-step workflows — `install`, `configure`,
   `build`, `modify`, `run`, `test`, `debug`, `use` — see
   [TASKS.md](TASKS.md).**

## Related skills

- [`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md) —
  the library this tool wraps. The per-GPU `doca_gpu`
  context, the GPU-visible `doca_gpu_eth_*` and RDMA-side
  handles, the CUDA-side persistent-kernel pattern, the
  dual capability-discovery rule (DOCA cap-query AND
  `cudaGetDeviceProperties`), and the env preconditions
  (`nvidia_peermem` loaded, CUDA buffers registered with
  DOCA) live there.
- [`../../libs/doca-rdma/SKILL.md`](../../libs/doca-rdma/SKILL.md) —
  the underlying RDMA library. The RDMA queue this tool
  binds is created and connected via `doca-rdma`; the
  queue lifecycle, transport type (RC vs UC vs UD),
  permission matrix, and connection method are owned
  there.
- [`../../libs/doca-verbs/SKILL.md`](../../libs/doca-verbs/SKILL.md) —
  the raw-verbs escape hatch beneath `doca-rdma` /
  `doca-gpunetio`. This tool stays on the higher-level
  surfaces; `doca-verbs` is the right place only if the
  user needs a specific WR flag / QP attribute the
  GPUNetIO + RDMA surfaces do not expose.
- [`../doca-gpunetio-ib-write-lat/SKILL.md`](../doca-gpunetio-ib-write-lat/SKILL.md) —
  the latency analog of this tool. Same physical
  operation; same runtime framework; different metric
  class (BW vs latency). The two together carry the
  full GPUNetIO-side throughput / latency picture.
- [`doca-gpi`](../../libs/doca-gpi/SKILL.md) — the GPI
  programming surface (CUDA-kernel-initiated RDMA), the
  alternative runtime framework for the same physical
  operation. `doca/tools/` ships no GPI `ib_write_lat` /
  `ib_write_bw` benchmark binary, so the GPI comparison is
  against the library surface, not a sibling tool. The
  selection rule in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  is the decision aid.
- [`doca-version`](../../doca-version/SKILL.md) — the
  canonical version-detection chain, four-way match rule,
  NGC container semantics, and headers-win-over-docs
  rule. The `## Version compatibility` section in this
  skill is a thin overlay; the body lives there.
- [`doca-setup`](../../doca-setup/SKILL.md) — env
  preparation, install verification, GPU + CUDA Toolkit
  pairing, `nvidia_peermem` load, hugepages, NUMA, and
  the *I have no install yet* path with the public NGC
  DOCA container.
- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md) —
  routing to the public DOCA documentation set (DOCA GPU
  NetIO, DOCA RDMA pages on `docs.nvidia.com`) and the
  `docs.nvidia.com/cuda/` pointer for the CUDA Toolkit.
- [`doca-debug`](../../doca-debug/SKILL.md) — the
  cross-cutting debug ladder. The tool surfaces its own
  error taxonomy; when the cause is below DOCA, the
  taxonomy hands off here.
- [`doca-hardware-safety`](../../doca-hardware-safety/SKILL.md) —
  the bundle-wide hardware-safety meta-policy. The
  `## Safety policy` overlay cross-links it.