Kda And Context Parallel

---
name: kda-and-context-parallel
description: "KDA ops, KimiDeltaAttention/KDAConfig, safe-gate gating, optional
  FlashKDA/TileLang routing, and context-parallel guidance for
  flash-linear-attention."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# KDA and Context Parallel

Use this sub-skill when the task is specifically about Kimi Delta Attention (KDA), KDA layer/model configuration, KDA gate modes, `safe_gate`/`lower_bound`, KDA optional backends, or KDA context-parallel behavior.

## Route here for

- Calling or diagnosing `chunk_kda`, `fused_recurrent_kda`, `KimiDeltaAttention`, `KDAConfig`, `KDAModel`, or `KDAForCausalLM`.
- Choosing between chunk KDA, fused recurrent KDA, KimiDeltaAttention's training/decode modes, and KDA model config fields.
- Handling raw gate inputs versus precomputed log decays, `A_log`, `dt_bias`, `use_qk_l2norm_in_kernel`, `use_gate_in_kernel`, `use_beta_sigmoid_in_kernel`, `allow_neg_eigval`, `safe_gate`, and `lower_bound`.
- Deciding whether a KDA call can use FlashKDA, TileLang KDA backward, Triton-Ascend KDA, distributed `cp_context`, or intra-card context parallel.
- Planning KDA native validation candidates without launching native tests or distributed jobs by default.

## Do not use this sub-skill for

- Generic backend registry mechanics, non-KDA dispatch policy, or broad Triton kernel style; route to the ops/dispatch sub-skill if present.
- Generic benchmark loops, result tables, or autotuning methodology; route to the benchmarking sub-skill if present.
- Non-KDA linear-attention families unless the task is comparing their context-parallel contract with KDA.

## First actions

1. Identify the surface: operator call, KDA layer/model config, optional backend, distributed CP, or intra-card CP.
2. Read the bundled reference that matches the surface:
   - `references/kda-workflows.md` for KDA APIs, gate contracts, layer/model config, backend gates, and native validation candidates.
   - `references/context-parallel.md` for `cp_context`, distributed KDA CP, intra-card CP, prerequisites, and skip rules.
   - `references/troubleshooting.md` for common failures and the fastest diagnosis path.
3. If a quick environment sanity check is useful, run the bundled helper with `--help` first. Use `scripts/smoke_kda.py --require-cuda` only when a CUDA device is intentionally available; it does not launch distributed work.

## Core operating rules

- KDA tensors use q/k heads `H` and value/gate heads `HV`; grouped value attention is valid only when `HV % H == 0`.
- `chunk_kda` is the training-capable high-throughput route and the only KDA operator route that accepts `cp_context`.
- `fused_recurrent_kda` is the recurrent route for decode/small-token inference and does not accept `cp_context`.
- For raw gates, pass `use_gate_in_kernel=True`, `A_log`, optional `dt_bias`, and raw `g`; for precomputed gates, pass `use_gate_in_kernel=False` and ensure `g` is already the log-space decay tensor expected by the operator.
- Prefer the explicit safe-gate contract `use_gate_in_kernel=True`, `safe_gate=True`, `lower_bound=-5.0`; if using precomputed gates with `safe_gate=True`, make the clamp/provenance of `g` explicit because the operator will not apply `lower_bound` in that mode.
- Do not mix `cp_context` with `initial_state` or `output_final_state=True`; `cp_context` owns the rank-local `cu_seqlens` metadata.
- Treat FlashKDA and intra-card CP as inference-only optional accelerators. If their verifier conditions are not all satisfied, expect silent fallback to the default path unless backend dispatch is disabled globally.