Kaggle Gpu

---
name: kaggle-gpu
description: Training on Kaggle's free GPUs and TPUs under their weekly quotas — choosing T4x2, P100, TPU v5e-8 or a CPU kernel, remote training runs with kgkit gpu (build/push/wait/collect), one-fold screens with pruning, resuming past the session limit, porting to torch_xla, and budgeting accelerator hours until the deadline. Use when training on Kaggle notebooks/kernels, planning or rationing GPU/TPU quota, choosing an accelerator, or when a remote run was slow, idle, queued, timed out or wasted hours.
---

# Training on Kaggle GPUs

The quota is the scarce resource: score gained per GPU-hour is what you optimise, not wall time.
Unused quota expires at the weekly reset, so hours you don't spend are also lost.

## 1. Facts to work from (verify live, they change)

- `kaggle quota` (CLI ≥ 2.2): GPU hours (30 h/week) and **TPU hours (20 h/week, a separate quota)**
  used/remaining and `refreshAt` (weekly reset, UTC). `python -m kgkit gpu quota` caches both in
  `.kaggle-gm/gpu_quota.json`, re-checks the runs it has as in flight (finished ones release their
  reservation), lists what still holds hours, and warns when hours are about to expire unused. The live call is
  tried twice; when it still fails the report says `NOT LIVE`, why, and how old the cached value is, and counts the
  runs logged since against it (it cannot see sessions started elsewhere, such as the website or another machine:
  run `gpu quota` again before committing hours on a cached figure).
- Per-session limit 12 h (GPU/CPU), 9 h (TPU). The push timeout `-t` (`--hours`) caps a run below that, and a
  capped run never burns more than its cap. A push without `-t` reserves the full session limit until the run is
  seen finished.
- **`-t` is a hard kill, not only a quota reservation.** Kaggle ends the session at the cap wherever the script is
  (status `CANCEL_ACKNOWLEDGED`, i.e. `cancelled`) and keeps only the files already in `/kaggle/working`. A run
  that ends early is charged only the hours it ran, so a margin costs nothing: cap a kernel of your own at
  ~1.3-1.5x its measured runtime. `gpu build` kernels stop themselves a few minutes before the cap with their
  checkpoints saved.
- Batch sessions at once per account: **2 GPU, 1 TPU**, across all competitions. A push beyond that
  prints `Maximum batch GPU session count of 2 reached` and exits 0. `kgkit gpu push` checks your busy
  sessions first.
- **Inputs are bound at push time.** `kernel_sources` resolve to the outputs that exist when the kernel is
  pushed, not when it starts. A job queued behind a run that is still going starts without that run's new
  output. `kgkit gpu push` refuses when an attached kernel is still queued or running.
- `kaggle kernels logs <user>/<kernel> -f` streams the log of a running GPU kernel (CLI 2.2.4); outputs
  appear when the session ends. For TPU sessions the log stays empty until the end.
- Accelerators (`--accelerator` / `machine_shape`): **`NvidiaTeslaT4`** = 2× T4 16 GB (Turing sm_75,
  fp16 tensor cores, no bf16); **`NvidiaTeslaP100`** = 1× P100 16 GB (sm_60, faster fp32/HBM, no tensor
  cores, no Triton → no `torch.compile`). The quota is charged per session-hour either way, so
  **T4×2 is the default**. One quota hour buys two GPUs, but only if both are busy. Choose P100 only for
  fp32-bound or single-process jobs that cannot be split (verify by measuring step time). Kaggle's PyTorch of
  November 2026 (2.11, CUDA 13) no longer ships `sm_60` kernels, so PyTorch jobs cannot use the P100 there: check
  `torch.cuda.get_arch_list()` first. Kaggle answers a
  shape it does not take with a bare 400, e.g. `NvidiaTeslaT4Highmem`, which `kernels pull -m` can record for
  a past session. `kgkit gpu push` checks the shape before pushing (`--force` if Kaggle has added a new one).
- **A kernel source of 1 MB or more is refused with a bare 400 too** (a script or a notebook, its outputs and
  pasted images included). Data goes in datasets, not in the source. `kgkit gpu build` warns and names the biggest
  parts of the script, and `kgkit gpu push` stops before sending one that big.
- GPU sessions have few CPU cores (typically 4) and ~30 GB RAM. JPEG/DICOM decoding plus heavy CPU augmentation
  starves the GPUs.
- An interactive GPU session burns quota while it is open, even when idle. Batch runs (`kaggle kernels
  push`, "Save Version") stop charging when the script ends. CPU-only sessions don't touch the GPU quota.
- Output: `/kaggle/working` is saved, about 20 GB. A kernel's output can be attached as input to another
  kernel (`kernel_sources`), which is how resume and train→inference chaining work.

## 2. The pipeline (`python -m kgkit gpu ...`)

```
quota / plan        ->  how many hours (GPU and TPU), until when, what to reserve, what holds a reservation
build               ->  .kaggle-gm/build/<name>-train/ (ignored): runner + your src/ + kgkit + competition.json + data/folds.csv
                        (packed column by column: 700k rows of id,fold take ~0.3 MB of the script, not 2.4 MB)
push                ->  input-kernel and busy-session checks, quota check, -t cap, --accelerator, run log
                        (.kaggle-gm/gpu_runs.jsonl). Also any other kernel folder: `gpu push <dir> --hours H`
                        reads kernel-metadata.json (notebooks, inference and holdout-diagnostic kernels)
wait                ->  polls every 90 s; on cancelled/error prints the log tail, says whether the --hours cap
                        stopped it, and lists the files the session saved (still downloadable with
                        `gpu collect`)   (run it in the background)
status              ->  re-checks every run still marked in flight; finished ones free their hours
collect             ->  outputs (no weights by default) -> ledger import, artefacts, GPU/TPU-hour accounting,
                        diagnostics (idle GPU, input-bound, deadline hit, pruned). For a kernel `gpu build` did
                        not make (no kg_run.json): Kaggle's status is the outcome, the files go to
                        data/ext/kout-<slug>/, the session's hours are accounted, the ledger is yours
                        (`ledger import` / `ledger add`)
release <kernel>    ->  drop a reservation by hand (a run that was deleted or never started)
cancel <kernel>     ->  how to stop it: running = website "Stop session"; queued = only `kaggle kernels
                        delete` frees the slot (irreversible: ask the user)
log                 ->  every remote run with its cost and result
```
Raw `kaggle kernels push` calls are logged by the plugin's hook too (every push in a compound command,
each from the folder its `cd` steps lead to), and a re-push of the same kernel replaces its older
reservation. After a `cd` it cannot trace (a variable, a loop) the hook reports the push unlogged: give
`-p` an absolute path there. A push in a command that runs in the background (say, a loop that retries until
a session slot frees up) is logged as pending: its whole cap stays reserved until Kaggle lists a version of
the kernel run after it, which `gpu status` and `gpu wait <kernel>` check (`gpu release <kernel>` if the
push never went through). Your username (for `"id"` in kernel-metadata.json): `python -m kgkit whoami`.

The runner links the attached inputs into `./data` and points `./artifacts` and `./subs` into
`/kaggle/working`. Your training command then runs unchanged, with the same relative paths and the same frozen
folds (so `folds_hash` matches the local ledger). It runs one process per (job, fold), one per GPU,
sets `KG_DEADLINE` so the templates checkpoint before the session ends, and assembles each finished job once.

The frozen split reaches the session cell for cell. `build` packs each column of `data/folds.csv` in the cheapest
exact form: an integer range costs nothing, the folds take a byte per row, other columns go as text, and a big text
column (hash-like ids) is read from `data/train.csv` in the session when that file has it row for row. The runner
writes the file back and checks it against yours; if they differ (a `data/train.csv` that is not the competition's),
it stops before anything runs. If the split is too big even packed (a few million rows), put `data/folds.csv` in a
private dataset (ask the user first) and build with `--split-from <owner>/<slug>`: the run links it from there,
checked byte for byte.

```bash
# full 5-fold run of a promoted idea: folds 0-4 spread over both T4s
python -m kgkit gpu build --name cnx384 --hours 7 --dataset me/knee-png-512 \
    --cmd "python src/train_image.py --name cnx384 --img-size 384 --folds {fold}"
python -m kgkit gpu push .kaggle-gm/build/cnx384-train # outward-facing: confirm with the user first
python -m kgkit gpu wait me/cnx384-train --collect     # background
```

Custom scripts work if they follow the contract: accept `--folds <k>` and `--assemble`, write
`artifacts/<name>/fold<k>_oof.npy` (+ `_idx.npy`, `_test.npy`), and use `kgkit.budget.TrainBudget`
for deadline, resume and pruning. The image and transformer templates already do all of this.

A kernel of your own (`gpu push <dir> --hours H`) has no runner and no deadline: Kaggle kills it at H h (see
section 1). So size H from a measured runtime with a margin, and write the kernel so that a kill costs one phase,
not the run: save the model to `/kaggle/working` right after training, before a long inference or scoring phase;
write predictions in chunks (or per fold) as they are made, so a restart can skip what exists; and print progress
with the elapsed time and an ETA, so the log tail `gpu wait` prints shows where the kill came. When a cap does stop
it, `gpu wait` says so and lists the files saved before the kill.

Where the inputs go (match your local layout, so the command finds them):
- `--input-layout flat` (the default): competition data, `--dataset` and `--kernel` contents side by side in `data/`.
  A name two inputs share (`oof_predictions.parquet`, `submission.csv`, `README.md`) is linked from the first
  only. The run log warns about each one, and `collect` lists them.
- `--input-layout nested`: each input in its own folder, `data/ext/<slug>/` (datasets) and
  `data/ext/kout-<slug>/` (kernel outputs). Competition data stays in `data/`. Use it whenever several inputs
  can share file names, e.g. a stack over public OOF libraries.
- `--dataset <owner>/<slug>=data/<folder>` (or `--kernel`) puts one input where your scripts expect it, e.g.
  the folder you gave `kgkit datasets pull`. `=data` puts it beside the competition data.

An input Kaggle did not mount (not attached, or a kernel still running at push time) is a warning in the log
and a `collect` finding, and so is a missing `--resume-from` kernel, whose folds would be trained again.

```bash
# CPU stack over many OOF libraries (no GPU quota): every library keeps its own files
python -m kgkit gpu build --name stack --hours 2 --accelerator none --input-layout nested \
    --dataset megayak/s6e10-oof mitudru/s6e10-oof=data/ext/mitudru-oof --kernel thisray/s6e10-stack \
    --cmd "python src/stack.py"
```

## 3. Grandmaster protocol for the most leaderboard per GPU-hour

1. **Nothing gets debugged on Kaggle GPUs.** Run `--smoke` locally (CPU is fine) before every push.
   A crash after 20 minutes of pip install and data linking is pure waste.
2. **Preprocess once, off-quota.** Decode DICOM, resize to the training resolution(s), tokenise,
   and cache as uint8 PNG/npy, either locally or in a CPU kernel. Upload the result as a private dataset and attach it with
   `--dataset`. This is usually the single biggest speed-up (2-5× on image tasks).
3. **Screen, then promote.** Test ideas on **one fold** at reduced cost (lower resolution,
   fewer epochs, or a smaller backbone), against the baseline *run under the same reduced settings on
   the same fold* (a "control" screen, run once per screening setting and reused). The baseline's
   full-run fold score is not a fair reference for a cheaper run.
   - Pack two screens into one session so both T4s work:
     `gpu build --name idea-a --folds 0 --cmd "...idea-a... --folds {fold}" --extra-job idea-b "...idea-b... --folds {fold}"`.
   - `--prune-against baseline` stops a screen whose best-so-far trails the accepted baseline's
     learning curve, compared at the same fraction of the schedule, by more than the baseline's fold
     std. This margin is deliberately generous, so only clearly bad ideas are stopped, and they then
     cost 30-40% of a run instead of 100%. (`best` is accepted as an alias.)
   - Promote on a paired test of that fold's OOF against the control:
     `kgkit ledger compare artifacts/<idea> artifacts/<control> --truth data/train.csv:<target>
     --folds data/folds.csv:fold --fold 0` (`--groups` for patients/sessions). z ≥ 2 → promote;
     a positive gain below that → a second fold or seed first. The bootstrap does not see seed noise,
     so treat a z that only just clears 2 on one seed with suspicion. Comparing against the baseline's fold std
     is the wrong test: one fold has no fold-to-fold variance, and the fold std overstates the noise
     of a paired difference.
   - Record `--screen-gain` and, after the full run, `--full-gain` on the backlog item. `kgkit backlog
     fidelity` then tells you whether these screens rank ideas the way full CV does.
4. **Full CV only for promoted ideas.** Run all folds so the ensemble has OOFs on the frozen split. Cap at
   ~1.3× the estimate (per-fold time from `gpu log` × folds ÷ 2 GPUs + ~15 min overhead).
5. **Progressive resizing.** Train most epochs at a lower resolution and fine-tune the last 20-30% at
   full resolution. This gives most of the full-resolution score for 40-60% of the compute. Choose the
   final resolution from screen curves, not by habit.
6. **Diversity beats seeds.** For the final ensemble, a second backbone family or resolution adds more than
   extra seeds of the same model. Spend late quota on diverse members, then blend on OOF
   (`/kg-ensemble`). Seeds are for the last spare hours.
7. **Resume, don't restart.** A run that hits the deadline checkpoints (`fold<k>_resume.pt`, curve,
   status). Rebuild with `--resume-from <user>/<kernel>` (new slug, e.g. `--slug <name>-train-r2`):
   finished folds are skipped and timed-out folds continue mid-schedule.
8. **Chain, don't download.** Attach the training kernel's output to the inference kernel
   (`kernel_sources`) instead of downloading weights and re-uploading them. `collect` skips weights by default.
9. **Use it or lose it.** Hours left at the weekly reset are gone. Plan the current week's remainder
   first and fill it completely before the reset: screens, then the first promotion or a full-fold run
   of the current best (always useful as a baseline OOF and ensemble member). In later weeks, top up
   the last hours with queued, already-validated work (extra folds/seeds of ensemble members, the
   final-resolution variant, pseudo-label rounds). Never start untested exploration just to burn hours.

## 4. Budgeting until the deadline (`python -m kgkit gpu plan`)

- Total ≈ hours left now + resets before the deadline × weekly quota. Count the resets from
  `refreshAt` in 7-day steps; one that lands before the deadline counts, even on the last day.
  `gpu plan` does this arithmetic.
- Reserve the **final week's quota** for final-model training plus inference kernels (code competitions
  run GPU inference notebooks too), with 1-2 sessions of slack for failures.
- Exploration budget = the rest. With measured cost per fold c, a screen costs ≈ c/2 quota-hours
  (2 per session) and a full K-fold promotion ≈ c·K/2. Plan about 4-6 screens per promotion.
- Re-plan after every `collect`. Costs are measured, not guessed.

## 5. When a run disappoints

`collect` prints the diagnosis. Act on it:

| Finding | Fix |
|---|---|
| second T4 idle | ≥ 2 tasks per session (2 folds, or `--extra-job`), or DDP |
| GPU util < 60 % | input-bound: pre-resize, uint8, GPU augmentation, fewer CPU transforms, `persistent_workers`. Confirm with the script's `--bench` before the next push |
| too short to diagnose | a smoke/dev run (< 20 min) is mostly start-up; judge throughput on a longer run |
| deadline hit | `--resume-from`; next time raise `--hours` or lower epochs/res |
| stopped by the `--hours` cap (`cancelled`, log ends at the cap) | `gpu collect <kernel>` for what it saved; re-push with a margin (~1.3-1.5x the runtime) and save the model before the long phase. Same cap again stops it at the same point |
| pruned | discard or change the idea materially; don't re-run it unchanged |
| fold failed | read the log tail; reproduce with `--smoke` locally before re-pushing |

Making each quota hour do more work (measuring input-bound vs compute-bound, AMP, channels_last, compile, data
loaders, fold-parallel vs DDP, GBDTs and RAPIDS on the GPU, inference speed) is the `compute-efficiency` skill. With no
local GPU, a short bench session (`gpu build --name bench --cmd "python src/train_image.py --bench" --hours 0.3`)
shows whether a script is input-bound for minutes of quota; `gpu collect` adds its record to `reports/bench.jsonl`
with the speed-up against the previous bench.

## 6. TPU v5e-8 when the GPU quota is gone

The TPU quota (20 h/week) is separate and expires at the same reset, so it is free hours once the GPU
quota is spent. Measured on a bandwidth-bound ConvNeXt-small (Oct 2026): one v5e chip ≈ **5 T4s** per
step, bf16 losses matching the T4 run to 3-4 decimals. The catch is the queue:

- Batch TPU jobs wait **10-24 h** (12 h 49 min measured) before they start, plus ~20 min of start-up. One
  batch TPU session per account; a queued job holds that slot for its whole wait.
- Expected finish = now + ~13 h queue + 20 min + (T4 time / 5). Use the TPU when that beats the GPU
  finish: for training longer than a GPU session, or when the GPU quota is gone. For runs under ~12 h
  with GPU hours left, the GPU usually wins on wall time. Queue about half a day before you need it,
  and keep a GPU fallback scheduled for after the reset.
- `machine_shape`: ask for `TpuV5E8`. `TpuV5E1`/`TpuV5E4`/`TpuV6E1` are silently stored as `TpuV5E8`;
  `TpuV6E8` (often no capacity) and `Tpu1VmV38` (v3-8) are separate pools, queued too. `gpu build` targets
  v5e-8 and v6e-8.
- A queued job cannot be cancelled through the API (it has no session id yet). The only way to free the
  slot is `kaggle kernels delete` (irreversible; ask the user), then a re-push, which starts the queue
  again from zero. So **validate before queueing**: run the XLA path locally on PJRT CPU, then the same
  code on a ~5 min 2×T4 dev run. A bug found after the queue costs a whole queue cycle.
- `kgkit gpu build --accelerator TpuV5E8 --hours <= 9` runs one process per chip for `{fold}` tasks (PJRT
  `TPU_VISIBLE_CHIPS=<k>` with single-chip process bounds), so each fold/seed keeps the per-process batch
  it had on one GPU. A command without `{fold}` gets the whole TPU (`PJRT_DEVICE=TPU`). Hours are
  accounted as TPU-hours, not GPU-hours.
- Porting a CUDA trainer to torch_xla (bf16, static shapes, `mark_step`, `xm.save`, timm checkpointing,
  resuming CUDA checkpoints): [references/tpu-xla.md](references/tpu-xla.md).

## Guardrails

- `kgkit gpu push` (and raw `kaggle kernels push` with GPU or TPU) spends the user's quota and is
  outward-facing. Confirm with the user unless they authorised remote runs for this task.
- Never print credentials. The username comes from `python -m kgkit whoami` (`KAGGLE_USERNAME`, the
  `username` field of kaggle.json, or `kaggle config view` for token logins). It never reads the key.
- Deleting a kernel (the only way to drop a queued TPU job) destroys every version and output: the user
  decides.
- Keep kernels private (the default) unless the competition rules require publishing.