Add System

---
name: add-system
description: Add a new ML *system* to the TabArena benchmark. Use this skill whenever the user wants to integrate a whole pipeline rather than a single model — AutoML frameworks (AutoGluon, LightAutoML, FLAML, auto-sklearn), LLM-driven agents, hosted prediction APIs (TabPFN-3-API), or a model run through a heavier self-managing interface (TabFM+). Triggers on "add X as a system", "benchmark the X AutoML framework", "wrap this API for TabArena", "integrate this agent". Creates the per-system folder (`system.py`, `hpo.py`, `info.py`) and picks the right `method_class` / `tags`. For a single model under TabArena's shared tuning protocol, use `add-model` instead.
argument-hint: <SystemName> [<pip-package>] [<doc-url>]
user-invocable: true
---

# Add a System to TabArena

## Model or system?

Ask this first, because it decides everything else.

- **Model** — one method that TabArena tunes, using the shared search-space protocol and compute constraints. It plugs into AutoGluon, has an `ag_key`, and gets default / tuned / tuned+ensembled variants. Use the **`add-model`** skill.
- **System** — a pipeline that manages its own budget, model selection, tuning and ensembling. TabArena hands it the data and the constraints and records what comes back. AutoML frameworks, agents, hosted APIs, and models run through a self-managing interface all land here.

A useful test: if you would have to invent a search space for it, it is a model. If inventing one makes no sense because the thing does its own searching, it is a system.

## Layout

Every system lives in one folder at `packages/tabarena/src/tabarena/systems/<system_key>/`, mirroring `models/`:

```
systems/<system_key>/
  __init__.py   re-exports the three below
  system.py     the ExternalSystemModel subclass
  hpo.py        the SystemConfigGenerator (which configurations to benchmark)
  info.py       the SystemInfo + its MethodMetadata
```

`systems/_registry.py::discover_systems()` walks these `info` modules into `SYSTEM_REGISTRY`, keyed by `method_metadata.method`. Read `systems/autogluon/` (a framework) and `systems/tabfm_plus/` (a model through a heavier interface) before writing a new one.

Systems stay **out** of the AutoGluon model registry on purpose: no `ag_key`, no search space, and they run through the experiment bundle's `system_experiments=True` mode.

## Step 1: `system.py`

Subclass `ExternalSystemModel` (`tabarena/benchmark/exec_models/external.py`) and implement `_fit_system`, `_predict` and `_predict_proba`. Read that class's docstring for the full argument contract; the parts that matter most:

- Everything the fit needs is **passed in**, never read off `self`: the raw frames, `target_name`, `problem_type`, `eval_metric`, `validation_metadata`, the compute budget (`num_cpus` / `num_gpus` / `memory_limit` / `time_limit`) and the per-split `random_state`.
- `X` is yours to edit in place. There is no validation split; carve your own from `X`/`y` if the system wants one.
- Add `__init__` arguments for the system's settings and forward `**kwargs` to `super().__init__`. Those arguments are what the config generator varies.
- The compute and time budgets are **not** init knobs. They come per split from the runner, so every system is held to the same constraints.
- Add `cleanup` to free files and memory. Only delete a directory you created; an explicitly passed `path` belongs to the caller.

Keep the library import inside `_fit_system` (or the method that needs it), never at module top level, so an install without the extra still imports.

A system's environment work is its own. TabArena adds no system-specific warm-up, persist or checkpoint prefetch code: a system that imports its stack, downloads checkpoints or loads from disk inside its fit or predict is measured the way it ships (the `AutoGluonSystemModel` docstring spells this out for AutoGluon). Two generic hooks exist, documented in the `ExternalSystemModel` docstring:

- Warm-up: a system may declare `warmup_modules = ("yourlib",)` and/or `warmup_torch_device = True`; that is import and CUDA-context work only, and the synthetic dummy fit is off for systems. Do not write per-system warm-up logic.
- `uses_ray`: keep the base default (True) unless no code path of the system can start Ray; the SLURM worker skips the Ray runtime for fits that answer False.

`SystemInfo.prefetch_weights` is a hook for the system's own tooling; the benchmark setup does not call it, and `offline_weights="auto"` stays off while a system is selected because its checkpoints are not prefetched.

## Step 2: `hpo.py`

```python
gen_<system_key> = SystemConfigGenerator(
    model_cls=<SystemName>SystemModel,
    name="<SystemName>",          # a system has no ag_name/ag_key, so this is required
    manual_configs=[{}, {"preset": "high_quality"}],
)
```

Each config becomes one benchmarked variant. Prefer a small set of meaningful presets over a search space: the point of a system is that it searches for you.

## Step 3: `info.py`

```python
<system_key>_method_metadata = MethodMetadata.system(
    method="<SystemName>",
    name="<SystemName>",
    suite="tabarena-<YYYY-MM-DD>",   # required, must differ from `method`
    compute="cpu" | "gpu",
    date="<YYYY-MM-DD>",
    date_introduced="<YYYY-MM>",
    reference_url="...",
    license="Apache-2.0",            # what governs using the system, incl. every bundled model's weights
    commercial_use=True,             # False when any bundled component is non-commercial (e.g. TabPFN-3)
    tags=(),                         # see below
    verified=False,                  # until signed off
)

<system_key>_info = SystemInfo(
    system_cls=<SystemName>SystemModel,
    config_generator=gen_<system_key>,
    method_metadata=<system_key>_method_metadata,
    pip_extra=("<package>==<version>",),
    prefetch_weights=None,           # optional hook for the system's own tooling; the benchmark setup does not call it
)
```

`MethodMetadata.system(...)` fixes `method_type="baseline"` (a system's raw results are recorded that way by the runner) and sets `method_class="system"`. `SystemInfo` asserts the latter, so a misdeclared system fails at import instead of misclassifying on the leaderboard.

### Choosing tags

`tags` is what lets a reader rule a system out, and it decides which entrant pools it competes in (`evaluation/entrants.py`). Only two values exist; ask the user when either is unclear rather than guessing.

| tag | when | effect |
|---|---|---|
| `with-llm` | an LLM is involved anywhere, agents included | competes only where the `llm` category is selected |
| `closed-source-api` | runs behind a remote API we cannot inspect | competes only where the `api` category is selected |

A system carrying both tags needs both categories selected, so it never appears on the strength of a property the reader excluded.

No tags means open-source, local and LLM-free, which is the common case (AutoGluon, LightAutoML, FLAML, TabFM+) and puts the system in the `open` category. The two tags are independent: an open-source agent is `("with-llm",)`, and a hosted non-LLM predictor is `("closed-source-api",)`. Each combination of categories is published as its own pool, so a new tag doubles the artifact count.

If the system needs a property neither tag covers, do not invent one inline. Add it to `MethodTag` in `models/_method_metadata.py`, give it presentation in `website_format.TAG_SPECS`, and decide which pools admit it in `evaluation/entrants.py`. All three or none, otherwise it will not render.

## Step 4: the pip extra

Add the system's dependency to the `[project.optional-dependencies]` block in `packages/tabarena/pyproject.toml`, matching `SystemInfo.pip_extra`.

## Step 5: register and verify

- Add the metadata to `tabarena_method_metadata_collection` in `contexts/tabarena/methods.py` once results exist (see the `upload-method` skill for processing and hosting them).
- `pytest tests/tabarena/systems/ -q` — the registry test checks the new system is discovered and declares `method_class="system"`.
- `ruff check` **and** `ruff format --check` on every touched file.

There is no per-system fit test. Verify the wrapper with the quickstart in `examples/benchmarking/run_quickstart_tabarena_system.py`, which runs two configs of the demo system on the small datasets' first split (`examples/beyondarena/run_quickstart_beyondarena_system.py` is the BeyondArena counterpart and runs the shipped AutoGluon wrapper).

## Step 6: Report and open the PR

Summarize the new files, the edited files, the chosen `tags` and why, and the open TODOs (results,
`verified`, the `methods.py` registration once artifacts exist).

When asked to open the PR, use `.github/pull_request_template.md`: a two-to-four sentence summary,
everything longer inside the collapsed `<details><summary>Details</summary>` block, the commands run
under Tests. Fill in the "Model or system submission" section for a system (delete the model lines)
and keep the closing contribution line. Do not paste the Report into the PR body; the Report is for
the chat, the PR body is for the reviewers. State the TabArena-Lite (or BeyondArena `core`) results
with the hardware and the entry-point script if they exist; if they do not, say so, since a
maintainer will ask (TabArena verifies submitted results by re-running them, it does not benchmark
on request). Questions go through the issue forms in `.github/ISSUE_TEMPLATE/`.

## What happens on the leaderboard

The system appears under the 📊 **System** family, typed from `method_class` rather than from its name, with a chip per tag. It shows up in whichever entrant pools admit it, and its presence changes every other entrant's Elo and Improvability in those pools, since both are measured against the field.