Multimodal Automl

---
name: multimodal-automl
description: "Use AutoGluon MultiModalPredictor for text, image,
  tabular+text/image, document, NER, semantic matching, object detection,
  semantic segmentation, feature extraction, zero-shot, and deployment
  workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# MultiModal AutoML

Use this sub-skill when the user needs AutoGluon AutoMM or `MultiModalPredictor` for text, image, mixed tabular+text/image, document, NER, semantic matching, object detection, semantic segmentation, feature extraction, zero-shot inference, or multimodal deployment/export tasks.

Route structured-only supervised learning to `../tabular-ml/` and forecasting to `../time-series-forecasting/`. Do not treat external dataset downloads, competition submissions, or raw repository CI examples as runtime requirements; distill the needed data/API pattern and work with the user's local files.

## Start Here

- For constructor and method signatures, problem types, metrics, and model config knobs, read `references/api-reference.md`.
- For end-to-end task recipes, including semantic matching and save/load, read `references/workflows.md`.
- For DataFrame, image, document, NER, matching, segmentation, and local path expectations, read `references/data-formats.md`.
- For COCO/VOC/DataFrame object detection handling, read `references/object-detection.md`.
- For ONNX, TensorRT, embedding extraction, and offline packaging, read `references/deployment.md`.
- For dependency, backend, schema, and checkpoint failure modes, read `references/troubleshooting.md`.

## Safe Bundled Helpers

- Use `scripts/inspect_multimodal_inputs.py --help` to validate local CSV/JSON data, image/document paths, matching identifiers, COCO metadata, or VOC-style annotations without training or downloading.
- Use `scripts/multimodal_smoke.py --help` to check installed AutoGluon multimodal imports, API signatures, problem types, and optional backend versions without training by default.

## Routing Guide

| User request | Read first | Key API |
| --- | --- | --- |
| Text/image/mixed classification or regression | `references/workflows.md` | `MultiModalPredictor(label=..., problem_type=...)` |
| Document image/PDF classification | `references/data-formats.md` | `column_types={...: "document"}` or inferred document columns |
| NER/entity extraction | `references/data-formats.md` | `problem_type="ner"`, `column_types={text_col: "text_ner"}` |
| Text/image/image-text similarity or retrieval | `references/workflows.md` | `query=...`, `response=...`, `match_label=...`, `candidate_data`, `id_mappings` |
| Object detection | `references/object-detection.md` | `problem_type="object_detection"`, COCO/VOC/DataFrame data |
| Semantic segmentation | `references/data-formats.md` | `problem_type="semantic_segmentation"`, image and mask columns |
| Feature extraction or zero-shot image classification | `references/workflows.md` | `problem_type="feature_extraction"` or `"zero_shot_image_classification"` |
| ONNX/TensorRT/export/offline deployment | `references/deployment.md` | `save`, `load`, `export_onnx`, `optimize_for_inference` |

## Default Agent Approach

1. Identify the task family and route away if it is structured-only tabular or time-series.
2. Confirm data shape before code: label/query/response columns, local image/document paths, object-detection annotation format, segmentation mask availability, and optional backend constraints.
3. Prefer CPU-safe, no-download validation first with the bundled scripts; only propose training/export after local data and dependencies are plausible.
4. Keep `path`/`save_path` explicit for real training, use short `time_limit` only for smoke examples, and set backend hyperparameters intentionally for CPU/GPU.
5. For matching/retrieval, distinguish row-pair prediction from candidate search: `predict(data, candidate_data=...)`, `evaluate(query_data=..., response_data=...)`, or embedding-based search.
6. For deployment, call out pickle trust for `load`, optional `standalone=True` packaging, and ONNXRuntime/TensorRT dependency requirements.