Speechlm2

---
name: speechlm2
description: "Use for NeMo SpeechLM2/SALM/SALMAutomodel, duplex STT/S2S/EAR-TTS,
  Nemotron VoiceChat, HuggingFace export/conversion, vLLM plugin, Automodel
  backends, and voice-agent troubleshooting."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# SpeechLM2

Use this sub-skill for NeMo SpeechLM2 speech-LLM work: SALM/SALMAutomodel, duplex STT and speech-to-speech models, Duplex EAR-TTS inside SpeechLM2 pipelines, Nemotron VoiceChat, HuggingFace/safetensors export, the NeMo SpeechLM vLLM plugin, Automodel optional compiled backends, and local voice-agent configuration.

This guidance is self-contained. It distills repository evidence from `docs/source/speechlm2/intro.rst`, `docs/source/speechlm2/models.rst`, `docs/source/speechlm2/datasets.rst`, `docs/source/speechlm2/configs.rst`, `docs/source/speechlm2/training_and_scaling.rst`, `docs/source/checkpoints/intro.rst`, `examples/speechlm2/*.py`, `examples/speechlm2/conf/*.yaml`, `examples/voice_agent/**`, `nemo/collections/speechlm2/**`, `nemo/agents/voice_agent/**`, and `tests/collections/speechlm2/**`.

## Route Here When

- The task mentions `nemo.collections.speechlm2`, `SALM`, `SALMAutomodel`, `SALMWithAsrDecoder`, `DuplexS2SModel`, `DuplexS2SSpeechDecoderModel`, `DuplexSTTModel`, `DuplexEARTTS`, or `NemotronVoiceChat`.
- The user needs speech+LLM generation, SpeechLM2 training configs, Lhotse conversation datasets, duplex speech-to-speech/STT evaluation, or SpeechLM2 inference with audio placeholder tokens.
- The task involves SpeechLM2 HuggingFace export, `model.safetensors`, distributed checkpoint consolidation, `NemotronVoiceChat.from_pretrained`, or vLLM serving with `nemo_speechlm`.
- The task involves NeMo voice-agent server/client configuration, tool calling, vLLM/HF LLM backend selection, streaming ASR/TTS service composition, or backchannel/turn-taking settings.

## Route Elsewhere

- Classic ASR transcription, ASR fine-tuning, CTC/RNNT decoding, Parakeet/Canary model recipes, or ASR manifests belong in `../asr/SKILL.md` unless SpeechLM2 is explicitly involved.
- TTS-only FastPitch, HiFiGAN, Magpie, Kokoro, or speech synthesis outside a SpeechLM2/Nemotron VoiceChat/voice-agent pipeline belongs in `../tts/SKILL.md`.
- Repo development, test authoring, packaging, lint, CI, and source-tree maintenance belong in the repo-development sub-skill when available.

## First Steps

1. Identify the workflow: SALM generation/eval, SALM/SALMAutomodel training, duplex STT/S2S, Duplex EAR-TTS inference, Nemotron VoiceChat, HF export, vLLM serving, or voice-agent deployment.
2. Read `references/workflows.md` for the concrete API/command-shape workflow and safe adaptation notes.
3. Read `references/configuration.md` before editing YAML, Hydra overrides, dataset paths, conversation roles, parallelism, or voice-agent config.
4. Read `references/checkpoints-and-export.md` before loading, resuming, converting, exporting, or serving checkpoints.
5. Read `references/troubleshooting.md` when imports, CUDA, compiled backends, data schemas, Hydra overrides, vLLM, or voice-agent services fail.
6. Run `scripts/check_speechlm2_config.py --help` and then use it on user-provided SpeechLM2 YAML/JSON configs before long-running jobs.

## Core Facts

- SpeechLM2 is active development code for augmenting pretrained LLMs with speech understanding and speech generation; expect APIs/configs to change more often than classic ASR/TTS.
- `SALM` uses HuggingFace Transformers for the LLM backbone; `SALMAutomodel` uses NeMo Automodel and is the right path for FSDP2, TP, PP, CP, EP/MoE, HSDP, native Automodel LoRA, and distributed inference.
- Distributed SALM generation/evaluation requires `SALMAutomodel`; plain `SALM` should stay single-process/single-device for inference.
- `NemotronVoiceChat` is inference-only. Train or fine-tune its STT and EAR-TTS components separately, then export a joint VoiceChat checkpoint.
- SpeechLM2 model checkpoints are HuggingFace-style directories with `config.json` and `model.safetensors`; `.nemo` is used in this collection mainly as an input format for pretrained ASR/perception components.
- Training and most inference require a compatible GPU/CUDA/PyTorch stack. Optional Automodel compiled backends such as TransformerEngine, DeepEP, grouped GEMM, flash-attn, vLLM, Pipecat, browser/client tooling, and Node/npm must be installed only when that workflow actually needs them.

## Bundled Assets

- `references/workflows.md`: SALM/SALMAutomodel generation and training, duplex STT/S2S/EAR-TTS, Nemotron VoiceChat, vLLM, and voice-agent workflows.
- `references/configuration.md`: SpeechLM2 YAML structure, data schemas, Hydra overrides, parallelism, and voice-agent configuration patterns.
- `references/checkpoints-and-export.md`: checkpoint formats, `from_pretrained`, `save_pretrained`, HF export, vLLM readiness, and Nemotron VoiceChat conversion.
- `references/troubleshooting.md`: actionable failure diagnosis for install/import, CUDA/backends, data/config, API/CLI, export, vLLM, and voice-agent issues.
- `scripts/check_speechlm2_config.py`: deterministic local checker for SpeechLM2 YAML/JSON-like configs; it performs text-level validation without importing NeMo or downloading models.