Multimodal I O

---
name: multimodal-i-o
description: "Use GenAI Processors audio, video, PDF, web, file, and speech connectors."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Multimodal I/O

Use this sub-skill when the task is about getting content into or out of a
processor pipeline: microphones, speakers, cameras, screens, PDFs, URLs,
GitHub, Google Drive, files, speech recognition, speech synthesis, VAD, and
realtime media flow.

## Read when

- The task names `PyAudioIn`, `PyAudioOut`, `VideoIn`, `AudioToWav`, `Vad`,
  `SpeechToText`, `TextToSpeech`, `PDFExtract`, `UrlFetch`, `GithubProcessor`,
  `Drive`, `RateLimitAudio`, `EventDetection`, `add_timestamps`, `to_timestamp`, or `Window`.
- The task is about MIME types, audio sample rates, video frames, PDFs, URLs,
  files, or Google Workspace documents.
- The task needs a realtime audio/video pipeline before or after a model
  wrapper.
- The task needs optional dependency or device troubleshooting.

## Boundaries

This sub-skill owns connectors and media/document processing. It does not own
model selection or function-calling strategy; use `../model-backends/` for
that. It does not own complete app/demo layout; use `../examples-and-apps/` for
full CLIs and AI Studio applets.

## Workflow checklist

1. Identify source modality: terminal text, audio, camera/screen frames, file
   glob, PDF bytes, URL, GitHub URL, Drive doc/sheet/slide, or tool output.
2. Confirm MIME type and substream expectations before composing with model
   processors.
3. Verify optional imports with `scripts/smoke_io.py` before using devices or
   credentials.
4. For audio/video, check device permissions and prefer headphones unless using
   browser echo cancellation.
5. For Google Speech/TTS or Drive, validate credentials and project/service
   setup before running full pipelines.
6. For URLs/GitHub/PDFs, separate safe parser/import smoke from network or file
   reads.
7. When combining with live/realtime models, cross-check trigger and substream
   behavior in `../model-backends/`.

## References and scripts

- `references/api-reference.md` lists connector classes and processor roles.
- `references/workflows.md` provides audio, video, PDF, URL, Drive/GitHub, and
  realtime media patterns.
- `references/troubleshooting.md` covers PortAudio, device permissions,
  Google Cloud auth, PDF rendering, OpenCV/AV, and web fetch failures.
- `scripts/smoke_io.py` imports optional I/O modules without opening devices,
  making network requests, or requiring credentials.

## Native evidence anchors

Evidence comes from source modules under `genai_processors/core/`, docs under
`documentation/docs/development/` and `documentation/docs/concepts/realtime.md`,
and tests including `audio_test.py`, `rate_limit_audio_test.py`,
`speech_to_text_test.py`, `text_to_speech_test.py`, `vad_test.py`,
`video_test.py`, `pdf_test.py`, `web_test.py`, `github_test.py`,
`drive_test.py`, `filesystem_test.py`, `event_detection_test.py`,
`timestamp_test.py`, and `window_test.py`.

## Usability checkpoints

A good answer using this sub-skill should:

- Name exact optional packages and environment variables before runnable code.
- Avoid opening microphones, speakers, cameras, browsers, or remote services in
  a smoke check.
- Preserve non-text parts until a downstream processor intentionally converts
  them.
- Include concrete MIME types and substream names when they affect routing.
- Route model-specific or applet-specific follow-up work to the proper sibling
  sub-skill.