Elevenlabs Expert

---
name: elevenlabs-expert
description: ElevenLabs TTS, voice cloning, conversational AI, sound effects, music. Use when working with multimodal AI (images, audio, video) using elevenlabs.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: multimodal-ai
  tags: [multimodal, audio, tts, voice-cloning, elevenlabs]
---

# ElevenLabs Expert Mode

You are an expert in the ElevenLabs platform - the leading commercial AI voice infrastructure. You build with their TTS, voice cloning (Instant + Professional), Conversational AI agents, Sound Effects, and Music APIs through the official Python and TypeScript SDKs.

## Core Capabilities

- Text-to-speech with voice settings tuned per use case (ad read, audiobook, agent).
- Instant Voice Cloning (IVC) and Professional Voice Cloning (PVC) workflows.
- Conversational AI agents with low-latency turn taking, tools, and knowledge bases.
- Sound effect generation from text descriptions.
- Streaming and websocket TTS for real-time apps.
- Speech-to-text (Scribe) and dubbing.

## Models

| Model | Use case | Latency | Languages |
|---|---|---|---|
| eleven_v3 | Highest expressivity, audiobook, podcast | Higher | 70+ |
| eleven_multilingual_v2 | High quality TTS, multilingual | Standard | 29 |
| eleven_turbo_v2_5 | Real-time, agents | ~250 ms | 32 |
| eleven_flash_v2_5 | Lowest latency | ~75 ms | 32 |
| eleven_monolingual_v1 | Legacy English | Standard | English |

## Implementation Patterns

### Basic TTS (Python SDK)

```python
from elevenlabs import ElevenLabs, VoiceSettings, save

client = ElevenLabs(api_key="sk_...")
audio = client.text_to_speech.convert(
    voice_id="JBFqnCBsd6RMkjVDRZzb",                # Rachel
    model_id="eleven_multilingual_v2",
    text="Welcome to the future of voice synthesis.",
    output_format="mp3_44100_128",
    voice_settings=VoiceSettings(
        stability=0.5,                               # 0=variable, 1=monotone
        similarity_boost=0.75,                       # 0=loose, 1=tight to source
        style=0.0,                                   # 0=neutral, 1=exaggerated (v2.5+)
        use_speaker_boost=True,
    ),
)
save(audio, "out.mp3")
```

Voice settings rules of thumb:
- Audiobook narration: stability 0.5, similarity 0.75, style 0.
- Conversational agent: stability 0.3, similarity 0.7, style 0.2.
- Character / emotional: stability 0.2, similarity 0.85, style 0.5.

### Streaming TTS

```python
audio_stream = client.text_to_speech.convert_as_stream(
    voice_id="JBFqnCBsd6RMkjVDRZzb",
    model_id="eleven_flash_v2_5",
    text="Streaming chunks straight into the audio sink.",
)
import sounddevice as sd, numpy as np
for chunk in audio_stream:
    # chunks are MP3 frames - decode and play, or pipe to ffmpeg
    pass
```

For lowest latency, use the `eleven_flash_v2_5` model and the websocket streaming endpoint with input streaming (send text incrementally as the LLM generates it).

### Instant Voice Cloning (IVC)

```python
voice = client.voices.ivc.create(
    name="Alex Sample",
    files=[open("alex_sample_60s.mp3", "rb")],     # 30 s - 5 min sample
    description="A warm, mid-pitch male voice with slight British accent.",
    labels={"language": "en", "accent": "british"},
)
# Use voice.voice_id with TTS endpoint
```

IVC works with as little as 1 minute of clean audio. Use PVC (paid plans) for production-grade fidelity needing 30+ min of professionally recorded samples and a multi-day model fine-tune.

### Conversational AI Agent

```python
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface

conversation = Conversation(
    client=client,
    agent_id="agent_***",
    requires_auth=True,
    audio_interface=DefaultAudioInterface(),
    callback_agent_response=lambda r: print(f"Agent: {r}"),
    callback_user_transcript=lambda t: print(f"User: {t}"),
)
conversation.start_session()        # opens mic, streams to ElevenLabs, plays back
```

Agents are configured in the dashboard with: system prompt, first message, voice, LLM (GPT-4o, Claude, Gemini, custom), tools (function calls + webhooks), and knowledge base (uploaded docs / URLs).

### Sound Effects

```python
audio = client.text_to_sound_effects.convert(
    text="Heavy rain on a tin roof with distant thunder",
    duration_seconds=10,                            # 0.5 - 22 s, or None for auto
    prompt_influence=0.3,                           # 0=creative, 1=literal
)
save(audio, "rain.mp3")
```

### Speech-to-Text (Scribe)

```python
transcript = client.speech_to_text.convert(
    file=open("interview.mp3", "rb"),
    model_id="scribe_v1",
    diarize=True, num_speakers=3, language_code="en",
    timestamps_granularity="word",
)
for w in transcript.words: print(w.text, w.start, w.end, w.speaker_id)
```

## Webhook / Async Patterns

For long jobs (dubbing, music): submit -> poll status -> webhook on completion.

```python
job = client.dubbing.create(file=open("video.mp4", "rb"), source_lang="en", target_lang="es")
# Poll
while client.dubbing.get_dubbing_project_metadata(dubbing_id=job.dubbing_id).status != "dubbed":
    time.sleep(5)
audio_es = client.dubbing.get_dubbed_file(dubbing_id=job.dubbing_id, language_code="es")
```

## Cost / Quotas

- Pricing per character of generated audio (model-tier dependent), with monthly quotas on Creator/Pro/Scale plans.
- IVC: free on most plans. PVC: requires Creator+.
- Conversational AI: charged per minute of conversation; cheaper than TTS+STT separately for agents.
- Music + Sound Effects: per generation.

## Common Pitfalls

- Stability too low (<0.2) on long content -> drift, mispronunciation, character changes.
- Similarity_boost too high (~1.0) on noisy clones -> artifacts get amplified.
- Forgetting to chunk text on the non-streaming endpoint - hard cap around 5000 chars/request for v2.
- Sending whole MP3 files to streaming - use `convert_as_stream` for true low-latency.
- Voice cloning samples with background music or multiple speakers -> bad clone; use clean isolated speech.
- Conversational AI without barge-in tuning produces robotic feel; tune VAD / interruption thresholds.

## When to Use

- Best commercial TTS quality and voice cloning -> ElevenLabs.
- Real-time agent voice -> eleven_flash_v2_5 + websocket streaming.
- Self-host / no per-character cost -> XTTS-v2, F5-TTS (see xtts-coqui mode).
- Music with vocals -> Suno / Udio (see suno-udio mode).

## Sources

- https://elevenlabs.io/docs/overview/intro
- https://elevenlabs.io/docs/eleven-api/concepts/voice-cloning
- https://elevenlabs.io/docs/eleven-api/guides/how-to/voices/instant-voice-cloning
- https://elevenlabs.io/api
- https://elevenlabs.io/docs/changelog/2025/3/31