2068 Openai Api Routing 3626ac97

# OpenAI API Routing and Proxy Patterns

voice-mode uses the OpenAI SDK, which means you can redirect all API traffic through a custom router using the standard `OPENAI_BASE_URL` environment variable - no voice-mode specific code or configuration needed.

## Why OpenAI API Compatibility Matters

By using only standard OpenAI endpoints, voice-mode becomes a simple API client that can work with any OpenAI-compatible service. This design enables:

1. **Service Agnostic**: Works with OpenAI, Azure OpenAI, local services, or any compatible API
2. **Simple Configuration**: Just change the `BASE_URL` environment variable
3. **External Control**: All routing logic lives outside voice-mode
4. **Future Proof**: New providers can be added without code changes

## Routing Patterns

### 1. Simple Proxy with Fallback

```nginx
# nginx configuration example
upstream openai_primary {
    server api.openai.com:443;
}

upstream openai_fallback {
    server 127.0.0.1:8880;  # Local Kokoro TTS
}

location /v1/audio/speech {
    proxy_pass https://openai_primary;
    proxy_intercept_errors on;
    error_page 502 503 504 = @fallback;
}

location @fallback {
    proxy_pass http://openai_fallback;
}
```

### 2. Cost-Optimized Router

```python
# Python routing proxy example
from fastapi import FastAPI, Request
import httpx

app = FastAPI()

@app.post("/v1/audio/speech")
async def route_tts(request: Request):
    body = await request.json()
    
    # Route based on model
    if body.get("model") == "tts-1":
        # Use local Kokoro for standard TTS
        async with httpx.AsyncClient() as client:
            return await client.post(
                "http://127.0.0.1:8880/v1/audio/speech",
                json=body
            )
    else:
        # Use OpenAI for advanced models
        async with httpx.AsyncClient() as client:
            return await client.post(
                "https://api.openai.com/v1/audio/speech",
                json=body,
                headers={"Authorization": request.headers.get("Authorization")}
            )
```

### 3. Voice-Specific Routing

```javascript
// Node.js Express router
app.post('/v1/audio/speech', async (req, res) => {
    const { voice, model } = req.body;
    
    // Route Sky voice to Kokoro, others to OpenAI
    if (voice === 'af_sky') {
        const response = await fetch('http://127.0.0.1:8880/v1/audio/speech', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ ...req.body, model: 'kokoro-v0.87' })
        });
        res.send(await response.arrayBuffer());
    } else {
        // Forward to OpenAI
        const response = await fetch('https://api.openai.com/v1/audio/speech', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': req.headers.authorization
            },
            body: JSON.stringify(req.body)
        });
        res.send(await response.arrayBuffer());
    }
});
```

### 4. Load Balancer with Health Checks

```yaml
# HAProxy configuration
global
    maxconn 4096

defaults
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

backend tts_servers
    balance roundrobin
    option httpchk GET /health
    
    server openai api.openai.com:443 ssl verify none check
    server kokoro1 127.0.0.1:8880 check
    server kokoro2 127.0.0.1:8881 check backup
```

## Configuration

### Simple Global Routing

The easiest approach is to use the OpenAI SDK's built-in support:

```bash
# Route ALL OpenAI API calls through your proxy
export OPENAI_BASE_URL="https://router.example.com/v1"
export OPENAI_API_KEY="your-key"
```

That's it! The OpenAI SDK automatically uses this base URL for all API calls.

**Note**: If `STT_BASE_URL` or `TTS_BASE_URL` are not explicitly set, voice-mode defaults to using the OpenAI API. When `OPENAI_BASE_URL` is set, the OpenAI SDK will use it automatically for these default cases.

### Provider-Specific Routing

For more granular control, voice-mode also supports provider-specific endpoints:

```bash
# Use different endpoints for STT and TTS
export STT_BASE_URL="http://127.0.0.1:2022/v1"  # Local Whisper
export TTS_BASE_URL="http://127.0.0.1:8880/v1"  # Local Kokoro
```

Your router receives all requests and can then:
- Route to different providers
- Add authentication
- Log requests
- Implement rate limiting
- Cache responses
- Transform requests/responses

## Benefits

1. **Zero Code Changes**: voice-mode doesn't need to know about routing logic
2. **Centralized Control**: Manage all API routing in one place
3. **Dynamic Switching**: Change routing rules without restarting voice-mode
4. **Multi-Service**: One router can handle STT, TTS, and other OpenAI APIs
5. **Monitoring**: Add metrics, logging, and observability at the proxy layer

## Example: LiteLLM Proxy

[LiteLLM](https://github.com/BerriAI/litellm) provides a ready-made proxy for OpenAI-compatible routing:

```bash
# Install LiteLLM
pip install litellm[proxy]

# Configure proxy
cat > litellm_config.yaml << EOF
model_list:
  - model_name: tts-1
    litellm_params:
      model: openai/tts-1
      api_key: $OPENAI_API_KEY
  - model_name: whisper-1
    litellm_params:
      model: openai/whisper-1
      api_key: $OPENAI_API_KEY
EOF

# Start proxy
litellm --config litellm_config.yaml --port 9000

# Configure voice-mode
export TTS_BASE_URL="http://127.0.0.1:9000"
export STT_BASE_URL="http://127.0.0.1:9000"
```

This architecture ensures voice-mode remains simple while enabling sophisticated deployment patterns through external routing layers.