Openrouter Load Balancing

---
name: openrouter-load-balancing
description: 'Distribute OpenRouter requests across multiple keys and models for high
  throughput. Use when scaling beyond single-key rate limits or building high-availability
  systems. Triggers: ''openrouter load balance'', ''openrouter scaling'', ''distribute
  openrouter requests'', ''multiple api keys''.

  '
allowed-tools: Read, Write, Edit, Grep, Bash(python3:*)
version: 1.20.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
tags:
- saas
- openrouter
- scaling
- high-availability
- load-balancing
compatibility: Designed for Claude Code
---
# OpenRouter Load Balancing

## Overview

A single OpenRouter API key has rate limits (requests/minute and tokens/minute). To scale beyond those limits, distribute requests across multiple keys. OpenRouter also provides server-side load balancing via provider routing and the `:nitro` variant for low-latency inference. This skill covers multi-key rotation, health-based routing, circuit breakers, and concurrent request patterns.

## Prerequisites

- Two or more OpenRouter API keys exported as `OPENROUTER_KEY_1`, `OPENROUTER_KEY_2`, `OPENROUTER_KEY_3` so the `KeyPool` has keys to rotate — see the `openrouter-install-auth` skill for creating and exporting keys
- `OPENROUTER_API_KEY` exported for the single-key concurrent-processing pattern
- Python 3.8+ with the OpenAI SDK and `requests` (`pip install openai requests`) — the concurrent example uses `AsyncOpenAI` from the same package
- Adequate credits on every key in the pool; per-key quota is visible via `GET /api/v1/auth/key`

## Instructions

1. Export your pool keys and build the `KeyPool` from Multi-Key Round Robin — it round-robins across keys, trips a circuit breaker after 3 consecutive errors, and auto-recovers a key after a 60s cooldown.
2. Send traffic through `balanced_completion()`: on `RateLimitError` it calls `pool.mark_error(key)` and retries with the next healthy key.
3. For batch workloads, use `parallel_completions()` from Concurrent Request Processing — an `asyncio.Semaphore` (`max_concurrent=3-5`) caps in-flight requests against a single key.
4. Layer on server-side distribution per Provider-Level Load Balancing: pass `extra_body={"provider": {"order": [...], "allow_fallbacks": True}}` so OpenRouter spreads the same model across Anthropic, AWS Bedrock, and GCP Vertex.
5. Monitor quota per key with `check_rate_limits()` (`GET /api/v1/auth/key`) from Rate Limit Awareness, and when 429s hit all keys simultaneously, apply the fixes in Error Handling (more keys, request queuing).

## Multi-Key Round Robin

```python
import os, itertools, time, logging
from openai import OpenAI, RateLimitError
from dataclasses import dataclass, field

log = logging.getLogger("openrouter.lb")

@dataclass
class KeyPool:
    """Round-robin API key pool with health tracking."""
    keys: list[str]
    _cycle: itertools.cycle = field(init=False, repr=False)
    _health: dict[str, dict] = field(init=False, default_factory=dict)

    def __post_init__(self):
        self._cycle = itertools.cycle(self.keys)
        self._health = {k: {"errors": 0, "last_error": 0, "healthy": True} for k in self.keys}

    def next_key(self) -> str:
        """Get next healthy key."""
        attempts = 0
        while attempts < len(self.keys):
            key = next(self._cycle)
            h = self._health[key]
            # Recover after 60s cooldown
            if not h["healthy"] and time.time() - h["last_error"] > 60:
                h["healthy"] = True
                h["errors"] = 0
            if h["healthy"]:
                return key
            attempts += 1
        # All keys unhealthy -- return any and hope for the best
        return next(self._cycle)

    def mark_error(self, key: str):
        h = self._health[key]
        h["errors"] += 1
        h["last_error"] = time.time()
        if h["errors"] >= 3:  # Circuit breaker: 3 errors → unhealthy
            h["healthy"] = False
            log.warning(f"Key {key[:12]}... marked unhealthy after {h['errors']} errors")

    def mark_success(self, key: str):
        self._health[key]["errors"] = 0
        self._health[key]["healthy"] = True

pool = KeyPool(keys=[
    os.environ.get("OPENROUTER_KEY_1", ""),
    os.environ.get("OPENROUTER_KEY_2", ""),
    os.environ.get("OPENROUTER_KEY_3", ""),
])

def balanced_completion(messages, model="anthropic/claude-3.5-sonnet", **kwargs):
    """Send request using next healthy key from the pool."""
    key = pool.next_key()
    client = OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=key,
        default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
    )
    try:
        response = client.chat.completions.create(
            model=model, messages=messages, **kwargs
        )
        pool.mark_success(key)
        return response
    except RateLimitError:
        pool.mark_error(key)
        # Retry with next key
        return balanced_completion(messages, model, **kwargs)
```

## Concurrent Request Processing

```python
import asyncio
from openai import AsyncOpenAI

async def parallel_completions(prompts: list[str], model="openai/gpt-4o-mini",
                                max_concurrent=5, **kwargs):
    """Process multiple prompts concurrently with rate limiting."""
    semaphore = asyncio.Semaphore(max_concurrent)
    client = AsyncOpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=os.environ["OPENROUTER_API_KEY"],
        default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
    )

    async def process_one(prompt: str):
        async with semaphore:
            response = await client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                **kwargs,
            )
            return response.choices[0].message.content

    return await asyncio.gather(*[process_one(p) for p in prompts])

# Usage
results = asyncio.run(parallel_completions(
    ["Summarize X", "Translate Y", "Analyze Z"],
    max_concurrent=3,
    max_tokens=500,
))
```

## Provider-Level Load Balancing

```python
# OpenRouter can distribute across providers for the same model
response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=200,
    extra_body={
        "provider": {
            # Let OpenRouter pick the best available provider
            "order": ["Anthropic", "AWS Bedrock", "GCP Vertex"],
            "allow_fallbacks": True,
        },
    },
)
```

## Rate Limit Awareness

```python
import requests

def check_rate_limits(api_key: str) -> dict:
    """Check current rate limit status for a key."""
    resp = requests.get(
        "https://openrouter.ai/api/v1/auth/key",
        headers={"Authorization": f"Bearer {api_key}"},
    )
    data = resp.json()["data"]
    return {
        "requests_limit": data["rate_limit"]["requests"],
        "interval": data["rate_limit"]["interval"],
        "credits_used": data["usage"],
        "credits_limit": data.get("limit"),
    }

# Check all keys in pool
for key in pool.keys:
    limits = check_rate_limits(key)
    print(f"Key {key[:12]}...: {limits}")
```

## Output

- Chat completion responses served through whichever pool key was healthy at send time, plus per-key health state: error counts, `healthy` flags, and log lines like `Key sk-or-v1-abc... marked unhealthy after 3 errors`
- An ordered list of completion strings from `parallel_completions()` — one per input prompt, gathered concurrently
- Rate-limit status dicts per key from `check_rate_limits()`: `requests_limit`, `interval`, `credits_used`, `credits_limit`

## Examples

Six requests through a two-key pool split evenly, and the pool's stats confirm the distribution:

```python
for i in range(6):
    balanced_completion(f"Request {i}: Hello!")
print(pool.get_stats())
# {'sk-or-v1-abc': {'requests': 3, 'errors': 0},
#  'sk-or-v1-def': {'requests': 3, 'errors': 0}}
```

Zero errors means no key tripped the circuit breaker; a nonzero `errors` count on one key with requests skewing to the other shows health-based routing doing its job. More worked examples: `references/examples.md`.

## Error Handling

| Error | Cause | Fix |
|-------|-------|-----|
| 429 on all keys | All keys rate-limited simultaneously | Add more keys; implement request queuing |
| Uneven load distribution | Round-robin not accounting for in-flight requests | Use weighted distribution based on current load |
| Key health false positive | Transient error marked key unhealthy | Use sliding window (3 errors in 60s) before marking unhealthy |
| Concurrent request failures | Too many parallel requests | Reduce semaphore limit; add backoff |

## Enterprise Considerations

- Create separate API keys per service/team with individual credit limits for cost isolation
- Use 3+ keys to multiply effective rate limits (each key gets its own quota)
- Implement circuit breakers: mark keys unhealthy after N consecutive errors, recover after cooldown
- Use `asyncio.Semaphore` to control concurrency and prevent overwhelming the API
- Monitor per-key error rates and latency to detect degraded keys early
- Combine multi-key rotation with provider routing for maximum resilience

## References

- Examples | Errors
- Rate Limits | [Provider Routing](https://openrouter.ai/docs/features/provider-routing)