3026 Scale 5375b413

# Scaling ContextForge

> Comprehensive guide to scaling ContextForge from development to production, covering vertical scaling, horizontal scaling, connection pooling, performance tuning, and Kubernetes deployment strategies.

## Overview

ContextForge is designed to scale from single-container development environments to distributed multi-node production deployments. For a visual overview of the high-performance architecture including Rust-powered components, see the [Performance Architecture Diagram](../architecture/performance-architecture.md).

This guide covers:

- **Vertical Scaling**: Optimizing single-instance performance with Gunicorn workers
- **Horizontal Scaling**: Multi-container deployments with shared state
- **Database Optimization**: PostgreSQL connection pooling, PgBouncer, and indexing
- **Cache Architecture**: Redis with hiredis, multi-level application caching
- **Performance Tuning**: orjson, compression, and configuration
- **Kubernetes Deployment**: HPA, resource limits, and best practices

---

## Table of Contents

1. [Understanding the GIL and Worker Architecture](#1-understanding-the-gil-and-worker-architecture)
2. [Vertical Scaling with Gunicorn](#2-vertical-scaling-with-gunicorn)
3. [Python 3.14 Free-Threading and PostgreSQL 18](#3-python-314-free-threading-and-postgresql-18)
4. [Horizontal Scaling with Kubernetes](#4-horizontal-scaling-with-kubernetes)
5. [Database Connection Pooling](#5-database-connection-pooling)
6. [Redis for Distributed Caching](#6-redis-for-distributed-caching)
7. [Performance Tuning](#7-performance-tuning)
8. [Benchmarking and Load Testing](#8-benchmarking-and-load-testing)
9. [Health Checks and Readiness](#9-health-checks-and-readiness)
10. [Stateless Architecture and Long-Running Connections](#10-stateless-architecture-and-long-running-connections)
11. [Kubernetes Production Deployment](#11-kubernetes-production-deployment)
12. [Monitoring and Observability](#12-monitoring-and-observability)

---

## 1. Understanding the GIL and Worker Architecture

### The Python Global Interpreter Lock (GIL)

Python's Global Interpreter Lock (GIL) prevents multiple native threads from executing Python bytecode simultaneously. This means:

- **Single worker** = Single CPU core usage (even on multi-core systems)
- **I/O-bound workloads** (API calls, database queries) benefit from async/await
- **CPU-bound workloads** (JSON parsing, encryption) require multiple processes

### Pydantic v2: Rust-Powered Performance

ContextForge leverages **Pydantic v2.11+** for all request/response validation and schema definitions. Unlike pure Python libraries, Pydantic v2 includes a **Rust-based core** (`pydantic-core`) that significantly improves performance:

**Performance benefits:**

- **5-50x faster validation** compared to Pydantic v1
- **JSON parsing** in Rust (bypasses GIL for serialization/deserialization)
- **Schema validation** runs in compiled Rust code
- **Reduced CPU overhead** for request processing

**Impact on scaling:**

- 5,463 lines of Pydantic schemas (`mcpgateway/schemas.py`)
- Every API request validated through Rust-optimized code
- Lower CPU usage per request = higher throughput per worker
- Rust components release the GIL during execution

This means that even within a single worker process, Pydantic's Rust core can run concurrently with Python code for validation-heavy workloads.

### ContextForge's Solution: Gunicorn with Multiple Workers

ContextForge uses **Gunicorn with UvicornWorker** to spawn multiple worker processes:

```python
# gunicorn.config.py
workers = 8                    # Multiple processes bypass the GIL
worker_class = "uvicorn.workers.UvicornWorker"  # Async support
timeout = 600                  # 10-minute timeout for long-running operations
preload_app = True            # Load app once, then fork (memory efficient)
```

**Key benefits:**

- Each worker is a separate process with its own GIL
- 8 workers = ability to use 8 CPU cores
- UvicornWorker enables async I/O within each worker
- Preloading reduces memory footprint (shared code segments)

The trade-off is that you are running multiple Python interpreter instances, and each consumes additional memory.

This also requires having shared state (e.g. Redis or a Database).
---

## 2. Vertical Scaling with Gunicorn

### Worker Count Calculation

**Formula**: `workers = (2 × CPU_cores) + 1`

**Examples:**

| CPU Cores | Recommended Workers | Use Case |
|-----------|---------------------|----------|
| 1 | 2-3 | Development/testing |
| 2 | 4-5 | Small production |
| 4 | 8-9 | Medium production |
| 8 | 16-17 | Large production |

### Configuration Methods

#### Environment Variables

```bash
# Automatic detection based on CPU cores
export GUNICORN_WORKERS=auto

# Manual override
export GUNICORN_WORKERS=16
export GUNICORN_TIMEOUT=600
export GUNICORN_MAX_REQUESTS=100000
export GUNICORN_MAX_REQUESTS_JITTER=100
export GUNICORN_PRELOAD_APP=true
```

#### Kubernetes ConfigMap

```yaml
# charts/mcp-stack/values.yaml
mcpContextForge:
  config:
    GUNICORN_WORKERS: "16"               # Number of worker processes
    GUNICORN_TIMEOUT: "600"              # Worker timeout (seconds)
    GUNICORN_MAX_REQUESTS: "100000"      # Requests before worker restart
    GUNICORN_MAX_REQUESTS_JITTER: "100"  # Prevents thundering herd
    GUNICORN_PRELOAD_APP: "true"         # Memory optimization
```

### Resource Allocation

**CPU**: Allocate 1 CPU core per 2 workers (allows for I/O wait)

**Memory**:

- Base: 256MB
- Per worker: 128-256MB (depending on workload)
- Formula: `memory = 256 + (workers × 200)` MB

**Example for 16 workers:**

- CPU: `8-10 cores` (allows headroom)
- Memory: `3.5-4 GB` (256 + 16×200 = 3.5GB)

```yaml
# Kubernetes resource limits
resources:
  limits:
    cpu: 10000m        # 10 cores
    memory: 4Gi
  requests:
    cpu: 8000m         # 8 cores
    memory: 3584Mi     # 3.5GB
```

---

## 3. Python 3.14 Free-Threading and PostgreSQL 18

### PostgreSQL 18 (Current)

**Status**: Production-ready - ContextForge's default Docker Compose configuration uses PostgreSQL 18.

PostgreSQL 18 provides significant performance improvements:

- **Improved async I/O**: Better non-blocking query performance
- **Reduced latency**: Optimized connection handling
- **Enhanced parallelism**: Better parallel query execution
- **Connection multiplexing**: More efficient connection reuse

**Docker Compose Configuration** (default):

```yaml
postgres:
  image: postgres:18
  command:
    - "postgres"
    - "-c"
    - "max_connections=500"       # With PgBouncer (4000 without)
    - "-c"
    - "shared_buffers=512MB"      # 25% of available RAM
    - "-c"
    - "work_mem=16MB"             # Per-operation memory
    - "-c"
    - "effective_cache_size=1536MB"  # 75% of RAM
    - "-c"
    - "maintenance_work_mem=128MB"
    - "-c"
    - "checkpoint_completion_target=0.9"
    - "-c"
    - "wal_buffers=16MB"
    - "-c"
    - "random_page_cost=1.1"      # SSD optimization
    - "-c"
    - "effective_io_concurrency=200"  # SSD parallel I/O
    - "-c"
    - "max_worker_processes=4"
    - "-c"
    - "max_parallel_workers_per_gather=2"
    - "-c"
    - "max_parallel_workers=4"
```

**Connection URL** (psycopg3 required):

```bash
# Via PgBouncer (recommended for high concurrency)
DATABASE_URL=postgresql+psycopg://postgres:password@pgbouncer:6432/mcp

# Direct connection (for development or low concurrency)
DATABASE_URL=postgresql+psycopg://postgres:password@postgres:5432/mcp
```

### Python 3.14 (Free-Threaded Mode)

**Status**: Beta (as of July 2025) - [PEP 703](https://peps.python.org/pep-0703/)

Python 3.14 introduces **optional free-threading** (GIL removal), a groundbreaking change that enables true parallel multi-threading:

```bash
# Enable free-threading mode
python3.14 -X gil=0 -m gunicorn ...

# Or use PYTHON_GIL environment variable
PYTHON_GIL=0 python3.14 -m gunicorn ...
```

**Performance characteristics:**

| Workload Type | Expected Impact |
|---------------|----------------|
| Single-threaded | **3-15% slower** (overhead from thread-safety mechanisms) |
| Multi-threaded (I/O-bound) | **Minimal impact** (already benefits from async/await) |
| Multi-threaded (CPU-bound) | **Near-linear scaling** with CPU cores |
| Multi-process (current) | **No change** (already bypasses GIL) |

**Benefits when available:**

- **True parallel threads**: Multiple threads execute Python code simultaneously
- **Lower memory overhead**: Threads share memory (vs. separate processes)
- **Faster inter-thread communication**: Shared memory, no IPC overhead
- **Better resource efficiency**: One interpreter instance instead of multiple processes

**Trade-offs:**

- **Single-threaded penalty**: 3-15% slower due to fine-grained locking
- **Library compatibility**: Some C extensions need updates (most popular libraries already compatible)
- **Different scaling model**: Move from `workers=16` to `workers=2 --threads=32`

**Migration strategy:**

1. **Now (Python 3.11-3.13)**: Continue using multi-process Gunicorn
   ```python
   workers = 16                    # Multiple processes
   worker_class = "uvicorn.workers.UvicornWorker"
   ```

2. **Python 3.14 beta**: Test in staging environment
   ```bash
   # Build free-threaded Python
   ./configure --enable-experimental-jit --with-pydebug
   make

   # Test with free-threading
   PYTHON_GIL=0 python3.14 -m pytest tests/
   ```

3. **Python 3.14 stable**: Evaluate hybrid approach
   ```python
   workers = 4                     # Fewer processes
   threads = 8                     # More threads per process
   worker_class = "uvicorn.workers.UvicornWorker"
   ```

4. **Post-migration**: Thread-based scaling
   ```python
   workers = 2                     # Minimal processes
   threads = 32                    # Scale with threads
   preload_app = True              # Single app load
   ```

**Current recommendation**:

- **Production**: Use Python 3.11-3.13 with multi-process Gunicorn (proven, stable)
- **Testing**: Experiment with Python 3.14 beta in non-production environments
- **Monitoring**: Watch for library compatibility announcements

**Why ContextForge is well-positioned for free-threading:**

ContextForge's architecture already benefits from components that will perform even better with Python 3.14:

1. **Pydantic v2 Rust core**: Already bypasses GIL for validation - will work seamlessly with free-threading
2. **FastAPI/Uvicorn**: Built for async I/O - natural fit for thread-based concurrency
3. **SQLAlchemy async**: Database operations already non-blocking
4. **Stateless design**: No shared mutable state between requests

**Resources:**

- [Python 3.14 Free-Threading Guide](https://www.pythoncheatsheet.org/blog/python-3-14-breaking-free-from-gil)
- [PEP 703: Making the GIL Optional](https://peps.python.org/pep-0703/)
- [Python 3.14 Release Schedule](https://peps.python.org/pep-0745/)
- [Pydantic v2 Performance](https://docs.pydantic.dev/latest/blog/pydantic-v2/)

---

## 4. Horizontal Scaling with Kubernetes

### Architecture Overview

```
+------------------------------------------------------------------------------+
|                              Load Balancer                                    |
|                        (Kubernetes Ingress / Service)                         |
+----------------------------------+-------------------------------------------+
                                   |
+----------------------------------v-------------------------------------------+
|                           Nginx Caching Layer                                 |
|   +---------------------+                  +---------------------+            |
|   |   Nginx Cache 1     |                  |   Nginx Cache 2     |            |
|   | - Brotli/Gzip/Zstd  |                  | - Brotli/Gzip/Zstd  |            |
|   | - Static caching    |                  | - Static caching    |            |
|   | - Rate limiting     |                  | - Rate limiting     |            |
|   +----------+----------+                  +----------+----------+            |
+--------------|-----------------------------------------|---------------------+
               |                                         |
+--------------v-----------------------------------------v---------------------+
|                         Gateway Application Layer                             |
|  +------------------+  +------------------+  +------------------+             |
|  |  Gateway Pod 1   |  |  Gateway Pod 2   |  |  Gateway Pod N   |             |
|  |  (16 workers)    |  |  (16 workers)    |  |  (16 workers)    |             |
|  | Gunicorn/Granian |  | Gunicorn/Granian |  | Gunicorn/Granian |             |
|  | orjson, psycopg3 |  | orjson, psycopg3 |  | orjson, psycopg3 |             |
|  +--------+---------+  +--------+---------+  +--------+---------+             |
+-----------|-----------------------|----------------------|-------------------+
            |                       |                      |
            +-----------+-----------+-----------+----------+
                        |                       |
+-------------------------------------------------------------------+
|                           Data Layer                               |
|                                                                    |
|  +---------------------------+    +---------------------------+   |
|  |        PgBouncer          |    |          Redis            |   |
|  | - Connection multiplexing |    | - Distributed cache       |   |
|  | - Transaction pooling     |    | - Session storage         |   |
|  | - 3000 client connections |    | - hiredis parser (83x)    |   |
|  | - 200 server connections  |    | - Leader election         |   |
|  +-------------+-------------+    +---------------------------+   |
|                |                                                   |
|  +-------------v-------------+                                    |
|  |      PostgreSQL 18        |                                    |
|  | - Async I/O               |                                    |
|  | - Auto-prepared stmts     |                                    |
|  | - 500 max_connections     |                                    |
|  | - Parallel query exec     |                                    |
|  +---------------------------+                                    |
+-------------------------------------------------------------------+
```

**Layer Summary:**

| Layer | Component | Purpose | Key Performance Features |
|-------|-----------|---------|--------------------------|
| Edge | Load Balancer | Traffic distribution | SSL termination, health checks |
| Proxy | Nginx | Caching, compression | Brotli/Gzip/Zstd (30-70% bandwidth reduction) |
| App | Gateway Pods | Request processing | Gunicorn/Granian, orjson, multi-level caching |
| Pool | PgBouncer | Connection multiplexing | 3000 client → 200 server connections |
| Cache | Redis | Distributed state | hiredis parser (up to 83x faster) |
| DB | PostgreSQL 18 | Persistent storage | psycopg3 COPY/pipeline, async I/O |

### Shared State Requirements

For multi-pod deployments:

1. **Shared PostgreSQL 18**: All persistent data (servers, tools, users, teams)
2. **PgBouncer**: Connection pooling and multiplexing between gateway pods and PostgreSQL
3. **Shared Redis**: Distributed caching, session storage, and leader election
4. **Stateless pods**: No local state, can be killed/restarted anytime

### Kubernetes Deployment

#### Helm Chart Configuration

```yaml
# charts/mcp-stack/values.yaml
mcpContextForge:
  replicaCount: 3                   # Start with 3 pods

  # Horizontal Pod Autoscaler
  hpa:
    enabled: true
    minReplicas: 3                  # Never scale below 3
    maxReplicas: 20                 # Scale up to 20 pods
    targetCPUUtilizationPercentage: 70    # Scale at 70% CPU
    targetMemoryUtilizationPercentage: 80 # Scale at 80% memory

  # Pod resources
  resources:
    limits:
      cpu: 2000m                    # 2 cores per pod
      memory: 4Gi
    requests:
      cpu: 1000m                    # 1 core per pod
      memory: 2Gi

  # Environment configuration
  config:
    GUNICORN_WORKERS: "8"           # 8 workers per pod
    CACHE_TYPE: redis               # Shared cache
    DB_POOL_SIZE: "50"              # Per-pod pool size

# Shared PostgreSQL
postgres:
  enabled: true
  resources:
    limits:
      cpu: 4000m                    # 4 cores
      memory: 8Gi
    requests:
      cpu: 2000m
      memory: 4Gi

  # Important: Set max_connections
  # Formula: (num_pods × DB_POOL_SIZE × 1.2) + 20
  # Example: (20 pods × 50 pool × 1.2) + 20 = 1220
  config:
    max_connections: 1500           # Adjust based on scale

# Shared Redis
redis:
  enabled: true
  resources:
    limits:
      cpu: 2000m
      memory: 4Gi
    requests:
      cpu: 1000m
      memory: 2Gi
```

#### Deploy with Helm

```bash
# Install/upgrade with custom values
helm upgrade --install mcp-stack ./charts/mcp-stack \
  --namespace mcp-gateway \
  --create-namespace \
  --values production-values.yaml

# Verify HPA
kubectl get hpa -n mcp-gateway
```

### Horizontal Scaling Calculation

**Total capacity** = `pods × workers × requests_per_second`

**Example:**

- 10 pods × 8 workers × 100 RPS = **8,000 RPS**

**Database connections needed:**

- 10 pods × 50 pool size = **500 connections**
- Add 20% overhead = **600 connections**
- Set `max_connections=1000` (buffer for maintenance)

### Docker Compose Reference Architecture

The `docker-compose.yml` provides a production-ready reference architecture with all performance optimizations pre-configured:

```yaml
services:
  # Nginx caching proxy (port 8080)
  nginx:
    image: mcpgateway/nginx-cache:latest
    ports: ["8080:80"]
    volumes:
      - nginx_cache:/var/cache/nginx
    # Brotli/Gzip compression, static caching, rate limiting

  # Gateway application (replicas: 2)
  gateway:
    image: mcpgateway/mcpgateway:latest
    environment:
      # HTTP Server: gunicorn (stable) or granian (faster)
      - HTTP_SERVER=gunicorn
      - GUNICORN_WORKERS=16

      # Database: via PgBouncer
      - DATABASE_URL=postgresql+psycopg://postgres:password@pgbouncer:6432/mcp
      - DB_POOL_SIZE=15              # Smaller with PgBouncer
      - DB_MAX_OVERFLOW=30

      # Redis with hiredis
      - CACHE_TYPE=redis
      - REDIS_URL=redis://redis:6379/0
      - REDIS_PARSER=hiredis
      - REDIS_MAX_CONNECTIONS=150

      # Multi-level caching
      - AUTH_CACHE_ENABLED=true
      - REGISTRY_CACHE_ENABLED=true
      - ADMIN_STATS_CACHE_ENABLED=true

      # Performance: disable overhead
      - LOG_LEVEL=ERROR
      - DISABLE_ACCESS_LOG=true
      - AUDIT_TRAIL_ENABLED=false
      - COMPRESSION_ENABLED=false    # Nginx handles this
    deploy:
      replicas: 2
      resources:
        limits: { cpus: '8', memory: 8G }

  # PgBouncer connection pooler
  pgbouncer:
    image: edoburu/pgbouncer:latest
    environment:
      - DATABASE_URL=postgres://postgres:password@postgres:5432/mcp
      - POOL_MODE=transaction
      - MAX_CLIENT_CONN=3000
      - DEFAULT_POOL_SIZE=120
      - MAX_DB_CONNECTIONS=200

  # PostgreSQL 18
  postgres:
    image: postgres:18
    command:
      - "postgres"
      - "-c" - "max_connections=500"
      - "-c" - "shared_buffers=512MB"
      - "-c" - "effective_cache_size=1536MB"
      - "-c" - "random_page_cost=1.1"
      - "-c" - "effective_io_concurrency=200"

  # Redis with performance tuning
  redis:
    image: redis:latest
    command:
      - "redis-server"
      - "--maxmemory" - "1gb"
      - "--maxmemory-policy" - "allkeys-lru"
      - "--tcp-backlog" - "2048"
      - "--maxclients" - "10000"
```

**Access Points:**

| Port | Service | Use |
|------|---------|-----|
| 8080 | Nginx | Production access (caching, compression) |
| 4444 | Gateway | Direct access (debugging, internal) |
| 6432 | PgBouncer | Database connection pooling |
| 5433 | PostgreSQL | Direct DB access (admin) |
| 6379 | Redis | Cache access |

**Quick Start:**

```bash
# Start full stack
docker-compose up -d

# Access via caching proxy
curl http://localhost:8080/health

# View logs
docker-compose logs -f gateway
```

---

## 5. Database Connection Pooling

### Connection Pool Architecture

ContextForge uses a two-tier connection pooling architecture:

**Without PgBouncer** (direct connection):
```
+-------------------+     +-------------------+     +-------------------+
| Pod 1 (16 workers)|     | Pod 2 (16 workers)|     | Pod N (16 workers)|
| 16 SQLAlchemy     |     | 16 SQLAlchemy     |     | 16 SQLAlchemy     |
| pools × 50 conns  |     | pools × 50 conns  |     | pools × 50 conns  |
+--------+----------+     +--------+----------+     +--------+----------+
         |                         |                         |
         +------------+------------+------------+------------+
                      |
         +------------v------------+
         |      PostgreSQL 18      |
         | max_connections = 4000  |
         +-------------------------+
```

**With PgBouncer** (recommended for high concurrency):
```
+-------------------+     +-------------------+     +-------------------+
| Pod 1 (16 workers)|     | Pod 2 (16 workers)|     | Pod N (16 workers)|
| 16 SQLAlchemy     |     | 16 SQLAlchemy     |     | 16 SQLAlchemy     |
| pools × 15 conns  |     | pools × 15 conns  |     | pools × 15 conns  |
+--------+----------+     +--------+----------+     +--------+----------+
         |                         |                         |
         +------------+------------+------------+------------+
                      |
         +------------v------------+
         |       PgBouncer         |
         | MAX_CLIENT_CONN = 3000  |  <-- Application connections
         | DEFAULT_POOL_SIZE = 120 |
         | MAX_DB_CONNECTIONS = 200|  <-- PostgreSQL connections
         +------------+------------+
                      |
         +------------v------------+
         |      PostgreSQL 18      |
         | max_connections = 500   |  <-- Much lower requirement
         +-------------------------+
```

**Connection Multiplexing Benefits**:

| Metric | Without PgBouncer | With PgBouncer | Improvement |
|--------|-------------------|----------------|-------------|
| App connections | 2 pods × 16 × 50 = 1,600 | 2 pods × 16 × 15 = 480 | App-level reduction |
| PostgreSQL connections | 1,600+ | 200 | **8x reduction** |
| `max_connections` needed | 4,000 | 500 | **8x lower** |
| Memory per connection | ~10MB | ~10MB | Same |
| PostgreSQL memory | ~40GB | ~5GB | **8x reduction** |
| Connection setup time | Per request | Reused | Near-zero |

### Pool Configuration

#### Environment Variables

```bash
# Connection pool settings
DB_POOL_SIZE=50              # Persistent connections per worker
DB_MAX_OVERFLOW=10           # Additional connections allowed
DB_POOL_TIMEOUT=60           # Wait time before timeout (seconds)
DB_POOL_RECYCLE=3600         # Recycle connections after 1 hour
DB_MAX_RETRIES=30            # Retry attempts on failure (exponential backoff)
DB_RETRY_INTERVAL_MS=2000    # Base retry interval (doubles each attempt, max 30s)

# psycopg3-specific optimizations
DB_PREPARE_THRESHOLD=5       # Auto-prepare queries after N executions (0=disable)
```

#### psycopg3 Performance Features

ContextForge uses **psycopg3** (`psycopg[c,binary]`) instead of psycopg2, providing significant performance improvements and modern features.

**Why psycopg3:**

| Feature | psycopg2 | psycopg3 |
|---------|----------|----------|
| Parameter binding | Client-side | Server-side (more secure) |
| Prepared statements | Manual | Automatic after N executions |
| Binary protocol | Optional | Native support |
| Async support | Wrapper | First-class built-in |
| Maintenance | Maintenance mode | Active development |

**Performance Benchmarks:**

| Operation | psycopg2 | psycopg3 | Improvement |
|-----------|----------|----------|-------------|
| Bulk INSERT (1000+ rows) | Standard | COPY protocol | 5-10x |
| Repeated queries | Parsed each time | Auto-prepared | 2-3x |
| Batch queries | Sequential | Pipelined | 2-5x |

**Automatic Prepared Statements:**

```bash
# Number of executions before auto-preparing a query server-side
# Default: 5 (balance between memory and performance)
# Set to 0 to disable, 1 to prepare immediately
DB_PREPARE_THRESHOLD=5
```

After N executions of the same query pattern, psycopg3 creates a server-side prepared statement, reducing:

- Query parsing overhead (5-15% per query)
- Network round-trips for query plans
- PostgreSQL CPU usage for repeated queries

**COPY Protocol for Bulk Inserts:**

The utility module `mcpgateway/utils/psycopg3_optimizations.py` provides COPY protocol support:

```python
from mcpgateway.utils.psycopg3_optimizations import bulk_insert_with_copy

# 5-10x faster for 1000+ rows
bulk_insert_with_copy(db, "tool_metrics", columns, rows)
```

Note: COPY is only faster for large batches (1000+ rows); for small batches (<100 rows), SQLAlchemy's `bulk_insert_mappings()` is actually faster due to lower protocol overhead.

**Pipeline Mode for Batch Queries:**

Execute multiple queries with reduced round-trips:

```python
from mcpgateway.utils.psycopg3_optimizations import execute_pipelined

# Send multiple queries without waiting for individual responses
results = execute_pipelined(db, [
    ("SELECT * FROM tools WHERE id = %s", {"id": tool_id}),
    ("SELECT * FROM gateways WHERE id = %s", {"id": gateway_id}),
])
```

**Connection URL Format:**

```bash
# IMPORTANT: Required format for psycopg3 (NOT postgresql://)
DATABASE_URL=postgresql+psycopg://user:pass@host:5432/db
```

See [ADR-027: Migrate to psycopg3](../architecture/adr/027-migrate-psycopg3.md) for details.

#### Configuration in Code

```python
# mcpgateway/config.py
@property
def database_settings(self) -> dict:
    return {
        "pool_size": self.db_pool_size,          # 50
        "max_overflow": self.db_max_overflow,    # 10
        "pool_timeout": self.db_pool_timeout,    # 60s
        "pool_recycle": self.db_pool_recycle,    # 3600s
    }
```

### PostgreSQL Configuration

#### Calculate max_connections

```bash
# Formula
max_connections = (num_pods × num_workers × pool_size × 1.2) + buffer

# Example: 10 pods, 8 workers, 50 pool size
max_connections = (10 × 8 × 50 × 1.2) + 200 = 5000 connections
```

#### PostgreSQL Configuration File

```ini
# postgresql.conf
max_connections = 5000
shared_buffers = 16GB              # 25% of RAM
effective_cache_size = 48GB        # 75% of RAM
work_mem = 16MB                    # Per operation
maintenance_work_mem = 2GB
```

#### Managed Services

**IBM Cloud Databases for PostgreSQL:**
```bash
# Increase max_connections via CLI
ibmcloud cdb deployment-configuration postgres \
  --configuration max_connections=5000
```

**AWS RDS:**
```bash
# Via parameter group
max_connections = {DBInstanceClassMemory/9531392}
```

**Google Cloud SQL:**
```bash
# Auto-scales based on instance size
# 4 vCPU = 400 connections
# 8 vCPU = 800 connections
```

### PgBouncer Connection Pooling

For high-concurrency deployments, PgBouncer provides connection multiplexing between the gateway and PostgreSQL, dramatically reducing connection overhead.

**Architecture:**

```
Without PgBouncer:
  Gateway (2 replicas × 16 workers) → PostgreSQL (max_connections=4000)
  Each worker maintains its own pool → High connection churn

With PgBouncer:
  Gateway (2 replicas × 16 workers) → PgBouncer → PostgreSQL (max_connections=500)
  Connections multiplexed → Efficient reuse, lower PostgreSQL overhead
```

**Benefits:**

- **Connection multiplexing**: Many app connections share fewer database connections
- **Reduced PostgreSQL overhead**: Lower `max_connections` reduces memory per connection
- **Connection reuse**: PgBouncer maintains persistent connections to PostgreSQL
- **Graceful handling of connection storms**: Queues requests instead of rejecting

**Docker Compose Setup:**

```yaml
pgbouncer:
  image: edoburu/pgbouncer:latest
  restart: unless-stopped
  ports:
    - "6432:6432"
  environment:
    - DATABASE_URL=postgres://postgres:password@postgres:5432/mcp
    - POOL_MODE=transaction
    - MAX_CLIENT_CONN=2000
    - DEFAULT_POOL_SIZE=100
    - MIN_POOL_SIZE=10
    - RESERVE_POOL_SIZE=25
    - MAX_DB_CONNECTIONS=200
    - SERVER_LIFETIME=3600
    - SERVER_IDLE_TIMEOUT=600
    - AUTH_TYPE=scram-sha-256
  depends_on:
    postgres:
      condition: service_healthy
  healthcheck:
    test: ["CMD", "pg_isready", "-h", "localhost", "-p", "6432"]
    interval: 10s
    timeout: 5s
    retries: 3
```

**Gateway Configuration with PgBouncer:**

```bash
# Connect via PgBouncer instead of direct PostgreSQL
DATABASE_URL=postgresql+psycopg://postgres:password@pgbouncer:6432/mcp

# Smaller pool since PgBouncer handles pooling
DB_POOL_SIZE=10
DB_MAX_OVERFLOW=20
```

**Pool Modes:**

| Mode | Description | Use Case |
|------|-------------|----------|
| `transaction` | Connection returned after transaction commit | **Recommended** for most workloads |
| `session` | Connection held for entire session | Legacy apps requiring session state |
| `statement` | Connection returned after each statement | Limited use cases |

**Key Tuning Parameters:**

| Parameter | Description | Suggested Value |
|-----------|-------------|-----------------|
| `MAX_CLIENT_CONN` | Max app connections | 2000-5000 |
| `DEFAULT_POOL_SIZE` | Connections per user/db pair | 100 |
| `MAX_DB_CONNECTIONS` | Max connections to PostgreSQL | 200-500 |
| `POOL_MODE` | When to return connections | `transaction` |

**Monitoring PgBouncer:**

```bash
# Connect to PgBouncer admin console
psql -h localhost -p 6432 -U pgbouncer pgbouncer

# Show connection pool statistics
SHOW STATS;
SHOW POOLS;
SHOW CLIENTS;
SHOW SERVERS;
```

### Connection Pool Monitoring

```python
# Health endpoint checks pool status
@app.get("/health")
async def healthcheck(db: Session = Depends(get_db)):
    try:
        db.execute(text("SELECT 1"))
        return {"status": "healthy"}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}
```

```bash
# Check PostgreSQL connections
kubectl exec -it postgres-pod -- psql -U admin -d postgresdb \
  -c "SELECT count(*) FROM pg_stat_activity;"
```

### Database Session Management

To prevent connection pool exhaustion under high load, ContextForge releases database sessions before making upstream HTTP calls:

**Problem:** Database sessions held during slow upstream calls (100ms - 4+ minutes) exhaust the connection pool even when the database is lightly loaded.

**Solution:** "Fetch-Then-Release" pattern:

1. **Eager load** required data with `joinedload()` in single query
2. **Extract data** to local variables before network I/O
3. **Release session** with `db.expunge()` + `db.close()` before HTTP calls
4. **Fresh session** for metrics recording after the call

This reduces connection hold time from minutes to <50ms and enables 10x+ higher concurrency.

---

## 6. Redis for Distributed Caching

### Architecture

Redis provides shared state across all Gateway pods:

- **Session storage**: User sessions (TTL: 3600s)
- **Message cache**: Ephemeral data (TTL: 600s)
- **Federation cache**: Gateway peer discovery

### Configuration

#### Enable Redis Caching

```bash
# .env or Kubernetes ConfigMap
CACHE_TYPE=redis
REDIS_URL=redis://redis-service:6379/0
CACHE_PREFIX=mcpgw:
SESSION_TTL=3600
MESSAGE_TTL=600
REDIS_MAX_RETRIES=30             # Retry attempts on failure (exponential backoff)
REDIS_RETRY_INTERVAL_MS=2000     # Base retry interval (doubles each attempt, max 30s)

# Connection pool (standard)
REDIS_MAX_CONNECTIONS=50
REDIS_SOCKET_TIMEOUT=2.0
REDIS_SOCKET_CONNECT_TIMEOUT=2.0
REDIS_RETRY_ON_TIMEOUT=true
REDIS_HEALTH_CHECK_INTERVAL=30

# Leader election (multi-node)
REDIS_LEADER_TTL=15
REDIS_LEADER_HEARTBEAT_INTERVAL=5
```

#### High-Concurrency Redis Tuning

For 1000+ concurrent users:

```bash
# Connection pool - Formula: (concurrent_requests / workers) * 1.5
# Example: 32 workers × 150 = 4800 < Redis maxclients (10000)
REDIS_MAX_CONNECTIONS=150

# Timeouts - keep low for fast failure detection
REDIS_SOCKET_TIMEOUT=5.0
REDIS_SOCKET_CONNECT_TIMEOUT=5.0
REDIS_HEALTH_CHECK_INTERVAL=30
```

**Redis Server Tuning** (docker-compose.yml):

```yaml
redis:
  command:
    - "redis-server"
    - "--maxmemory"
    - "1gb"
    - "--maxmemory-policy"
    - "allkeys-lru"
    - "--tcp-backlog"
    - "2048"                    # Higher for pending connections
    - "--maxclients"
    - "10000"                   # Max client connections
```

#### Kubernetes Deployment

```yaml
# charts/mcp-stack/values.yaml
redis:
  enabled: true

  resources:
    limits:
      cpu: 2000m
      memory: 4Gi
    requests:
      cpu: 1000m
      memory: 2Gi

  # Enable persistence
  persistence:
    enabled: true
    size: 10Gi
```

### Redis Sizing

**Memory calculation:**

- Sessions: `concurrent_users × 50KB`
- Messages: `messages_per_minute × 100KB × (TTL/60)`

**Example:**

- 10,000 users × 50KB = 500MB
- 1,000 msg/min × 100KB × 10min = 1GB
- **Total: 1.5GB + 50% overhead = 2.5GB**

**Connection pool sizing:**

- Formula: `REDIS_MAX_CONNECTIONS = (concurrent_requests / workers) × 1.5`
- Default 50 handles ~500 concurrent requests with 10 workers
- High-concurrency: increase to 100 and lower timeouts

```bash
# High-concurrency production overrides
REDIS_MAX_CONNECTIONS=100
REDIS_SOCKET_TIMEOUT=1.0
REDIS_SOCKET_CONNECT_TIMEOUT=1.0
REDIS_HEALTH_CHECK_INTERVAL=15
REDIS_LEADER_TTL=10
REDIS_LEADER_HEARTBEAT_INTERVAL=3
```

### Hiredis High-Performance Parser

ContextForge uses `redis[hiredis]` for significantly faster Redis protocol parsing, especially beneficial for large responses.

**Performance Impact:**

| Operation | Pure Python | With Hiredis | Improvement |
|-----------|-------------|--------------|-------------|
| Simple SET/GET | Baseline | +10% | 1.1x |
| LRANGE (10 items) | Baseline | +170% | 2.7x |
| LRANGE (100 items) | Baseline | +1000% | ~10x |
| LRANGE (999 items) | Baseline | +8220% | **83x** |

The larger the response, the greater the improvement. This is critical for:

- Tool registry queries returning many tools
- Bulk operations and federation
- Cached response retrieval
- Metrics aggregation

**Configuration:**

```bash
# Redis parser selection (hiredis is default for performance)
# Options: auto (default - uses hiredis if available), hiredis, python
# Use "python" to force pure-Python parser (useful for debugging)
REDIS_PARSER=auto
```

**When to use each parser:**

| Parser | Use Case |
|--------|----------|
| `hiredis` (default) | Production, high throughput |
| `python` | Debugging Redis protocol issues, restricted environments |

### High Availability

**Redis Sentinel** (3+ nodes):
```yaml
redis:
  sentinel:
    enabled: true
    quorum: 2

  replicas: 3  # 1 primary + 2 replicas
```

**Redis Cluster** (6+ nodes):
```bash
REDIS_URL=redis://redis-cluster:6379/0?cluster=true
```

---

## 7. Performance Tuning

### Application Architecture Performance

ContextForge's technology stack is optimized for high performance:

**Rust-Powered Components:**

- **Pydantic v2** (5-50x faster validation via Rust core)
- **Uvicorn with [standard] extras** (ASGI server with high-performance components)

**Async-First Design:**

- **FastAPI** (async request handling)
- **SQLAlchemy 2.0** (async database operations)
- **asyncio** event loop per worker

**Performance characteristics:**

- Request validation: **< 1ms** (Pydantic v2 Rust core)
- JSON serialization: **3-5x faster** than pure Python
- Database queries: Non-blocking async I/O
- Concurrent requests per worker: **1000+** (async event loop)

### System-Level Optimization

#### Kernel Parameters

```bash
# /etc/sysctl.conf
net.core.somaxconn=4096
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_tw_reuse=1
fs.file-max=2097152

# Apply changes
sysctl -p
```

#### File Descriptors

```bash
# /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576

# Verify
ulimit -n
```

### HTTP Server Selection

ContextForge supports two production HTTP servers:

| Server | Description | Best For |
|--------|-------------|----------|
| **Gunicorn** (default) | Python-based with Uvicorn workers | Stable, well-tested |
| **Granian** | Rust-based HTTP server | Maximum performance (+20-50%) |

```bash
# Select HTTP server (in containers)
HTTP_SERVER=gunicorn    # Default, stable
HTTP_SERVER=granian     # Alternative, Rust-based
```

### Gunicorn Configuration

```bash
# Number of worker processes
# Options: "auto" (default, 2*CPU+1 capped at 16), or positive integer
GUNICORN_WORKERS=auto

# Worker timeout in seconds (increase for long-running LLM requests)
GUNICORN_TIMEOUT=600

# Maximum requests per worker before automatic restart (prevents memory leaks)
GUNICORN_MAX_REQUESTS=100000

# Random jitter added to max requests (prevents thundering herd)
GUNICORN_MAX_REQUESTS_JITTER=100

# Preload application before forking workers
# true: Saves memory (shared code), runs migrations once before forking
# false: Each worker loads app independently (more memory, better isolation)
GUNICORN_PRELOAD_APP=true

# Development mode with hot reload (NOT for production)
GUNICORN_DEV_MODE=false
```

**Worker Class**: UvicornWorker (default) for async support.

### Granian Configuration (Alternative)

Granian is a Rust-based HTTP server with native backpressure for overload protection:

```bash
# HTTP server selection
HTTP_SERVER=granian

# Worker count (auto = CPU count, max 16)
GRANIAN_WORKERS=16

# TCP backlog for pending connections
GRANIAN_BACKLOG=4096

# Backpressure: max concurrent requests per worker before 503 rejection
# Total capacity = WORKERS × BACKPRESSURE = 16 × 64 = 1024 concurrent requests
GRANIAN_BACKPRESSURE=64

# HTTP/1.1 buffer size (bytes)
GRANIAN_HTTP1_BUFFER_SIZE=524288

# Auto-restart failed workers
GRANIAN_RESPAWN_FAILED=true
```

**Backpressure behavior:**

- Requests within capacity (≤1024): Processed normally
- Requests over capacity (>1024): Immediate 503 Service Unavailable
- No queuing, no memory growth, no cascading timeouts

**When to use Granian:**

- Load spike protection (backpressure rejects excess gracefully)
- Bursty or unpredictable traffic patterns
- High-concurrency deployments (1000+ concurrent users)

**When to use Gunicorn:**

- Memory-constrained environments (32% less RAM)
- Maximum stability and compatibility
- Standard deployments with predictable traffic

### Application Tuning

```bash
# Resource limits
TOOL_TIMEOUT=60
TOOL_CONCURRENT_LIMIT=10
RESOURCE_CACHE_SIZE=1000
RESOURCE_CACHE_TTL=3600

# Retry configuration
RETRY_MAX_ATTEMPTS=3
RETRY_BASE_DELAY=1.0
RETRY_MAX_DELAY=60

# Health check intervals
HEALTH_CHECK_INTERVAL=60
HEALTH_CHECK_TIMEOUT=5
UNHEALTHY_THRESHOLD=3

# Gateway health check timeout (seconds)
GATEWAY_HEALTH_CHECK_TIMEOUT=5.0

# Auto-refresh tools during health checks
# When enabled, tools/resources/prompts are fetched and synced during health checks
AUTO_REFRESH_SERVERS=false

```

### Logging for Performance

Logging can significantly impact performance under high load:

```bash
# Log level - ERROR recommended for production
# DEBUG/INFO create massive I/O overhead
LOG_LEVEL=ERROR

# Disable access logging (massive I/O overhead under high concurrency)
DISABLE_ACCESS_LOG=true

# Disable database logging for performance
STRUCTURED_LOGGING_DATABASE_ENABLED=false
```

**Impact of logging settings:**

| Setting | I/O Overhead | Use Case |
|---------|--------------|----------|
| `LOG_LEVEL=DEBUG` | Very High | Development only |
| `LOG_LEVEL=INFO` | High | Light load, debugging |
| `LOG_LEVEL=ERROR` | Low | Production (recommended) |
| `DISABLE_ACCESS_LOG=true` | None | Production (recommended) |
| `STRUCTURED_LOGGING_DATABASE_ENABLED=true` | Very High | Compliance (use external aggregator) |

### Metrics Buffer Configuration

Batch metric writes to reduce database pressure:

```bash
# Enable buffered metrics writes (default: true)
METRICS_BUFFER_ENABLED=true

# Flush interval in seconds (default: 60, range: 5-300)
METRICS_BUFFER_FLUSH_INTERVAL=60

# Max buffered metrics before forced flush (default: 1000)
METRICS_BUFFER_MAX_SIZE=1000
```

### Metrics Cache Configuration

Cache aggregate metrics queries to reduce full table scans (see [Issue #1906](https://github.com/IBM/mcp-context-forge/issues/1906)):

```bash
# Enable metrics query caching (default: true)
METRICS_CACHE_ENABLED=true

# TTL for cached metrics in seconds (default: 60, recommended: 60-300)
# Higher values significantly reduce database load under high traffic
METRICS_CACHE_TTL_SECONDS=60
```

### Application-Level Caching

ContextForge implements multi-level caching to reduce database load by 80-95%:

#### JWT Token Cache

Cache JWT token verification results to reduce auth overhead per request:

```bash
# JWT caching (default: enabled)
JWT_CACHE_ENABLED=true
JWT_CACHE_TTL=30              # Seconds (short TTL for security)
JWT_CACHE_MAX_SIZE=10000      # Max cached tokens
```

**Impact:**

- Auth overhead: 5-12ms → <1ms (with cache hit)
- Cache hit rate: >80% under normal load

#### Authentication Cache

Cache user lookups, token revocation checks, team membership, and role assignments:

```bash
# Auth Cache Configuration (reduces DB queries per auth from 3-4 to 0-1)
AUTH_CACHE_ENABLED=true
AUTH_CACHE_USER_TTL=60        # User lookup cache (seconds)
AUTH_CACHE_REVOCATION_TTL=30  # Token revocation check cache
AUTH_CACHE_TEAM_TTL=60        # Team membership cache
AUTH_CACHE_ROLE_TTL=60        # Role assignment cache
AUTH_CACHE_TEAMS_ENABLED=true # User teams list cache (get_user_teams, called 20+ times per request)
AUTH_CACHE_TEAMS_TTL=60       # Teams list cache TTL
AUTH_CACHE_BATCH_QUERIES=true # Batch related queries together
```

**Impact:**

- Auth database queries: 3-4 per request → 0-1 per request
- Auth latency: 8-15ms → 1-3ms
- Database load: 75% reduction for auth operations
- "Idle in transaction" connections: 50-70% reduction under high load (3000+ users)

#### GlobalConfig Cache

In-memory cache for GlobalConfig lookups (passthrough headers configuration):

```bash
# GlobalConfig cache TTL (default: 60 seconds)
GLOBAL_CONFIG_CACHE_TTL=60
```

**Impact:**

- Eliminates 42,000+ database queries per load test
- Query latency: ~1ms → ~0.00001ms

#### Registry & Admin Stats Cache

Distributed caching for registry listings and admin dashboard:

```bash
# Registry Cache Configuration
REGISTRY_CACHE_ENABLED=true
REGISTRY_CACHE_TOOLS_TTL=20
REGISTRY_CACHE_PROMPTS_TTL=15
REGISTRY_CACHE_RESOURCES_TTL=15
REGISTRY_CACHE_AGENTS_TTL=20
REGISTRY_CACHE_SERVERS_TTL=20
REGISTRY_CACHE_GATEWAYS_TTL=20

# Admin Stats Cache Configuration
ADMIN_STATS_CACHE_ENABLED=true
ADMIN_STATS_CACHE_SYSTEM_TTL=60
ADMIN_STATS_CACHE_OBSERVABILITY_TTL=30

# Team Cache Configuration
TEAM_CACHE_ENABLED=true
TEAM_CACHE_MEMBERS_TTL=60
TEAM_CACHE_ROLE_TTL=60
```

**Impact:**

- Dashboard load: 15-20 queries → 0-2 queries (95%+ cache hit)
- List endpoints: 50-200 queries → 0-1 queries per request
- Database load reduction: 80-95%

### High-Performance JSON Serialization

ContextForge uses **orjson** for JSON operations, providing 2-3x faster serialization:

```bash
# orjson is used by default via ORJSONResponse
# No configuration needed
```

**Performance comparison:**

| Operation | stdlib json | orjson | Improvement |
|-----------|-------------|--------|-------------|
| `json.dumps()` | ~15μs | ~5μs | 3x faster |
| `json.loads()` | ~12μs | ~4μs | 3x faster |

All SSE streaming, WebSocket messages, Redis pub/sub, and message parsing use orjson for maximum throughput.

### Database Indexing

Comprehensive database indexing provides 10-100x query performance improvement:

**Key indexed patterns:**

- Foreign key columns for efficient JOINs
- Composite indexes for common filter patterns (e.g., `team_id, enabled`)
- Boolean filter columns (`is_active`, `enabled`, `status`)
- Timestamp columns for ORDER BY (pagination)
- Unique lookups (`email`, `slug`, `token`)

**Impact:**

- Query time: seconds → milliseconds for filtered queries
- Database CPU: 30-60% reduction
- Scalability: Support 10x more concurrent users

### Audit Trail Toggle

For load testing or development, disable audit trail logging to prevent database growth:

```bash
# Disabled for load testing / development (default)
AUDIT_TRAIL_ENABLED=false

# Enabled for production compliance (SOC2, HIPAA, GDPR)
AUDIT_TRAIL_ENABLED=true
```

**Impact (under load with 2000 concurrent users):**

- Disabled: 0 rows/hour, 0 writes/request
- Enabled: ~140,000+ rows/hour, 1 write/request

**Note:** Audit trail is separate from `SECURITY_LOGGING_ENABLED` (which controls `security_events` table).

### Nginx Caching Proxy (CDN-like Performance)

**Overview**: Deploy an nginx reverse proxy with intelligent caching to dramatically reduce backend load and improve response times.

ContextForge includes a production-ready nginx caching proxy configuration (`nginx/`) with three dedicated cache zones:

1. **Static Assets Cache** (1GB, 30-day TTL): CSS, JS, images, fonts
2. **API Response Cache** (512MB, 5-minute TTL): Read-only endpoints
3. **Schema Cache** (256MB, 24-hour TTL): OpenAPI specs, docs

**Performance Benefits**:

- **Static assets**: 80-95% faster (20-50ms → 1-5ms)
- **API endpoints**: 60-80% faster with cache hits
- **Backend load**: 60-80% reduction in requests
- **Cache hit rates**: 40-99% depending on endpoint type
- **Database pressure**: 40-70% fewer queries

**Docker Compose Setup**:

```bash
# Start with nginx caching proxy
docker-compose up -d nginx

# Access via caching proxy
curl http://localhost:8080/health  # Cached
curl http://localhost:4444/health  # Direct (bypass cache)

# Verify caching
curl -I http://localhost:8080/openapi.json | grep X-Cache-Status
# X-Cache-Status: MISS (first request)
# X-Cache-Status: HIT  (subsequent requests)
```

**Kubernetes Deployment**:

Add nginx sidecar to gateway pods:

```yaml
# production-values.yaml
mcpContextForge:
  # Enable nginx caching sidecar
  nginx:
    enabled: true
    cacheSize: 2Gi              # Total cache size
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 250m
        memory: 256Mi

  # Gateway configuration remains the same
  replicaCount: 5
  resources:
    limits:
      cpu: 4000m
      memory: 8Gi
```

**Cache Configuration**:

```bash
# Nginx cache zones are pre-configured in nginx/nginx.conf
# Adjust TTLs by editing nginx.conf:

# Static assets: 30 days (aggressive caching)
proxy_cache_valid 200 30d;

# API responses: 5 minutes (balance freshness/performance)
proxy_cache_valid 200 5m;

# OpenAPI schema: 24 hours (rarely changes)
proxy_cache_valid 200 24h;
```

**Cache Bypass Rules**:

Nginx automatically bypasses cache for:

- POST, PUT, PATCH, DELETE requests
- WebSocket connections (`/servers/*/ws`)
- Server-Sent Events (`/servers/*/sse`)
- JSON-RPC endpoint (`/`)

**Rate Limiting** (tuned for 3000 concurrent users):

```nginx
# Zone: 10MB shared memory (~160,000 unique IPs)
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=3000r/s;
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;

# Applied to API endpoints
limit_req zone=api_limit burst=3000 nodelay;
limit_conn conn_limit 3000;
```

| Parameter | Value | Effect |
|-----------|-------|--------|
| `rate=3000r/s` | 3000 tokens/sec | Sustained request rate |
| `burst=3000` | Bucket size | Instant burst capacity |
| `limit_conn 3000` | Per IP | Max concurrent connections |

Tune for production by lowering limits (e.g., `rate=100r/s`, `burst=50`).

**High-Concurrency Worker Tuning**:

```nginx
worker_processes auto;
worker_rlimit_nofile 65535;
worker_connections 8192;

# Listen with performance optimizations
listen 80 backlog=4096 reuseport;

# Upstream keepalive pool
upstream gateway_backend {
    least_conn;
    server gateway:4444 max_fails=0;
    keepalive 512;
    keepalive_requests 100000;
}
```

**Access Logging**: Disabled by default (`access_log off`) for load testing performance. Enable for debugging only.

**Monitoring**:

```bash
# Check cache status (via X-Cache-Status header)
curl -I http://localhost:8080/tools | grep X-Cache-Status

# View cache size
docker-compose exec nginx du -sh /var/cache/nginx/*

# Analyze cache hit rate
docker-compose exec nginx cat /var/log/nginx/access.log | \
  grep -oP 'cache_status=\K\w+' | sort | uniq -c
```

**When to Use**:

- ✅ High traffic (>1000 req/sec)
- ✅ Read-heavy workloads
- ✅ Static asset delivery
- ✅ Mobile/remote clients (reduces bandwidth)
- ❌ Write-heavy workloads (limited benefit)
- ❌ Real-time updates required (<1 min staleness)

**Documentation**: See `nginx/README.md` for detailed configuration.

---

### Response Compression

**Bandwidth Optimization**: Reduce data transfer by 30-70% with automatic response compression.

ContextForge includes built-in response compression middleware that automatically compresses JSON, HTML, CSS, and JavaScript responses.

#### Compression Algorithms Comparison

| Algorithm | Speed | Ratio | Browser Support | Best For |
|-----------|-------|-------|-----------------|----------|
| **Brotli** | Medium | Best (15-25% smaller than Gzip) | Modern browsers | Production, CDNs |
| **Zstd** | Fastest | Very good | Growing support | High-throughput APIs |
| **GZip** | Fast | Good | Universal | Legacy compatibility |

**Algorithm Negotiation**: Client sends `Accept-Encoding` header, server responds with best supported algorithm.

```
Client: Accept-Encoding: br, gzip, deflate, zstd
Server: Content-Encoding: br  (uses Brotli if available)
```

#### Gateway vs Nginx Compression

| Aspect | Gateway Compression | Nginx Compression |
|--------|---------------------|-------------------|
| When to use | Single container, no proxy | Multi-tier with nginx |
| CPU location | Application pods | Proxy pods |
| Caching | After compression | Stores compressed |
| Configuration | Environment variables | nginx.conf |

**Recommendation**:

- **With Nginx proxy**: Disable gateway compression, let Nginx handle it
- **Direct access**: Enable gateway compression

```bash
# With Nginx proxy (compression at proxy layer)
COMPRESSION_ENABLED=false

# Direct access (compression at application layer)
COMPRESSION_ENABLED=true
```

#### Configuration

```bash
# Enable compression (default: true)
COMPRESSION_ENABLED=true

# Minimum response size to compress (bytes)
# Responses smaller than this won't be compressed
COMPRESSION_MINIMUM_SIZE=500

# Compression quality levels
COMPRESSION_GZIP_LEVEL=6          # GZip: 1-9 (6=balanced)
COMPRESSION_BROTLI_QUALITY=4      # Brotli: 0-11 (4=balanced)
COMPRESSION_ZSTD_LEVEL=3          # Zstd: 1-22 (3=fast)
```

**Algorithm Priority**: Brotli (best) > Zstd (fast) > GZip (universal)

**Performance Impact**:

- **Bandwidth reduction**: 30-70% for JSON/HTML responses
- **CPU overhead**: <5% (Brotli level 4, GZip level 6)
- **Latency**: Minimal (<10ms for typical responses)
- **Scalability**: Increases effective throughput per pod

**Tuning for Scale**:

```bash
# High-traffic production (optimize for speed)
COMPRESSION_GZIP_LEVEL=4          # Faster compression
COMPRESSION_BROTLI_QUALITY=3      # Lower quality, faster
COMPRESSION_ZSTD_LEVEL=1          # Fastest

# Bandwidth-constrained (optimize for size)
COMPRESSION_GZIP_LEVEL=9          # Best compression
COMPRESSION_BROTLI_QUALITY=11     # Maximum quality
COMPRESSION_ZSTD_LEVEL=9          # Balanced slow

# Development (disable compression)
COMPRESSION_ENABLED=false         # No compression overhead
```

**Benefits at Scale**:

- **Lower bandwidth costs**: 30-70% reduction in egress traffic
- **Faster response times**: Smaller payloads transfer faster
- **Higher throughput**: More requests per second with same bandwidth
- **Better cache hit rates**: Smaller cached responses
- **Mobile-friendly**: Critical for mobile clients on slow networks

**Kubernetes Configuration**:

```yaml
# production-values.yaml
mcpContextForge:
  config:
    COMPRESSION_ENABLED: "true"
    COMPRESSION_MINIMUM_SIZE: "500"
    COMPRESSION_GZIP_LEVEL: "6"
    COMPRESSION_BROTLI_QUALITY: "4"
    COMPRESSION_ZSTD_LEVEL: "3"
```

**Monitoring Compression**:

```bash
# Check compression in action
curl -H "Accept-Encoding: br" https://gateway.example.com/openapi.json -v \
  | grep -i "content-encoding"
# Should show: content-encoding: br

# Measure compression ratio
UNCOMPRESSED=$(curl -s https://gateway.example.com/openapi.json | wc -c)
COMPRESSED=$(curl -H "Accept-Encoding: br" -s https://gateway.example.com/openapi.json | wc -c)
echo "Compression ratio: $((100 - COMPRESSED * 100 / UNCOMPRESSED))%"
```

---

## 8. Benchmarking and Load Testing

### Tools

**hey** - HTTP load generator
```bash
# Install
brew install hey           # macOS
sudo apt install hey       # Ubuntu

# Or from source
go install github.com/rakyll/hey@latest
```

**k6** - Modern load testing
```bash
brew install k6            # macOS
```

### Baseline Test

#### Prepare Environment

```bash
# Get JWT token
export MCPGATEWAY_BEARER_TOKEN=$(python3 -m mcpgateway.utils.create_jwt_token \
  --username admin@example.com --exp 10080 --secret my-test-key)

# Create test payload
cat > payload.json <<EOF
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {}
}
EOF
```

#### Run Load Test

```bash
#!/bin/bash
# test-load.sh

# Test parameters
REQUESTS=10000
CONCURRENCY=200
URL="http://localhost:4444/"

# Run test
hey -n $REQUESTS -c $CONCURRENCY \
    -m POST \
    -T application/json \
    -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \
    -D payload.json \
    $URL
```

### Interpret Results

```
Summary:
  Total:        5.2341 secs
  Slowest:      0.5234 secs
  Fastest:      0.0123 secs
  Average:      0.1045 secs
  Requests/sec: 1910.5623      ← Target metric

Status code distribution:
  [200] 10000 responses

Response time histogram:
  0.012 [1]     |
  0.050 [2341]  |■■■■■■■■■■■
  0.100 [4523]  |■■■■■■■■■■■■■■■■■■■■■■
  0.150 [2234]  |■■■■■■■■■■■
  0.200 [901]   |■■■■
  0.250 [0]     |
```

**Key metrics:**

- **Requests/sec**: Throughput (target: >1000 RPS per pod)
- **P99 latency**: 99th percentile (target: <500ms)
- **Error rate**: 5xx responses (target: <0.1%)

### Kubernetes Load Test

```bash
# Deploy test pod
kubectl run load-test --image=williamyeh/hey:latest \
  --rm -it --restart=Never -- \
  -n 100000 -c 500 \
  -H "Authorization: Bearer $TOKEN" \
  http://mcp-gateway-service/
```

### Advanced: k6 Script

```javascript
// load-test.k6.js
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 100 },   // Ramp up
    { duration: '5m', target: 100 },   // Sustained
    { duration: '2m', target: 500 },   // Spike
    { duration: '5m', target: 500 },   // High load
    { duration: '2m', target: 0 },     // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(99)<500'],  // 99% < 500ms
    http_req_failed: ['rate<0.01'],    // <1% errors
  },
};

export default function () {
  const payload = JSON.stringify({
    jsonrpc: '2.0',
    id: 1,
    method: 'tools/list',
    params: {},
  });

  const res = http.post('http://localhost:4444/', payload, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${__ENV.TOKEN}`,
    },
  });

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
}
```

```bash
# Run k6 test
TOKEN=$MCPGATEWAY_BEARER_TOKEN k6 run load-test.k6.js
```

---

## 9. Health Checks and Readiness

### Health Check Endpoints

ContextForge provides two health endpoints:

#### Liveness Probe: `/health`

**Purpose**: Is the application alive?

```python
@app.get("/health")
async def healthcheck(db: Session = Depends(get_db)):
    """Check database connectivity"""
    try:
        db.execute(text("SELECT 1"))
        return {"status": "healthy"}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}
```

**Response:**
```json
{
  "status": "healthy"
}
```

#### Readiness Probe: `/ready`

**Purpose**: Is the application ready to receive traffic?

```python
@app.get("/ready")
async def readiness_check(db: Session = Depends(get_db)):
    """Check if ready to serve traffic"""
    try:
        await asyncio.to_thread(db.execute, text("SELECT 1"))
        return JSONResponse({"status": "ready"}, status_code=200)
    except Exception as e:
        return JSONResponse(
            {"status": "not ready", "error": str(e)},
            status_code=503
        )
```

### Kubernetes Probe Configuration

```yaml
# charts/mcp-stack/templates/deployment-mcpgateway.yaml
containers:

  - name: mcp-context-forge

    # Startup probe (initial readiness)
    startupProbe:
      exec:
        command:

          - python3
          - /app/mcpgateway/utils/db_isready.py
          - --max-tries=1
          - --timeout=2
      initialDelaySeconds: 10
      periodSeconds: 5
      failureThreshold: 60        # 5 minutes max

    # Readiness probe (traffic routing)
    readinessProbe:
      httpGet:
        path: /ready
        port: 4444
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      successThreshold: 1
      failureThreshold: 3

    # Liveness probe (restart if unhealthy)
    livenessProbe:
      httpGet:
        path: /health
        port: 4444
      initialDelaySeconds: 10
      periodSeconds: 15
      timeoutSeconds: 2
      successThreshold: 1
      failureThreshold: 3
```

### Probe Tuning Guidelines

**Startup Probe:**

- Use for slow initialization (database migrations, model loading)
- `failureThreshold × periodSeconds` = max startup time
- Example: 60 × 5s = 5 minutes

**Readiness Probe:**

- Aggressive: Remove pod from load balancer quickly
- `failureThreshold` = 3 (fail fast)
- `periodSeconds` = 10 (frequent checks)

**Liveness Probe:**

- Conservative: Avoid unnecessary restarts
- `failureThreshold` = 5 (tolerate transient issues)
- `periodSeconds` = 15 (less frequent)

### Monitoring Health

```bash
# Check pod health
kubectl get pods -n mcp-gateway

# Detailed status
kubectl describe pod <pod-name> -n mcp-gateway

# Check readiness
kubectl get pods -n mcp-gateway \
  -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

# Test health endpoint
kubectl exec -it <pod-name> -n mcp-gateway -- \
  curl http://localhost:4444/health

# View probe failures
kubectl get events -n mcp-gateway \
  --field-selector involvedObject.name=<pod-name>
```

---

## 10. Stateless Architecture and Long-Running Connections

### Stateless Design Principles

ContextForge is designed to be **stateless**, enabling horizontal scaling:

1. **No local session storage**: All sessions in Redis
2. **No in-memory caching** (in production): Use Redis
3. **Database-backed state**: All data in PostgreSQL
4. **Shared configuration**: Environment variables via ConfigMap

### Session Management

#### Stateful Sessions (Not Recommended for Scale)

```bash
USE_STATEFUL_SESSIONS=true  # Event store in database
```

**Limitations:**

- Sessions tied to specific pods
- Requires sticky sessions (session affinity)
- Doesn't scale horizontally

#### Stateless Sessions (Recommended)

```bash
USE_STATEFUL_SESSIONS=false
JSON_RESPONSE_ENABLED=true
CACHE_TYPE=redis
```

**Benefits:**

- Any pod can handle any request
- True horizontal scaling
- Automatic failover

#### Session Cleanup Performance

For high session counts, ContextForge uses parallel session cleanup with bounded concurrency to efficiently manage database-backed sessions:

- Uses `asyncio.gather()` with semaphore for parallel database operations
- Default concurrency limit of 20 prevents DB pool exhaustion
- Achieves 11-13x speedup over sequential cleanup
- Runs automatically every 5 minutes

See [Parallel Session Cleanup](parallel-session-cleanup.md) for implementation details.

### Long-Running Connections

ContextForge supports long-running connections for streaming:

#### Server-Sent Events (SSE)

```python
# Endpoint: /servers/{id}/sse
@app.get("/servers/{server_id}/sse")
async def sse_endpoint(server_id: int):
    """Stream events to client"""
    # Connection can last minutes/hours
```

#### WebSocket

```python
# Endpoint: /servers/{id}/ws
@app.websocket("/servers/{server_id}/ws")
async def websocket_endpoint(server_id: int):
    """Bidirectional streaming"""
```

### Load Balancer Configuration

**Kubernetes Service** (default):
```yaml
# Distributes connections across pods
apiVersion: v1
kind: Service
metadata:
  name: mcp-gateway-service
spec:
  type: ClusterIP
  sessionAffinity: None        # No sticky sessions
  ports:

    - port: 80
      targetPort: 4444
```

**NGINX Ingress** (for WebSocket):
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/websocket-services: "mcp-gateway-service"
spec:
  rules:

    - host: gateway.example.com
      http:
        paths:

          - path: /
            pathType: Prefix
            backend:
              service:
                name: mcp-gateway-service
                port:
                  number: 80
```

### Connection Lifecycle

```
Client → Load Balancer → Pod A (SSE stream)
                ↓
            (Pod A dies)
                ↓
Client ← Load Balancer → Pod B (reconnect)
```

**Best practices:**

1. Client implements reconnection logic
2. Server sets `SSE_KEEPALIVE_INTERVAL=30` (keepalive events)
3. Load balancer timeout > keepalive interval

---

## 11. Kubernetes Production Deployment

### Reference Architecture

```yaml
# production-values.yaml
mcpContextForge:
  # --- Scaling ---
  replicaCount: 5

  hpa:
    enabled: true
    minReplicas: 5
    maxReplicas: 50
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

  # --- Resources ---
  resources:
    limits:
      cpu: 4000m          # 4 cores per pod
      memory: 8Gi
    requests:
      cpu: 2000m          # 2 cores per pod
      memory: 4Gi

  # --- Configuration ---
  config:
    # Gunicorn
    GUNICORN_WORKERS: "16"
    GUNICORN_TIMEOUT: "600"
    GUNICORN_MAX_REQUESTS: "100000"
    GUNICORN_PRELOAD_APP: "true"

    # Database
    DB_POOL_SIZE: "50"
    DB_MAX_OVERFLOW: "10"
    DB_POOL_TIMEOUT: "60"
    DB_POOL_RECYCLE: "3600"

    # Cache
    CACHE_TYPE: redis
    CACHE_PREFIX: mcpgw:
    SESSION_TTL: "3600"
    MESSAGE_TTL: "600"

    # Performance
    TOOL_CONCURRENT_LIMIT: "20"
    RESOURCE_CACHE_SIZE: "2000"

  # --- Health Checks ---
  probes:
    startup:
      type: exec
      command: ["python3", "/app/mcpgateway/utils/db_isready.py"]
      periodSeconds: 5
      failureThreshold: 60

    readiness:
      type: http
      path: /ready
      port: 4444
      periodSeconds: 10
      failureThreshold: 3

    liveness:
      type: http
      path: /health
      port: 4444
      periodSeconds: 15
      failureThreshold: 5

# --- PostgreSQL ---
postgres:
  enabled: true

  resources:
    limits:
      cpu: 8000m          # 8 cores
      memory: 32Gi
    requests:
      cpu: 4000m
      memory: 16Gi

  persistence:
    enabled: true
    size: 100Gi
    storageClassName: fast-ssd

  # Connection limits
  # max_connections = (50 pods × 16 workers × 50 pool × 1.2) + 200
  config:
    max_connections: 50000
    shared_buffers: 8GB
    effective_cache_size: 24GB
    work_mem: 32MB

# --- Redis ---
redis:
  enabled: true

  resources:
    limits:
      cpu: 4000m
      memory: 16Gi
    requests:
      cpu: 2000m
      memory: 8Gi

  persistence:
    enabled: true
    size: 50Gi
```

### Deployment Steps

```bash
# 1. Create namespace
kubectl create namespace mcp-gateway

# 2. Create secrets
kubectl create secret generic mcp-secrets \
  -n mcp-gateway \
  --from-literal=JWT_SECRET_KEY=$(openssl rand -hex 32) \
  --from-literal=AUTH_ENCRYPTION_SECRET=$(openssl rand -hex 32) \
  --from-literal=POSTGRES_PASSWORD=$(openssl rand -base64 32)

# 3. Install with Helm
helm upgrade --install mcp-stack ./charts/mcp-stack \
  -n mcp-gateway \
  -f production-values.yaml \
  --wait \
  --timeout 10m

# 4. Verify deployment
kubectl get pods -n mcp-gateway
kubectl get hpa -n mcp-gateway
kubectl get svc -n mcp-gateway

# 5. Run migration job
kubectl get jobs -n mcp-gateway

# 6. Test scaling
kubectl top pods -n mcp-gateway
```

### Pod Disruption Budget

```yaml
# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mcp-gateway-pdb
  namespace: mcp-gateway
spec:
  minAvailable: 3         # Keep 3 pods always running
  selector:
    matchLabels:
      app: mcp-gateway
```

### Network Policies

```yaml
# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mcp-gateway-policy
  namespace: mcp-gateway
spec:
  podSelector:
    matchLabels:
      app: mcp-gateway
  policyTypes:

    - Ingress
    - Egress
  ingress:

    - from:
        - podSelector:
            matchLabels:
              app: ingress-nginx
      ports:

        - protocol: TCP
          port: 4444
  egress:

    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:

        - protocol: TCP
          port: 5432

    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:

        - protocol: TCP
          port: 6379
```

---

## 12. Monitoring and Observability

### OpenTelemetry Integration

ContextForge includes built-in OpenTelemetry support:

```bash
# Enable observability
OTEL_ENABLE_OBSERVABILITY=true
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
OTEL_SERVICE_NAME=mcp-gateway
```

### Prometheus Metrics

Deploy Prometheus stack:

```bash
# Add Prometheus Helm repo
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts

# Install kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --create-namespace
```

### Key Metrics to Monitor

**Application Metrics:**

- Request rate: `rate(http_requests_total[1m])`
- Latency: `histogram_quantile(0.99, http_request_duration_seconds)`
- Error rate: `rate(http_requests_total{status=~"5.."}[1m])`

**System Metrics:**

- CPU usage: `container_cpu_usage_seconds_total`
- Memory usage: `container_memory_working_set_bytes`
- Network I/O: `container_network_receive_bytes_total`

**Database Metrics:**

- Connection pool usage: `db_pool_size` / `db_pool_connections_active`
- Query latency: `db_query_duration_seconds`
- Deadlocks: `pg_stat_database_deadlocks`

**HPA Metrics:**
```bash
kubectl get hpa -n mcp-gateway -w
```

### Grafana Dashboards

Import dashboards:

1. **Kubernetes Cluster Monitoring** (ID: 7249)
2. **PostgreSQL** (ID: 9628)
3. **Redis** (ID: 11835)
4. **NGINX Ingress** (ID: 9614)

### Alerting Rules

```yaml
# prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: mcp-gateway-alerts
  namespace: monitoring
spec:
  groups:

    - name: mcp-gateway
      interval: 30s
      rules:

        - alert: HighErrorRate
          expr: |
            rate(http_requests_total{status=~"5..", namespace="mcp-gateway"}[5m]) > 0.05
          for: 5m
          annotations:
            summary: "High error rate detected"

        - alert: HighLatency
          expr: |
            histogram_quantile(0.99,
              rate(http_request_duration_seconds_bucket[5m])) > 1
          for: 5m
          annotations:
            summary: "P99 latency exceeds 1s"

        - alert: DatabaseConnectionPoolExhausted
          expr: |
            db_pool_connections_active / db_pool_size > 0.9
          for: 2m
          annotations:
            summary: "Database connection pool >90% utilized"
```

---

## Summary and Checklist

### Performance Technology Stack

ContextForge is built on a high-performance foundation:

✅ **Pydantic v2.11+** - Rust-powered validation (5-50x faster than v1)
✅ **FastAPI** - Modern async framework with OpenAPI support
✅ **Uvicorn [standard]** - ASGI server with uvloop + httptools (15-30% faster)
✅ **Granian (optional)** - Rust-based HTTP server with native HTTP/2 (+20-50% faster)
✅ **SQLAlchemy 2.0** - Async database operations
✅ **psycopg3 [c,binary]** - Modern PostgreSQL adapter (auto-prepared statements, COPY protocol, pipeline mode)
✅ **orjson** - High-performance JSON serialization (3x faster)
✅ **hiredis** - C-based Redis parser (up to 83x faster for large responses)
✅ **Python 3.11+** - Current stable with excellent performance
🔮 **Python 3.14** - Future free-threading support (beta)

### Scaling Checklist

- [ ] **Vertical Scaling**

  - [ ] Configure Gunicorn workers: `(2 × CPU) + 1`
  - [ ] Allocate CPU: 1 core per 2 workers
  - [ ] Allocate memory: 256MB + (workers × 200MB)

- [ ] **Horizontal Scaling**

  - [ ] Deploy to Kubernetes with HPA enabled
  - [ ] Set `minReplicas` ≥ 3 for high availability
  - [ ] Configure shared PostgreSQL and Redis

- [ ] **Database Optimization**

  - [ ] Calculate `max_connections`: `(pods × workers × pool) × 1.2`
  - [ ] Set `DB_POOL_SIZE` per worker (recommended: 50)
  - [ ] Configure `DB_POOL_RECYCLE=3600` to prevent stale connections
  - [ ] Use psycopg3 URL format: `postgresql+psycopg://`
  - [ ] Tune `DB_PREPARE_THRESHOLD` for auto-prepared statements
  - [ ] Consider PgBouncer for connection multiplexing (high concurrency)
  - [ ] Verify database indexes are applied (10-100x query improvement)

- [ ] **Caching**

  - [ ] Enable Redis: `CACHE_TYPE=redis`
  - [ ] Set `REDIS_URL` to shared Redis instance
  - [ ] Configure TTLs: `SESSION_TTL=3600`, `MESSAGE_TTL=600`
  - [ ] Tune Redis pool: `REDIS_MAX_CONNECTIONS=150` (high concurrency)
  - [ ] Use hiredis parser: `REDIS_PARSER=hiredis` (up to 83x faster)
  - [ ] Enable JWT caching: `JWT_CACHE_ENABLED=true`
  - [ ] Enable auth caching: `AUTH_CACHE_ENABLED=true`
  - [ ] Enable registry caching: `REGISTRY_CACHE_ENABLED=true`
  - [ ] Enable admin stats caching: `ADMIN_STATS_CACHE_ENABLED=true`

- [ ] **Performance**

  - [ ] Select HTTP server: `HTTP_SERVER=gunicorn` (stable) or `granian` (faster)
  - [ ] Tune Gunicorn: `GUNICORN_PRELOAD_APP=true`, `GUNICORN_WORKERS=auto`
  - [ ] Or tune Granian: `GRANIAN_BACKLOG=4096`, `GRANIAN_BACKPRESSURE=64`
  - [ ] Set timeouts: `GUNICORN_TIMEOUT=600`
  - [ ] Configure retries: `RETRY_MAX_ATTEMPTS=3`
  - [ ] Enable compression: `COMPRESSION_ENABLED=true`
  - [ ] Disable audit trail for load testing: `AUDIT_TRAIL_ENABLED=false`

- [ ] **Logging & Metrics**

  - [ ] Set log level: `LOG_LEVEL=ERROR` (production)
  - [ ] Disable access logs: `DISABLE_ACCESS_LOG=true`
  - [ ] Disable DB logging: `STRUCTURED_LOGGING_DATABASE_ENABLED=false`
  - [ ] Enable metrics buffer: `METRICS_BUFFER_ENABLED=true`
  - [ ] Enable metrics cache: `METRICS_CACHE_ENABLED=true`

- [ ] **Health Checks**

  - [ ] Configure `/health` liveness probe
  - [ ] Configure `/ready` readiness probe
  - [ ] Set appropriate thresholds and timeouts

- [ ] **Monitoring**

  - [ ] Enable OpenTelemetry: `OTEL_ENABLE_OBSERVABILITY=true`
  - [ ] Deploy Prometheus and Grafana
  - [ ] Configure alerts for errors, latency, and resources

- [ ] **Load Testing**

  - [ ] Benchmark with `hey` or `k6`
  - [ ] Target: >1000 RPS per pod, P99 <500ms
  - [ ] Test failover scenarios

### Reference Documentation

- [Performance Architecture Diagram](../architecture/performance-architecture.md)
- [Gunicorn Configuration](../deployment/local.md)
- [Kubernetes Deployment](../deployment/kubernetes.md)
- [Helm Charts](../deployment/helm.md)
- [Performance Testing](../testing/performance.md)
- [Observability](observability.md)
- [Configuration Guide](configuration.md)
- [Database Tuning](tuning.md)

---

## Additional Resources

### External Links

- [Gunicorn Documentation](https://docs.gunicorn.org/)
- [Uvicorn Deployment](https://www.uvicorn.org/deployment/)
- [Granian Documentation](https://granian.dev/)
- [uvloop GitHub](https://github.com/MagicStack/uvloop)
- [httptools GitHub](https://github.com/MagicStack/httptools)
- [Kubernetes HPA](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
- [PostgreSQL Connection Pooling](https://www.postgresql.org/docs/current/runtime-config-connection.html)
- [PgBouncer Documentation](https://www.pgbouncer.org/config.html)
- [psycopg3 Documentation](https://www.psycopg.org/psycopg3/docs/)
- [psycopg3 COPY Protocol](https://www.psycopg.org/psycopg3/docs/basic/copy.html)
- [psycopg3 Pipeline Mode](https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html)
- [Redis Cluster](https://redis.io/docs/reference/cluster-spec/)
- [hiredis GitHub](https://github.com/redis/hiredis)
- [orjson GitHub](https://github.com/ijl/orjson)
- [OpenTelemetry Python](https://opentelemetry.io/docs/instrumentation/python/)

### Community

- [GitHub Discussions](https://github.com/ibm/mcp-context-forge/discussions)
- [Issue Tracker](https://github.com/ibm/mcp-context-forge/issues)

---

*Last updated: 2025-12-27*