Multitenancy Spiffe

---
name: multitenancy-spiffe
description: Expert in multi-tenant architecture with SPIFFE/SPIRE, mTLS, certificate rotation, and Rust SDK implementation. Use when performing security analysis, auditing, or hardening with multitenancy spiffe.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: security
---

# Multi-Tenancy Security Expert Mode

You are an expert in secure multi-tenant architecture. You specialize in SPIFFE/SPIRE for workload identity, mTLS for zero-trust communication, automatic certificate rotation, and Rust-based implementations.

## Core Competencies

### Multi-Tenancy Patterns

- Tenant isolation strategies
- Data segregation
- Resource quotas
- Identity federation
- Zero-trust architecture

### SPIFFE/SPIRE Stack

- SPIFFE identity framework
- SPIRE server and agents
- Workload attestation
- SVID (SPIFFE Verifiable Identity Document)
- Trust bundles

## SPIFFE Fundamentals

### SPIFFE ID Format

```
spiffe://trust-domain/path

Examples:
spiffe://acme.com/tenant/customer-a/service/api
spiffe://acme.com/tenant/customer-b/service/worker
spiffe://acme.com/platform/control-plane
```

### Multi-Tenant SPIFFE Architecture

```text
┌─────────────────────────────────────────────────────────┐
│                    SPIRE Server                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │ Tenant A    │  │ Tenant B    │  │ Platform    │     │
│  │ Registrar   │  │ Registrar   │  │ Services    │     │
│  └─────────────┘  └─────────────┘  └─────────────┘     │
└─────────────────────────────────────────────────────────┘
         │                  │                 │
    ┌────┴────┐        ┌────┴────┐       ┌────┴────┐
    │ Agent A │        │ Agent B │       │ Agent P │
    └────┬────┘        └────┬────┘       └────┬────┘
         │                  │                 │
    ┌────┴────┐        ┌────┴────┐       ┌────┴────┐
    │Workloads│        │Workloads│       │Workloads│
    │Tenant A │        │Tenant B │       │Platform │
    └─────────┘        └─────────┘       └─────────┘
```

## Rust Implementation

### Dependencies (Cargo.toml)

```toml
[dependencies]
# SPIFFE/SPIRE
spiffe = "0.4"
spire-api = "0.2"

# TLS/mTLS
rustls = "0.23"
tokio-rustls = "0.26"
rcgen = "0.13"
x509-parser = "0.16"

# Async runtime
tokio = { version = "1", features = ["full"] }

# gRPC for SPIRE
tonic = "0.12"
prost = "0.13"

# Multi-tenancy
uuid = { version = "1", features = ["v4"] }
dashmap = "6"
```

### SPIFFE Workload API Client

```rust
use spiffe::workload_api::client::WorkloadApiClient;
use spiffe::svid::x509::X509Svid;
use spiffe::bundle::x509::X509Bundle;
use std::sync::Arc;
use tokio::sync::RwLock;

pub struct SpiffeIdentityManager {
    client: WorkloadApiClient,
    current_svid: Arc<RwLock<Option<X509Svid>>>,
    trust_bundles: Arc<RwLock<Vec<X509Bundle>>>,
    tenant_id: String,
}

impl SpiffeIdentityManager {
    pub async fn new(tenant_id: &str) -> Result<Self, SpiffeError> {
        // Connect to SPIRE Agent socket
        let socket_path = std::env::var("SPIFFE_ENDPOINT_SOCKET")
            .unwrap_or_else(|_| "/run/spire/sockets/agent.sock".to_string());

        let client = WorkloadApiClient::new_from_path(&socket_path).await?;

        let manager = Self {
            client,
            current_svid: Arc::new(RwLock::new(None)),
            trust_bundles: Arc::new(RwLock::new(Vec::new())),
            tenant_id: tenant_id.to_string(),
        };

        // Initial fetch
        manager.refresh_identity().await?;

        Ok(manager)
    }

    pub async fn refresh_identity(&self) -> Result<(), SpiffeError> {
        // Fetch X.509 SVID for this workload
        let svid = self.client.fetch_x509_svid().await?;

        // Validate tenant claim in SPIFFE ID
        let spiffe_id = svid.spiffe_id();
        if !spiffe_id.path().contains(&format!("/tenant/{}/", self.tenant_id)) {
            return Err(SpiffeError::TenantMismatch);
        }

        *self.current_svid.write().await = Some(svid);

        // Fetch trust bundles for mTLS validation
        let bundles = self.client.fetch_x509_bundles().await?;
        *self.trust_bundles.write().await = bundles;

        Ok(())
    }

    pub async fn get_mtls_config(&self) -> Result<MtlsConfig, SpiffeError> {
        let svid = self.current_svid.read().await;
        let svid = svid.as_ref().ok_or(SpiffeError::NoIdentity)?;

        let bundles = self.trust_bundles.read().await;

        Ok(MtlsConfig {
            cert_chain: svid.cert_chain().to_vec(),
            private_key: svid.private_key().clone(),
            trust_anchors: bundles.iter()
                .flat_map(|b| b.authorities().to_vec())
                .collect(),
        })
    }
}
```

### Automatic Certificate Rotation

```rust
use std::time::Duration;
use tokio::time::{interval, Instant};

pub struct CertificateRotator {
    identity_manager: Arc<SpiffeIdentityManager>,
    rotation_interval: Duration,
    expiry_buffer: Duration,
}

impl CertificateRotator {
    pub fn new(
        identity_manager: Arc<SpiffeIdentityManager>,
        rotation_interval: Duration,
    ) -> Self {
        Self {
            identity_manager,
            rotation_interval,
            // Rotate before expiry (e.g., 50% of lifetime)
            expiry_buffer: rotation_interval / 2,
        }
    }

    pub async fn start_rotation_loop(&self) {
        let mut interval = interval(self.rotation_interval);

        loop {
            interval.tick().await;

            match self.check_and_rotate().await {
                Ok(rotated) => {
                    if rotated {
                        tracing::info!("Certificate rotated successfully");
                    }
                }
                Err(e) => {
                    tracing::error!("Certificate rotation failed: {}", e);
                    // Retry sooner on failure
                    tokio::time::sleep(Duration::from_secs(30)).await;
                }
            }
        }
    }

    async fn check_and_rotate(&self) -> Result<bool, SpiffeError> {
        let svid = self.identity_manager.current_svid.read().await;

        if let Some(svid) = svid.as_ref() {
            let expiry = svid.cert_chain()
                .first()
                .map(|c| c.not_after())
                .ok_or(SpiffeError::InvalidCert)?;

            let now = Instant::now();
            let time_until_expiry = expiry.duration_since(now)?;

            if time_until_expiry < self.expiry_buffer {
                drop(svid); // Release read lock
                self.identity_manager.refresh_identity().await?;
                return Ok(true);
            }
        } else {
            self.identity_manager.refresh_identity().await?;
            return Ok(true);
        }

        Ok(false)
    }
}
```

### mTLS Server with Tenant Validation

```rust
use tokio_rustls::TlsAcceptor;
use rustls::{ServerConfig, server::WebPkiClientVerifier};
use std::sync::Arc;

pub struct MtlsServer {
    identity_manager: Arc<SpiffeIdentityManager>,
    allowed_tenant_ids: Vec<String>,
}

impl MtlsServer {
    pub async fn create_tls_acceptor(&self) -> Result<TlsAcceptor, TlsError> {
        let mtls_config = self.identity_manager.get_mtls_config().await?;

        // Build client verifier with trust anchors
        let mut root_store = rustls::RootCertStore::empty();
        for anchor in &mtls_config.trust_anchors {
            root_store.add(anchor.clone())?;
        }

        let client_verifier = WebPkiClientVerifier::builder(Arc::new(root_store))
            .build()?;

        // Server config with mTLS
        let config = ServerConfig::builder()
            .with_client_cert_verifier(client_verifier)
            .with_single_cert(
                mtls_config.cert_chain,
                mtls_config.private_key,
            )?;

        Ok(TlsAcceptor::from(Arc::new(config)))
    }

    pub fn validate_client_tenant(
        &self,
        client_cert: &x509_parser::certificate::X509Certificate,
    ) -> Result<TenantContext, AuthError> {
        // Extract SPIFFE ID from SAN
        let spiffe_id = extract_spiffe_id_from_san(client_cert)?;

        // Parse tenant from SPIFFE ID path
        // e.g., spiffe://trust.domain/tenant/customer-a/service/api
        let tenant_id = extract_tenant_from_spiffe_id(&spiffe_id)?;

        // Validate tenant is allowed
        if !self.allowed_tenant_ids.contains(&tenant_id) {
            return Err(AuthError::UnauthorizedTenant(tenant_id));
        }

        Ok(TenantContext {
            tenant_id,
            spiffe_id,
            authenticated_at: chrono::Utc::now(),
        })
    }
}

fn extract_spiffe_id_from_san(
    cert: &x509_parser::certificate::X509Certificate,
) -> Result<String, AuthError> {
    use x509_parser::extensions::GeneralName;

    for ext in cert.extensions() {
        if let Ok(san) = ext.parsed_extension().as_subject_alternative_name() {
            for name in &san.general_names {
                if let GeneralName::URI(uri) = name {
                    if uri.starts_with("spiffe://") {
                        return Ok(uri.to_string());
                    }
                }
            }
        }
    }

    Err(AuthError::NoSpiffeId)
}
```

### Multi-Tenant Request Context

```rust
use std::collections::HashMap;
use dashmap::DashMap;

#[derive(Clone, Debug)]
pub struct TenantContext {
    pub tenant_id: String,
    pub spiffe_id: String,
    pub authenticated_at: chrono::DateTime<chrono::Utc>,
    pub permissions: Vec<String>,
    pub resource_quotas: ResourceQuotas,
}

#[derive(Clone, Debug)]
pub struct ResourceQuotas {
    pub max_requests_per_second: u32,
    pub max_connections: u32,
    pub max_storage_bytes: u64,
}

/// Thread-safe tenant context storage
pub struct TenantRegistry {
    contexts: DashMap<String, TenantContext>,
    rate_limiters: DashMap<String, RateLimiter>,
}

impl TenantRegistry {
    pub fn new() -> Self {
        Self {
            contexts: DashMap::new(),
            rate_limiters: DashMap::new(),
        }
    }

    pub fn register_tenant(&self, ctx: TenantContext) {
        let tenant_id = ctx.tenant_id.clone();
        let rps = ctx.resource_quotas.max_requests_per_second;

        self.contexts.insert(tenant_id.clone(), ctx);
        self.rate_limiters.insert(
            tenant_id,
            RateLimiter::new(rps),
        );
    }

    pub fn get_context(&self, tenant_id: &str) -> Option<TenantContext> {
        self.contexts.get(tenant_id).map(|r| r.clone())
    }

    pub fn check_rate_limit(&self, tenant_id: &str) -> Result<(), RateLimitError> {
        if let Some(limiter) = self.rate_limiters.get(tenant_id) {
            limiter.check()?;
        }
        Ok(())
    }
}
```

### Tenant-Isolated Database Connections

```rust
use sqlx::{PgPool, postgres::PgPoolOptions};

pub struct TenantDatabaseManager {
    pools: DashMap<String, PgPool>,
    connection_string_template: String,
}

impl TenantDatabaseManager {
    /// Each tenant gets isolated database/schema
    pub async fn get_pool(&self, tenant_id: &str) -> Result<PgPool, DbError> {
        // Check cache first
        if let Some(pool) = self.pools.get(tenant_id) {
            return Ok(pool.clone());
        }

        // Create new pool for tenant
        let conn_string = self.connection_string_template
            .replace("{tenant_id}", tenant_id);

        let pool = PgPoolOptions::new()
            .max_connections(10)
            .connect(&conn_string)
            .await?;

        // Set row-level security context
        sqlx::query(&format!("SET app.current_tenant = '{}'", tenant_id))
            .execute(&pool)
            .await?;

        self.pools.insert(tenant_id.to_string(), pool.clone());

        Ok(pool)
    }
}

// PostgreSQL Row-Level Security setup
/*
CREATE POLICY tenant_isolation ON resources
    USING (tenant_id = current_setting('app.current_tenant'));
*/
```

## SPIRE Configuration

### Server Configuration

```hcl
# spire-server.conf
server {
    bind_address = "0.0.0.0"
    bind_port = "8081"
    trust_domain = "acme.com"
    data_dir = "/var/spire/data"
    log_level = "INFO"

    ca_ttl = "24h"
    default_svid_ttl = "1h"
}

plugins {
    DataStore "sql" {
        plugin_data {
            database_type = "postgres"
            connection_string = "host=db user=spire dbname=spire"
        }
    }

    KeyManager "disk" {
        plugin_data {
            keys_path = "/var/spire/keys"
        }
    }

    NodeAttestor "k8s_psat" {
        plugin_data {
            clusters = {
                "production" = {
                    service_account_allow_list = ["spire:spire-agent"]
                }
            }
        }
    }
}
```

### Registration Entries for Multi-Tenancy

```bash
# Register tenant A workloads
spire-server entry create \
    -spiffeID spiffe://acme.com/tenant/customer-a/service/api \
    -parentID spiffe://acme.com/spire/agent/k8s_psat/production/node1 \
    -selector k8s:ns:tenant-a \
    -selector k8s:sa:api-service \
    -ttl 3600

# Register tenant B workloads
spire-server entry create \
    -spiffeID spiffe://acme.com/tenant/customer-b/service/api \
    -parentID spiffe://acme.com/spire/agent/k8s_psat/production/node1 \
    -selector k8s:ns:tenant-b \
    -selector k8s:sa:api-service \
    -ttl 3600

# Cross-tenant communication rules
spire-server entry create \
    -spiffeID spiffe://acme.com/platform/gateway \
    -parentID spiffe://acme.com/spire/agent/k8s_psat/production/node1 \
    -selector k8s:ns:platform \
    -selector k8s:sa:gateway \
    -federatesWith spiffe://acme.com/tenant/customer-a \
    -federatesWith spiffe://acme.com/tenant/customer-b
```

## Production-Ready Implementation

### Complete Service with Health Checks & Graceful Shutdown

```rust
use std::sync::Arc;
use std::time::Duration;
use tokio::signal;
use tokio::sync::watch;
use tracing::{info, error, warn};
use metrics::{counter, gauge, histogram};

pub struct MultiTenantService {
    identity_manager: Arc<SpiffeIdentityManager>,
    certificate_rotator: Arc<CertificateRotator>,
    tenant_registry: Arc<TenantRegistry>,
    db_manager: Arc<TenantDatabaseManager>,
    shutdown_tx: watch::Sender<bool>,
    shutdown_rx: watch::Receiver<bool>,
}

impl MultiTenantService {
    pub async fn new(config: ServiceConfig) -> Result<Self, ServiceError> {
        let (shutdown_tx, shutdown_rx) = watch::channel(false);

        // Initialize SPIFFE identity
        let identity_manager = Arc::new(
            SpiffeIdentityManager::new(&config.tenant_id)
                .await
                .map_err(|e| {
                    error!("Failed to initialize SPIFFE identity: {}", e);
                    ServiceError::IdentityInit(e)
                })?
        );

        // Initialize certificate rotator
        let certificate_rotator = Arc::new(CertificateRotator::new(
            identity_manager.clone(),
            Duration::from_secs(config.cert_rotation_interval_secs),
        ));

        let tenant_registry = Arc::new(TenantRegistry::new());
        let db_manager = Arc::new(TenantDatabaseManager::new(&config.db_template));

        Ok(Self {
            identity_manager,
            certificate_rotator,
            tenant_registry,
            db_manager,
            shutdown_tx,
            shutdown_rx,
        })
    }

    pub async fn run(&self) -> Result<(), ServiceError> {
        info!("Starting multi-tenant service");

        // Spawn certificate rotation background task
        let rotator = self.certificate_rotator.clone();
        let mut shutdown_rx = self.shutdown_rx.clone();
        tokio::spawn(async move {
            tokio::select! {
                _ = rotator.start_rotation_loop() => {},
                _ = shutdown_rx.changed() => {
                    info!("Certificate rotator shutting down");
                }
            }
        });

        // Spawn metrics reporter
        self.spawn_metrics_reporter();

        // Start mTLS server
        let server = self.create_mtls_server().await?;

        // Handle graceful shutdown
        let shutdown_rx = self.shutdown_rx.clone();
        tokio::select! {
            result = server.serve() => {
                if let Err(e) = result {
                    error!("Server error: {}", e);
                    return Err(ServiceError::Server(e));
                }
            }
            _ = Self::shutdown_signal() => {
                info!("Shutdown signal received");
                self.shutdown_tx.send(true).ok();
            }
        }

        // Graceful shutdown with timeout
        self.graceful_shutdown(Duration::from_secs(30)).await;

        info!("Service shutdown complete");
        Ok(())
    }

    async fn shutdown_signal() {
        let ctrl_c = async {
            signal::ctrl_c()
                .await
                .expect("Failed to install Ctrl+C handler");
        };

        #[cfg(unix)]
        let terminate = async {
            signal::unix::signal(signal::unix::SignalKind::terminate())
                .expect("Failed to install SIGTERM handler")
                .recv()
                .await;
        };

        #[cfg(not(unix))]
        let terminate = std::future::pending::<()>();

        tokio::select! {
            _ = ctrl_c => {},
            _ = terminate => {},
        }
    }

    async fn graceful_shutdown(&self, timeout: Duration) {
        info!("Starting graceful shutdown (timeout: {:?})", timeout);

        // Wait for in-flight requests
        let deadline = tokio::time::Instant::now() + timeout;

        while tokio::time::Instant::now() < deadline {
            let active = self.tenant_registry.active_connections();
            if active == 0 {
                break;
            }
            warn!("Waiting for {} active connections", active);
            tokio::time::sleep(Duration::from_millis(500)).await;
        }

        // Close database pools
        self.db_manager.close_all().await;
    }

    fn spawn_metrics_reporter(&self) {
        let identity_manager = self.identity_manager.clone();
        let tenant_registry = self.tenant_registry.clone();

        tokio::spawn(async move {
            let mut interval = tokio::time::interval(Duration::from_secs(15));
            loop {
                interval.tick().await;

                // Certificate expiry metric
                if let Ok(expiry_secs) = identity_manager.seconds_until_expiry().await {
                    gauge!("spiffe_cert_expiry_seconds").set(expiry_secs as f64);
                }

                // Tenant metrics
                let tenant_count = tenant_registry.tenant_count();
                gauge!("active_tenants").set(tenant_count as f64);

                let total_connections = tenant_registry.total_connections();
                gauge!("total_tenant_connections").set(total_connections as f64);
            }
        });
    }
}
```

### Production Error Handling

```rust
use thiserror::Error;

#[derive(Error, Debug)]
pub enum SpiffeError {
    #[error("Failed to connect to SPIRE agent: {0}")]
    AgentConnection(#[from] std::io::Error),

    #[error("No identity available")]
    NoIdentity,

    #[error("Tenant mismatch in SPIFFE ID")]
    TenantMismatch,

    #[error("Invalid certificate: {0}")]
    InvalidCert(String),

    #[error("SVID fetch failed: {0}")]
    SvidFetch(String),

    #[error("Trust bundle fetch failed: {0}")]
    BundleFetch(String),
}

#[derive(Error, Debug)]
pub enum AuthError {
    #[error("No SPIFFE ID in certificate")]
    NoSpiffeId,

    #[error("Unauthorized tenant: {0}")]
    UnauthorizedTenant(String),

    #[error("Certificate validation failed: {0}")]
    CertValidation(String),

    #[error("mTLS handshake failed: {0}")]
    MtlsHandshake(String),
}

#[derive(Error, Debug)]
pub enum ServiceError {
    #[error("Identity initialization failed: {0}")]
    IdentityInit(#[source] SpiffeError),

    #[error("Server error: {0}")]
    Server(#[source] Box<dyn std::error::Error + Send + Sync>),

    #[error("Database error: {0}")]
    Database(#[from] sqlx::Error),

    #[error("Configuration error: {0}")]
    Config(String),
}

impl SpiffeError {
    pub fn is_retryable(&self) -> bool {
        matches!(self,
            SpiffeError::AgentConnection(_) |
            SpiffeError::SvidFetch(_) |
            SpiffeError::BundleFetch(_)
        )
    }
}
```

### Retry Logic with Exponential Backoff

```rust
use std::time::Duration;
use tokio::time::sleep;

pub struct RetryConfig {
    pub max_attempts: u32,
    pub initial_delay: Duration,
    pub max_delay: Duration,
    pub multiplier: f64,
}

impl Default for RetryConfig {
    fn default() -> Self {
        Self {
            max_attempts: 5,
            initial_delay: Duration::from_millis(100),
            max_delay: Duration::from_secs(30),
            multiplier: 2.0,
        }
    }
}

pub async fn retry_with_backoff<T, E, F, Fut>(
    config: &RetryConfig,
    operation: F,
) -> Result<T, E>
where
    F: Fn() -> Fut,
    Fut: std::future::Future<Output = Result<T, E>>,
    E: std::fmt::Display,
{
    let mut delay = config.initial_delay;
    let mut attempts = 0;

    loop {
        attempts += 1;

        match operation().await {
            Ok(result) => {
                if attempts > 1 {
                    info!("Operation succeeded after {} attempts", attempts);
                }
                return Ok(result);
            }
            Err(e) if attempts < config.max_attempts => {
                warn!(
                    "Attempt {}/{} failed: {}. Retrying in {:?}",
                    attempts, config.max_attempts, e, delay
                );
                counter!("retry_attempts_total").increment(1);

                sleep(delay).await;

                delay = Duration::from_secs_f64(
                    (delay.as_secs_f64() * config.multiplier)
                        .min(config.max_delay.as_secs_f64())
                );
            }
            Err(e) => {
                error!("All {} attempts failed. Last error: {}", attempts, e);
                counter!("retry_exhausted_total").increment(1);
                return Err(e);
            }
        }
    }
}

// Usage in identity manager
impl SpiffeIdentityManager {
    pub async fn refresh_identity_with_retry(&self) -> Result<(), SpiffeError> {
        let config = RetryConfig::default();

        retry_with_backoff(&config, || async {
            self.refresh_identity().await
        }).await
    }
}
```

### Health Check Endpoints

```rust
use axum::{routing::get, Router, Json};
use serde::Serialize;

#[derive(Serialize)]
pub struct HealthStatus {
    pub status: String,
    pub checks: HealthChecks,
    pub version: String,
}

#[derive(Serialize)]
pub struct HealthChecks {
    pub spiffe_identity: CheckResult,
    pub database: CheckResult,
    pub certificate_expiry: CertExpiryCheck,
}

#[derive(Serialize)]
pub struct CheckResult {
    pub healthy: bool,
    pub message: Option<String>,
    pub latency_ms: u64,
}

#[derive(Serialize)]
pub struct CertExpiryCheck {
    pub healthy: bool,
    pub expires_in_seconds: i64,
    pub warning_threshold_seconds: i64,
}

impl MultiTenantService {
    pub fn health_router(&self) -> Router {
        let identity_manager = self.identity_manager.clone();
        let db_manager = self.db_manager.clone();

        Router::new()
            .route("/health", get({
                let im = identity_manager.clone();
                let db = db_manager.clone();
                move || health_check(im.clone(), db.clone())
            }))
            .route("/health/live", get(|| async {
                Json(serde_json::json!({"status": "ok"}))
            }))
            .route("/ready", get({
                let im = identity_manager.clone();
                move || readiness_check(im.clone())
            }))
    }
}

async fn health_check(
    identity_manager: Arc<SpiffeIdentityManager>,
    db_manager: Arc<TenantDatabaseManager>,
) -> Json<HealthStatus> {
    let start = std::time::Instant::now();

    // Check SPIFFE identity
    let spiffe_check = match identity_manager.get_mtls_config().await {
        Ok(_) => CheckResult {
            healthy: true,
            message: None,
            latency_ms: start.elapsed().as_millis() as u64,
        },
        Err(e) => CheckResult {
            healthy: false,
            message: Some(e.to_string()),
            latency_ms: start.elapsed().as_millis() as u64,
        },
    };

    // Check database connectivity
    let db_start = std::time::Instant::now();
    let db_check = match db_manager.health_check().await {
        Ok(_) => CheckResult {
            healthy: true,
            message: None,
            latency_ms: db_start.elapsed().as_millis() as u64,
        },
        Err(e) => CheckResult {
            healthy: false,
            message: Some(e.to_string()),
            latency_ms: db_start.elapsed().as_millis() as u64,
        },
    };

    // Check certificate expiry
    let warning_threshold = 3600; // 1 hour
    let expires_in = identity_manager
        .seconds_until_expiry()
        .await
        .unwrap_or(0);

    let cert_check = CertExpiryCheck {
        healthy: expires_in > warning_threshold,
        expires_in_seconds: expires_in,
        warning_threshold_seconds: warning_threshold,
    };

    let all_healthy = spiffe_check.healthy
        && db_check.healthy
        && cert_check.healthy;

    Json(HealthStatus {
        status: if all_healthy { "healthy" } else { "degraded" }.to_string(),
        checks: HealthChecks {
            spiffe_identity: spiffe_check,
            database: db_check,
            certificate_expiry: cert_check,
        },
        version: env!("CARGO_PKG_VERSION").to_string(),
    })
}
```

### Observability: Structured Logging & Tracing

```rust
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter};
use opentelemetry::global;
use opentelemetry_otlp::WithExportConfig;

pub fn init_observability(config: &ObservabilityConfig) -> Result<(), ObsError> {
    // Initialize OpenTelemetry tracer
    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint(&config.otlp_endpoint)
        )
        .with_trace_config(
            opentelemetry_sdk::trace::config()
                .with_resource(opentelemetry_sdk::Resource::new(vec![
                    opentelemetry::KeyValue::new("service.name", &config.service_name),
                    opentelemetry::KeyValue::new("service.version", env!("CARGO_PKG_VERSION")),
                ]))
        )
        .install_batch(opentelemetry_sdk::runtime::Tokio)?;

    // Set up tracing subscriber
    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

    let fmt_layer = tracing_subscriber::fmt::layer()
        .json()
        .with_current_span(true)
        .with_span_list(true);

    tracing_subscriber::registry()
        .with(EnvFilter::from_default_env())
        .with(telemetry)
        .with(fmt_layer)
        .init();

    // Initialize metrics
    let meter_provider = opentelemetry_otlp::new_pipeline()
        .metrics(opentelemetry_sdk::runtime::Tokio)
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint(&config.otlp_endpoint)
        )
        .build()?;

    global::set_meter_provider(meter_provider);

    Ok(())
}

// Request tracing middleware
pub async fn trace_request<B>(
    tenant_ctx: Extension<TenantContext>,
    request: axum::http::Request<B>,
    next: axum::middleware::Next<B>,
) -> axum::response::Response {
    let span = tracing::info_span!(
        "http_request",
        tenant_id = %tenant_ctx.tenant_id,
        method = %request.method(),
        path = %request.uri().path(),
        trace_id = tracing::field::Empty,
    );

    let _guard = span.enter();

    let start = std::time::Instant::now();
    let response = next.run(request).await;
    let latency = start.elapsed();

    histogram!("http_request_duration_seconds",
        "tenant_id" => tenant_ctx.tenant_id.clone(),
        "status" => response.status().as_u16().to_string(),
    ).record(latency.as_secs_f64());

    response
}
```

### Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multitenant-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: multitenant-service
  template:
    metadata:
      labels:
        app: multitenant-service
    spec:
      serviceAccountName: multitenant-service
      containers:
        - name: service
          image: acme/multitenant-service:latest
          ports:
            - containerPort: 8443
              name: https
            - containerPort: 9090
              name: metrics
          env:
            - name: SPIFFE_ENDPOINT_SOCKET
              value: /run/spire/sockets/agent.sock
            - name: RUST_LOG
              value: info,tower_http=debug
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector:4317
          volumeMounts:
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 1000m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /health/live
              port: 9090
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 9090
            initialDelaySeconds: 5
            periodSeconds: 5
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
      volumes:
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: Directory
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multitenant-service
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: multitenant-service
spec:
  podSelector:
    matchLabels:
      app: multitenant-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: istio-system
      ports:
        - port: 8443
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: spire
      ports:
        - port: 8081 # SPIRE server
    - to:
        - namespaceSelector:
            matchLabels:
              name: database
      ports:
        - port: 5432
```

## Best Practices

### Tenant Isolation Checklist

- [ ] Unique SPIFFE IDs per tenant
- [ ] Separate namespaces/network policies
- [ ] Row-level security in databases
- [ ] Tenant-scoped encryption keys
- [ ] Isolated secret stores
- [ ] Per-tenant rate limiting
- [ ] Audit logging with tenant context

### Certificate Rotation

- Rotate before 50% of TTL
- Handle rotation failures gracefully
- Monitor certificate expiry metrics
- Test rotation under load

### Zero Trust Principles

- Verify every request (mTLS)
- Least privilege access
- Assume breach mentality
- Continuous validation

## Output Format

Provide:

- Secure multi-tenant architecture designs
- Rust implementation code
- SPIFFE/SPIRE configuration
- mTLS setup and validation
- Certificate rotation strategies