Memory Uncertainty Relation Recurrent Networks

---
name: memory-uncertainty-relation-recurrent-networks
description: >
  Memory Uncertainty Relation in random recurrent networks: inequality bounding
  short-term memory from below as an uncertainty relation between memory capacity
  and state-space fluctuations. Defines harmonic memory as an analytically tractable
  lower bound achieved by optimal readout weights.
triggers:
  - short-term memory recurrent networks
  - reservoir computing memory capacity
  - harmonic memory
  - uncertainty relation neural dynamics
  - state-space fluctuations
  - random recurrent network memory
  - echo state network capacity
  - dynamical systems memory bounds
category: ai_collection
tags:
  - recurrent-networks
  - reservoir-computing
  - memory-capacity
  - neural-dynamics
  - information-theory
  - dynamical-systems
  - cs-NE
---

# Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks

## Overview

**Paper**: "Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks"  
**Authors**: Taichi Haruna, Kohei Nakajima  
**arXiv**: [2605.24628](https://arxiv.org/abs/2605.24628)  
**Published**: 2026-05-23  
**Categories**: nlin.AO, cond-mat.dis-nn, cs.NE

## Core Contribution

This work establishes a fundamental **uncertainty-type inequality** for short-term memory in dynamical systems:

$$\text{STM}(\mathcal{S}) \cdot \text{Fluct}(\mathcal{S}) \geq C$$

Where:
- **STM**: Short-term memory capacity measure
- **Fluct**: Size of state-space fluctuations induced by input signals
- **C**: A positive constant (lower bound)

This is analogous to Heisenberg's uncertainty principle — you cannot simultaneously minimize both memory and fluctuations below the bound.

## Key Definitions

### Short-Term Memory (STM) Capacity
Following Jaeger (2001), STM measures how well a reservoir can recall past inputs:

$$\text{STM} = \sum_{k=1}^{\infty} r^2(s_t, u_{t-k})$$

Where $r^2$ is the coefficient of determination between the state $s_t$ and a delayed input $u_{t-k}$.

### Harmonic Memory
The **harmonic memory** $H(\mathcal{S})$ is the suboptimal lower bound achievable by the best linear readout weight:

$$H(\mathcal{S}) = \frac{\left(\sum_k \text{cov}(s_t, u_{t-k})\right)^2}{\text{var}(s_t)}$$

This is always achievable (tight) — making it a **constructive lower bound**.

### State-Space Fluctuation
Measured as the variance of the reservoir state induced by the input process:

$$\text{Fluct}(\mathcal{S}) = \text{Tr}[\text{Var}(s_t)]$$

## Main Results

### Theorem 1: Memory Uncertainty Inequality
For any reservoir system $\mathcal{S}$ with state process $s_t$ driven by input $u_t$:

$$\text{STM}(\mathcal{S}) \geq H(\mathcal{S}) \geq \frac{C}{\text{Fluct}(\mathcal{S})}$$

**Interpretation**: To have high memory capacity, you need either large fluctuations OR the system must be close to the harmonic optimum.

### When Equality Holds
1. **Exact equality**: When the state is a simple linear function of past inputs
2. **Asymptotic equality**: When reservoir spectral radius → 1 (edge of chaos)
3. **Strict inequality**: For nonlinear reservoirs with complex internal dynamics

### Effect of State-Space Regularization
Adding L2 regularization to reservoir states:
- Reduces fluctuations
- May decrease memory capacity proportionally
- The ratio STM/Fluct is conserved — the bound is "tight" under regularization

## Implementation Guide

### Measuring STM in a Reservoir

```python
import numpy as np
from sklearn.linear_model import LinearRegression

def measure_stm(states, inputs, max_delay=50):
    """Measure short-term memory capacity of a reservoir.
    
    Args:
        states: (T, N) reservoir states
        inputs: (T,) input time series
        max_delay: maximum delay to consider
    
    Returns:
        stm: total STM capacity
        r2_per_delay: R² for each delay
    """
    T, N = states.shape
    r2_per_delay = []
    
    for k in range(1, max_delay + 1):
        # Align states with delayed inputs
        s = states[k:, :]
        u_delayed = inputs[:T - k].reshape(-1, 1)
        
        # Fit linear readout
        reg = LinearRegression().fit(s, u_delayed)
        r2 = reg.score(s, u_delayed)
        r2_per_delay.append(max(0, r2))  # Clamp to [0, 1]
    
    stm = sum(r2_per_delay)
    return stm, r2_per_delay


def measure_harmonic_memory(states, inputs, max_delay=50):
    """Compute harmonic memory lower bound.
    
    Args:
        states: (T, N) reservoir states (use single-neuron for simplicity)
        inputs: (T,) input time series
    
    Returns:
        harmonic_memory: lower bound on STM
    """
    T, N = states.shape
    
    # Sum of covariances: sum_k cov(s_t, u_{t-k})
    cov_sum = 0
    for k in range(1, min(T // 2, 200)):
        cov = np.cov(states[k:, 0], inputs[:T - k])[0, 1]
        cov_sum += cov
    
    # Variance of state
    state_var = np.var(states[:, 0])
    
    if state_var < 1e-10:
        return 0.0
    
    return (cov_sum ** 2) / state_var


def measure_fluctuation(states):
    """Measure state-space fluctuation (trace of covariance)."""
    return np.trace(np.cov(states.T))
```

### Reservoir with Tunable Spectral Radius

```python
class EchoStateNetwork:
    """Simple Echo State Network for memory capacity experiments."""
    
    def __init__(self, N=100, spectral_radius=0.9, input_scaling=0.1, 
                 density=0.1, seed=42):
        rng = np.random.RandomState(seed)
        
        # Random recurrent weights
        W = rng.randn(N, N)
        # Sparse connectivity
        mask = rng.uniform(0, 1, (N, N)) < density
        W *= mask
        # Scale to desired spectral radius
        sr = np.max(np.abs(np.linalg.eigvals(W)))
        self.W = spectral_radius * W / sr
        
        # Input weights
        self.W_in = input_scaling * rng.randn(N, 1)
        
        self.N = N
        self.spectral_radius = spectral_radius
    
    def run(self, inputs, washout=100):
        """Run reservoir on input sequence.
        
        Args:
            inputs: (T,) input time series
            washout: initial steps to discard
        
        Returns:
            states: (T - washout, N) reservoir states
        """
        T = len(inputs)
        states = np.zeros((T, self.N))
        x = np.zeros(self.N)
        
        for t in range(T):
            x = np.tanh(self.W @ x + self.W_in.flatten() * inputs[t])
            states[t] = x
        
        return states[washout:]
    
    def memory_profile(self, inputs, max_delay=100):
        """Full memory capacity profile."""
        states = self.run(inputs, washout=200)
        stm, r2s = measure_stm(states, inputs[200:], max_delay)
        harmonic = measure_harmonic_memory(states, inputs[200:], max_delay)
        fluct = measure_fluctuation(states)
        
        return {
            'stm': stm,
            'harmonic_memory': harmonic,
            'fluctuation': fluct,
            'r2_profile': r2s,
            'memory_uncertainty_product': stm * fluct,
        }
```

### Analyzing the Uncertainty Bound

```python
import matplotlib.pyplot as plt

def analyze_memory_uncertainty(spectral_radii=np.linspace(0.5, 0.99, 20)):
    """Analyze how the memory-fluctuation trade-off changes with spectral radius."""
    
    results = []
    rng = np.random.RandomState(0)
    inputs = rng.randn(2000)
    
    for sr in spectral_radii:
        esn = EchoStateNetwork(N=100, spectral_radius=sr)
        profile = esn.memory_profile(inputs)
        profile['spectral_radius'] = sr
        results.append(profile)
    
    # Plot
    fig, axes = plt.subplots(1, 3, figsize=(15, 4))
    
    sr_vals = [r['spectral_radius'] for r in results]
    stms = [r['stm'] for r in results]
    flucts = [r['fluctuation'] for r in results]
    products = [r['memory_uncertainty_product'] for r in results]
    
    axes[0].plot(sr_vals, stms, 'b-o', label='STM')
    axes[0].set_xlabel('Spectral Radius')
    axes[0].set_ylabel('STM Capacity')
    axes[0].set_title('Memory vs Spectral Radius')
    
    axes[1].plot(sr_vals, flucts, 'r-o', label='Fluctuation')
    axes[1].set_xlabel('Spectral Radius')
    axes[1].set_ylabel('State Fluctuation')
    axes[1].set_title('Fluctuation vs Spectral Radius')
    
    axes[2].plot(sr_vals, products, 'g-o', label='STM × Fluct')
    axes[2].axhline(y=min(products), color='k', linestyle='--', label='Lower Bound')
    axes[2].set_xlabel('Spectral Radius')
    axes[2].set_ylabel('STM × Fluctuation')
    axes[2].set_title('Uncertainty Product (should be ≥ bound)')
    axes[2].legend()
    
    plt.tight_layout()
    return results, fig
```

## Theoretical Implications

### For Reservoir Computing Design
| Design Goal | Implication from Uncertainty Relation |
|------------|--------------------------------------|
| **High STM** | Must accept high fluctuations OR operate near harmonic optimum |
| **Stable dynamics** | Low fluctuations → bounded memory capacity |
| **Edge of chaos** (sr → 1) | Approaches equality — maximum memory per unit fluctuation |
| **Deep nonlinear reservoirs** | Strict inequality — some memory "wasted" on nonlinear processing |

### Connection to Physics
- Analogous to **Heisenberg uncertainty**: ΔxΔp ≥ ℏ/2
- Here: STM × Fluct ≥ C
- The "harmonic memory" is the minimum uncertainty state — like a coherent state in QM

### Practical Guidelines
1. **Choose spectral radius near 1** for best memory efficiency
2. **Use harmonic memory as a diagnostic**: if STM ≈ H, the reservoir is close to optimal
3. **Regularization trade-off**: L2 regularization reduces fluctuations but proportionally reduces memory
4. **Nonlinearity**: Strong nonlinear activation (tanh saturation) increases the STM/H gap

## Applications

1. **Reservoir computing optimization**: Tune spectral radius to the harmonic optimum
2. **Memory-efficient RNNs**: Design architectures that approach the harmonic bound
3. **Cognitive modeling**: Memory capacity bounds as a model for working memory limitations
4. **Anomaly detection**: Compare measured STM vs harmonic bound to detect sub-optimal network states

## Pitfalls

- **Washout period**: Always discard initial transients when measuring STM
- **Input statistics matter**: The bound depends on input process statistics
- **Finite-time estimation**: Need long time series for accurate STM estimation (T ≥ 10 × max_delay)
- **Nonlinear reservoirs**: The harmonic bound may be loose for highly nonlinear systems

## Citation

```bibtex
@article{haruna2026memory,
  title={Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks},
  author={Haruna, Taichi and Nakajima, Kohei},
  journal={arXiv preprint arXiv:2605.24628},
  year={2026}
}
```