Efficient Coding Criticality Sloppiness

---
name: efficient-coding-criticality-sloppiness
description: "Efficient coding under resource constraints drives neural systems towards criticality and sloppiness. Links Fisher information maximization to power-law distributions and critical brain hypothesis."
triggers:
  - efficient coding
  - neural criticality
  - brain criticality
  - sloppiness
  - Fisher information neural
  - power-law neural
  - critical brain
  - neural avalanche
  - Gaussian population coding
  - soft modes neural
category: neuroscience
tags:
  - criticality
  - efficient-coding
  - Fisher-information
  - sloppiness
  - neural-population-coding
  - brain-criticality
  - power-law
source: "arXiv:2605.22598"
authors: ["He Xiao", "Xinyue Zhao", "Weikang Wang"]
---

# Efficient Coding Under Constraint → Neural Criticality & Sloppiness

## Overview

This skill encapsulates the theoretical framework from arXiv:2605.22598 showing that **maximizing Fisher information under resource constraints** naturally leads neural systems to operate near criticality — unifying efficient coding theory with the critical brain hypothesis.

**Key insight**: The brain's critical state is not accidental but is a **functional consequence** of information-theoretic optimization under metabolic/resource constraints.

## Core Methodology

### 1. Gaussian Population Coding Model
- Represent neural population responses as Gaussian distributions
- Define **Fisher information** as the metric for coding efficiency
- Introduce **resource constraints** (energy, metabolic cost, channel capacity)
- Maximize Fisher information subject to these constraints

### 2. Emergence of Criticality
Under constrained Fisher information maximization:
- **Soft modes** emerge: response directions with near-zero eigenvalues
- **Diverging correlation lengths**: long-range correlations characteristic of criticality
- **Power-law distributions**: neural avalanche statistics naturally follow power laws
- This recapitulates both **statistical criticality** and **dynamical criticality**

### 3. Unifying Two Criticality Perspectives
| Perspective | Mechanism | Observable |
|---|---|---|
| Statistical criticality | Diverging correlation length | Power-law spatial correlations |
| Dynamical criticality | Critical slowing down + bifurcation | Slow timescales, bifurcation point |

The spatial structure in the model bridges these two views.

### 4. Sloppiness as Emergent Property
- **Sloppiness**: neural systems have highly variable parameter sensitivities — some parameters matter enormously, others barely at all
- This framework shows sloppiness is a **natural consequence** of efficient coding: optimized systems develop stiff and sloppy parameter combinations automatically
- Sloppy directions ≈ soft modes ≈ directions that don't cost Fisher information

## Implementation Steps

### Step 1: Build the Population Coding Model
```python
import numpy as np
from scipy.linalg import eigh

# N neurons, d stimulus dimensions
N, d = 100, 2

# Tuning curves: f_i(s) = exp(-||s - mu_i||^2 / (2*sigma^2))
mu = np.random.randn(N, d)  # preferred stimuli
sigma = 1.0

def tuning_curve(s, mu, sigma):
    """Gaussian tuning curves for N neurons"""
    diffs = s - mu  # (N, d)
    return np.exp(-np.sum(diffs**2, axis=1) / (2 * sigma**2))
```

### Step 2: Compute Fisher Information Matrix
```python
def fisher_information(s, mu, sigma, noise_cov):
    """
    Compute Fisher Information Matrix at stimulus s.
    J(s) = (df/ds)^T * Sigma^{-1} * (df/ds)
    """
    f = tuning_curve(s, mu, sigma)
    # Gradient of tuning curves
    df_ds = -f[:, None] * (s - mu) / sigma**2  # (N, d)
    
    noise_inv = np.linalg.inv(noise_cov)
    J = df_ds.T @ noise_inv @ df_ds
    return J

# Total Fisher info (scalar): trace or determinant
def total_fisher(J):
    return np.trace(J)  # or np.linalg.det(J)
```

### Step 3: Optimize Under Resource Constraint
```python
from scipy.optimize import minimize

def constrained_fisher_maximization(N, d, resource_budget):
    """
    Maximize sum of Fisher information subject to:
    - Metabolic cost: sum(f_i) <= budget
    - Normalization constraints
    """
    def objective(params):
        mu = params[:N*d].reshape(N, d)
        sigma = params[N*d]
        # Compute expected Fisher info over stimulus space
        J_total = compute_expected_fisher(mu, sigma)
        return -J_total  # minimize negative = maximize
    
    def constraint_cost(params):
        mu = params[:N*d].reshape(N, d)
        sigma = params[N*d]
        avg_rate = compute_average_rate(mu, sigma)
        return resource_budget - avg_rate  # >= 0
    
    result = minimize(objective, x0, 
                      constraints={'type': 'ineq', 'fun': constraint_cost},
                      method='SLSQP')
    return result
```

### Step 4: Analyze Eigenspectrum for Criticality
```python
def analyze_criticality(J_fisher):
    """
    Check for criticality signatures in Fisher Information Matrix.
    Criticality: eigenvalue spectrum follows power law.
    """
    eigenvalues, eigenvectors = eigh(J_fisher)
    eigenvalues = np.sort(eigenvalues)[::-1]
    
    # Check power-law fit
    log_rank = np.log(np.arange(1, len(eigenvalues)+1))
    log_eig = np.log(eigenvalues + 1e-10)
    
    # Fit power law: log(lambda) ~ -alpha * log(rank)
    from numpy.polynomial import polynomial as P
    coeffs = np.polyfit(log_rank, log_eig, 1)
    alpha = -coeffs[0]
    
    return {
        'eigenvalues': eigenvalues,
        'power_law_exponent': alpha,
        'is_critical': abs(alpha - 1.0) < 0.3,  # ~power law with exponent ~1
        'num_soft_modes': np.sum(eigenvalues < 0.01 * eigenvalues[0])
    }
```

### Step 5: Measure Sloppiness
```python
def sloppiness_index(eigenvalues):
    """
    Sloppiness: large ratio between largest and smallest eigenvalues.
    Sloppy if lambda_max / lambda_min >> 1.
    """
    sorted_eig = np.sort(np.abs(eigenvalues))[::-1]
    ratio = sorted_eig[0] / (sorted_eig[-1] + 1e-10)
    
    # Sloppiness score: number of decades spanned
    import math
    decades = math.log10(ratio)
    return {
        'eigenvalue_ratio': ratio,
        'decades': decades,
        'is_sloppy': decades > 4  # sloppy if > 4 orders of magnitude
    }
```

## Key Results

1. **Fisher info maximization → Power-law eigenspectrum** (criticality signature)
2. **Resource constraints** (metabolic budget) are necessary — unconstrained optimization does not produce criticality
3. **Spatial structure** bridges statistical and dynamical criticality perspectives
4. **Sloppiness emerges automatically**: stiff directions encode task-relevant information; sloppy directions are near-null modes

## Practical Applications

### Neural Data Analysis
```python
def analyze_neural_population(spike_rates, stimulus_conditions):
    """
    Test if a recorded neural population shows critical signatures.
    1. Estimate Fisher information from data
    2. Check eigenspectrum for power-law
    3. Quantify sloppiness
    """
    # Estimate noise covariance
    noise_cov = np.cov(spike_rates.T)
    
    # Compute Fisher info at each stimulus
    J_list = []
    for s in stimulus_conditions:
        J = estimate_fisher_from_data(spike_rates, s)
        J_list.append(J)
    
    J_avg = np.mean(J_list, axis=0)
    criticality = analyze_criticality(J_avg)
    sloppiness = sloppiness_index(np.linalg.eigvalsh(J_avg))
    
    return criticality, sloppiness
```

### Neural Network Design
- Use as regularization: penalize deviation from critical eigenspectrum
- Initialize network weights to produce soft modes
- Use sloppiness as a training diagnostic

## Connections to Existing Theory

| Concept | Connection |
|---|---|
| Critical Brain Hypothesis | Derived from first principles here |
| Maximum Entropy Principle | Fisher information maximization is dual |
| Edge of Chaos | Critical point = edge of bifurcation |
| Free Energy Principle | Resource-constrained inference |
| Neural Manifold Hypothesis | Soft modes = low-dimensional manifold |

## Pitfalls

- **Resource constraint form matters**: different cost functions lead to different critical regimes
- **Finite-size effects**: criticality is approximate in finite populations
- **Noise model sensitivity**: Gaussian noise assumption may not hold for all neural systems
- Power-law fitting requires sufficient dynamic range (at least 2-3 decades)

## Citation

```bibtex
@article{xiao2026efficient,
  title={Efficient coding under constraint drives neural systems towards criticality and sloppiness},
  author={Xiao, He and Zhao, Xinyue and Wang, Weikang},
  journal={arXiv:2605.22598},
  year={2026}
}
```