Maximum Entropy Neural Connectivity

---
name: maximum-entropy-neural-connectivity
description: Maximum entropy principle for neural network connectivity — normative framework for understanding how task constraints shape neural connectivity structure without gradient descent.
tags: [neuroscience, neural-networks, maximum-entropy, connectivity, computational-neuroscience, brain-network]
category: ai_collection
version: "1.0"
source: "arXiv:2605.25607"
authors: ["Ludwig Hruza", "Srdjan Ostojic"]
---

# Maximum Entropy Neural Connectivity

## Overview

Maximum entropy framework for deriving minimally-biased neural network connectivity from task constraints, independent of learning algorithms. Bridges normative theory with gradient-descent trained networks.

**Trigger conditions:** Use when studying relationship between neural network structure and function, connectivity analysis, normative models of brain networks, context-dependent computation, neuro-AI alignment.

## Core Methodology

### Principle
Express connectivity as a **probability distribution over single-neuron weights**, impose task requirements as constraints, and find the distribution maximizing **Shannon entropy** — the unique "least biased" solution consistent with functional requirements.

### Mathematical Framework
1. **Prior**: Start with a homogeneous (uninformative) distribution over weights
2. **Constraints**: Express task requirements (e.g., input selection, context-dependent gating) as expected-value constraints on the distribution
3. **Maximization**: Solve max-entropy subject to constraints → yields exponential family distribution
4. **Weight scale parameter** λ: Controls balance between random/structured connectivity

### Key Analytical Tractability
- Map nonlinear networks → gain-modulated linear models
- Enables closed-form maximum entropy inference in 2-layer feedforward networks
- Context-dependent input-selection tasks become analytically tractable

## Key Findings

1. **Emergent neural populations**: Maximizing entropy under task constraints spontaneously produces neuron populations each defined by their pattern of contextual gain modulation

2. **Context-number transition**: Increasing number of contexts drives a phase transition from context-specialized → unspecialized/random populations

3. **Weight-scale transition**: Increasing weight scale drives transition from structured → random stimulus selectivity

4. **Matches gradient descent**: Maximum entropy connectivity matches both qualitatively AND quantitatively the structure of gradient-descent-trained networks across different learning regimes

## Implications for Neuroscience

- Neural connectivity structure can be understood as **entropy maximization under task constraints**, not just gradient-based learning
- Provides a **task-independent normative principle** for brain connectivity
- Explains why random connectivity can achieve task performance (high-entropy solution)
- Suggests that the brain may solve tasks through the minimum-structure solution consistent with functional requirements

## Implementation Steps

1. **Define task constraints**: Formalize cognitive task requirements as expected-value constraints on weight distributions
2. **Set up Lagrangian**: L(p) = H(p) - Σλᵢ⟨fᵢ(W)⟩ for each constraint fᵢ
3. **Solve max-entropy**: Optimize using exponential family parameterization
4. **Gain modulation mapping**: Transform nonlinear activations to linear gain-modulated model for tractability
5. **Vary weight scale λ**: Trace phase transitions from structured to random regime
6. **Compare to gradient descent**: Validate by training equivalent network and comparing connectivity statistics

## Code Pattern (Python)

```python
import numpy as np
from scipy.optimize import minimize
from scipy.special import softmax

def max_entropy_connectivity(task_constraints, n_neurons, n_contexts, weight_scale=1.0):
    """
    Compute maximum entropy connectivity distribution subject to task constraints.
    
    Args:
        task_constraints: list of (constraint_fn, target_value) tuples
        n_neurons: number of neurons in layer
        n_contexts: number of context signals
        weight_scale: λ parameter controlling structure/randomness balance
    
    Returns:
        weight_distribution: parameters of the exponential family distribution
        populations: identified neural population clusters
    """
    # Gain-modulated linear approximation
    # W_eff[i,j] = g_i(c) * W[i,j] where g_i(c) is context-dependent gain
    
    # Parameterize as exponential family
    # p(W) ∝ exp(Σ λᵢ fᵢ(W) - weight_scale * ||W||²)
    
    def neg_entropy(lambdas):
        # Compute -H(p_lambda) using partition function
        pass
    
    def constraint_violations(lambdas):
        # Check if E_p[fᵢ(W)] = target_i for all i
        pass
    
    # Solve dual problem
    result = minimize(neg_entropy, x0=np.zeros(len(task_constraints)),
                     constraints={'type': 'eq', 'fun': constraint_violations})
    
    return result

def identify_populations(weight_distribution, n_contexts):
    """Cluster neurons by contextual gain modulation pattern."""
    # Each neuron i has a gain profile g_i = [g_i(c1), g_i(c2), ...]
    # Cluster neurons with similar gain profiles → neural populations
    from sklearn.cluster import KMeans
    gains = compute_contextual_gains(weight_distribution, n_contexts)
    km = KMeans(n_clusters=n_contexts + 1)  # context-specialized + unspecialized
    populations = km.fit_predict(gains)
    return populations
```

## Key Parameters

| Parameter | Description | Effect |
|-----------|-------------|--------|
| Weight scale λ | Controls entropy vs structure | High λ → random; Low λ → structured |
| n_contexts | Number of context signals | More contexts → transition to unspecialized |
| Constraint tightness | How strongly task imposes structure | Tighter → more structured connectivity |

## Pitfalls

- **Analytical tractability only in 2-layer feedforward**: Deep networks require numerical approximations
- **Gain-modulation mapping is approximate**: Nonlinear networks mapped to linear gains as approximation
- **Degenerate tasks**: Tasks with redundant constraints may not uniquely determine connectivity
- **Finite-sample effects**: Maximum entropy solutions assume infinite population; finite networks show fluctuations

## Connections to Existing Skills

- Related to: `maximum-entropy-network-structure-function`, `neural-population-dynamics`, `brain-connectivity-analysis`
- Compare to: gradient-descent connectivity analysis in `low-rank-rnn-learning-dynamics`
- Extends: normative neuroscience frameworks in `efficient-coding-criticality`

## References

- Hruza & Ostojic (2026). "Balancing structure and randomness: maximum entropy networks for context-dependent computations." arXiv:2605.25607
- Jaynes (1957). Information theory and statistical mechanics.
- Sompolinsky et al. (1988). Chaos in random neural networks.