Sparse Linear Expert Transformer

---
name: sparse-linear-expert-transformer
description: "Sparsely gated tiny linear experts (sgatlin) methodology — replacing transformer feedforward layers with networks of sparsely-gated linear neurons for improved compute efficiency and interpretability. Covers isoflop comparison, linear expert removal of nonlinearity, semantic cluster interpretation, and causal factual recall. Activation: sparse linear expert, sgatlin, tiny linear expert, sparsely gated linear, linear MoE, sparse transformer FFN."
---

## Context

Mixture-of-Experts (MoE) models achieve scalability through sparsity — routing to a subset of experts per token. However, individual experts typically remain large and dense. This paper demonstrates that further increasing sparsity by shrinking each expert to a **single linear neuron** and selecting a tiny fraction of many available neurons improves both compute efficiency and interpretability.

Source paper: arXiv:2606.07414 "Sparsely gated tiny linear experts" (Simon Schug — Jun 2026).

## Core Methodology

### 1. Sparsely Gated Linear Neurons (sgatlin)

**Key Insight**: Removing the nonlinearity typically applied to MoE experts, while dramatically increasing the number of experts (each being a single neuron), yields better performance at the same compute budget.

- Replace all transformer feedforward (FFN) layers with sgatlin layers
- Each "expert" is a single linear neuron (no activation function)
- Sparse gating selects a tiny fraction of available neurons per token
- Total parameters scale with number of neurons, not with expert size

### 2. Isoflop Comparison Protocol

- Match compute budget (FLOPs) between baseline FFN and sgatlin
- Compare perplexity across language models at different compute budgets
- Key finding: sgatlin improves perplexity across all tested compute budgets
- The improvement comes from better parameter utilization through sparsity + linearity

### 3. Interpretability via Linearity + Sparsity

- **Semantic clustering**: Feedforward circuits in sgatlin form semantically structured clusters
- **Causal implication**: Individual linear neurons are causally implicated in factual recall
- **No replacement models needed**: Unlike standard FFN interpretation, sgatlin circuits can be interpreted directly without training additional replacement models
- The linearity of experts enables direct attribution analysis

### 4. Sparse Gating Mechanism

- Router computes scores over all available linear neurons
- Top-k selection (very small k relative to total neurons)
- Weighted sum of selected neurons produces the layer output
- Gating function is typically softmax or top-k with temperature

## Implementation Steps

1. **Replace FFN with sgatlin**: For each transformer layer, replace `FFN(x) = W2 * activation(W1 * x)` with `sgatlin(x) = Σ_{i∈topk} g_i(x) * (w_i^T x) * v_i`
2. **Scale neuron count**: Increase total neurons to match or exceed original parameter count
3. **Configure routing**: Set top-k value (typically k << total_neurons) for desired sparsity level
4. **Isoflop matching**: Ensure total FLOPs match baseline for fair comparison
5. **Training**: Train with standard language modeling objective
6. **Interpretability analysis**: After training, analyze semantic clusters via neuron weight inspection

## Key Results

- sgatlin improves perplexity over standard FFN at matched FLOPs
- Feedforward circuits form semantically structured clusters
- Neurons are causally implicated in factual recall
- Direct interpretation without auxiliary replacement models
- Path toward compute-efficient and interpretable transformer FFN layers

## Pitfalls

- **Nonlinearity removal is critical**: The key finding depends on removing the activation function from experts — adding it back degrades both performance and interpretability
- **Routing collapse**: With many tiny experts, routing may collapse to a subset — use entropy regularization on gating distribution
- **Gradient flow**: Linear experts may have different gradient dynamics — monitor training stability
- **Interpretability is not automatic**: Semantic clusters emerge but require careful analysis (weight inspection, causal intervention)

## Verification

- Reproduce isoflop comparison: same FLOPs, different architectures
- Verify perplexity improvement at matched compute budget
- Check semantic clustering via neuron weight analysis
- Perform causal intervention on individual neurons for factual recall

## Related Skills

- `emo-emergent-moe-modularity` — modular Mixture-of-Experts design
- `unipool-shared-expert-moe` — globally shared expert pool MoE
- `hierarchical-moe-detection` — hierarchical MoE for object detection
- `moe-optimal-transport-routing` — MoE routing via optimal transport
- `routing-distraction-multimodal-moe` — routing analysis in multimodal MoE

## Activation

sparse linear expert, sgatlin, tiny linear expert, sparsely gated linear, linear MoE, sparse transformer FFN, sparse gating interpretability, linear neuron expert, compute-efficient transformer