Generalized Discrete Diffusion Snapshots

---
name: generalized-discrete-diffusion-snapshots
title: "Generalized Discrete Diffusion from Snapshots"
version: 0.0.3
engine: skillxiv-v0.0.3-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.21342"
keywords: [Discrete Diffusion, Rate Matrices, Snapshot Learning, Language Modeling, Semantic Kernels]
description: "Unify discrete diffusion for language by replacing token-wise uniform masking with generalized noising via rate matrices and snapshot latents, enabling semantic-aware forward processes. Demonstrates 1.16 BPC on Text8 and 7.65 perplexity on OpenWebText, beating autoregressive baselines; enables generalized noising processes over arbitrary vocabularies with efficient training aligned to standard architectures."
---

## Problem Statement
Existing discrete diffusion models suffer two fundamental bottlenecks:
1. **Neighborhood-blind forward process**: Token-wise corruption mechanisms (masking, uniform replacement) ignore semantic proximity and similarity structure in discrete spaces.
2. **Constrained reverse dynamics**: Restrictive mean parametrization tightly couples how denoising uncertainty translates into reverse model dynamics.

This limits expressiveness and scalability compared to continuous diffusion.

## New Paradigm: Generalized Discrete Diffusion from Snapshots

Replace token-wise uniform processes with a unified framework supporting arbitrary noising via continuous-time Markov chains (rate matrices) and snapshot latent variables. Rather than tracking full noising paths, learn from single observations (x_t, t) pairs.

## Vocabulary

**Rate Matrix (Generator Matrix)**: Mathematical object Q_t characterizing continuous-time Markov chain evolution, factorized into:
- **Exit rates**: How often chains leave current states
- **Jump kernels**: Where chains transition to

**Interpolating Matrix**: K_t = α_t I_m + (1-α_t)Π_t, combining mixing rate intensity (α_t) with mixing matrix structure (Π_t) to control noise schedule.

**Uniformization**: Poisson-based sampling enabling exact forward noising without computing matrix exponentials—efficient for large vocabularies.

**Snapshot Latent**: Single observation tuple (x_t, t) replacing full noising trajectories as variational latent variable in training.

## Founding Experiments

### Language Modeling Performance

**GDDS Absorb on Text8**:
```
Bits per character (BPC): ≤1.16 (baseline AR: 1.35)
Achieves first discrete diffusion success beating autoregressive at this scale
```

**GDDS Gauss on OpenWebText**:
```
Perplexity: ≤7.65 (baseline AR: 20.49)
Zero-shot transfer shows consistent gains across 7 downstream tasks
```

**Language Generation Quality**:
- Superior quality-diversity tradeoff versus masked/uniform baselines
- GDDS Absorb achieves lower generative perplexity with fewer decoding steps

## Opened Directions

1. **Semantic-Informed Kernels**: Extend beyond Gaussian kernels to exploit richer token similarity structures from embeddings or semantic graphs.

2. **Non-Shared Exit Rates**: Generalize uniformization beyond uniform exit rate assumption—allow token-dependent rates for fine-grained control.

3. **Native Architectural Integration**: Develop denoising networks with built-in support for path-wise objectives without two-stream encoder complexity.

4. **Computational Efficiency at Scale**: Scale semantic kernels beyond current KNN approximations; optimize for large vocabulary settings.

5. **Information-Calibration Tradeoffs**: Deeper analysis of snapshot versus full-path learning dynamics and their impact on model specialization versus generalization.