Elsaa Efficient Low Rank Sparse Attention

---
name: elsaa-efficient-low-rank-sparse-attention
version: 1.0.0
description: Efficient Low-Rank and Sparse Attention Approximation (ELSAA) methodology for training Transformers with longer contexts while preserving both sharp token-level interactions and broad contextual mixing.
author: Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
license: MIT
arxiv_id: 2607.20214v1
tags:
  - transformer
  - attention
  - efficiency
  - long-context
  - low-rank
  - sparse
---

# ELSAA: Efficient Low-Rank and Sparse Attention Approximation

## Overview
ELSAA (Efficient Low-Rank and Sparse Attention Approximation) addresses the quadratic $N\\times N$ attention score matrix bottleneck in Transformers by approximating the attention score operator itself rather than decomposing learned projection matrices.

## Key Components

### Dual-Branch Architecture
- **Sparse Branch**: Captures selected high-similarity interactions between query-key pairs
- **Low-Rank Branch**: Summarizes diffuse global interactions through compressed representation

### Denominator-Aware Fusion
- Introduces a fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch
- Enables proper normalization over supports with very different denominator mass

## Implementation Guidelines

### Training Workflow
1. Use standard dense projections to produce Q, K, V matrices
2. Apply ELSAA approximation to the attention score operator
3. Compute sparse and low-rank branches in parallel
4. Apply denominator-aware fusion to combine branches
5. Continue with standard Transformer forward pass

### Benefits
- Avoids materializing the full quadratic score matrix
- Enables longer-context training
- Preserves both sharp token-level interactions and broad contextual mixing
- Compatible with existing Transformer architectures

## Use Cases
- Long-context language modeling
- Document-level understanding tasks
- Memory-efficient Transformer training
- Applications requiring both local precision and global context

## Activation Keywords
ELSAA, efficient attention, low-rank attention, sparse attention, long-context transformers, quadratic bottleneck

## References
- arXiv: [2607.20214v1](https://arxiv.org/abs/2607.20214v1)
- Authors: Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
- Published: July 22, 2026