Advancing long-context LLM pretraining with Lighthouse Attention
As large language models process increasingly long contexts, standard attention mechanisms face significant compute and memory challenges because they scale quadratically with sequence length. This research surveys Lighthouse Attention, a training-time hierarchical selection approach that reduces the context processed by dense attention while remaining compatible with established attention mechanisms.
Rethinking attention through hierarchical selection
Lighthouse Attention wraps standard dense attention rather than permanently replacing it. The approach constructs a multi-resolution hierarchy by symmetrically pooling queries, keys, and values, then uses lightweight, gradient-free scoring to identify relevant regions and progressively refine token selection. The selected tokens form a smaller dense subsequence that optimized kernels such as FlashAttention can process before mapping outputs back to the original sequence representation.
Combining efficient training with dense-attention compatibility
A key feature of Lighthouse Attention is its two-stage training strategy. Most pretraining uses hierarchical selection to reduce the token set passed to dense attention, followed by a short dense-attention resume phase. This approach aims to recover a checkpoint compatible with standard dense attention while separating the token-selection process from the underlying attention-kernel implementation.
Positioning Lighthouse in the efficient attention landscape
The survey examines Lighthouse alongside FlashAttention, native sparse attention (NSA), mixture of block attention (MoBA), HISA, and DeepSeek sparse attention (DSA). While these approaches improve long-context efficiency through kernel optimization, sparse routing, compression, or hierarchical indexing, Lighthouse stands out for its training-focused design, symmetric Q/K/V hierarchy, gradient-free selection, dense-kernel compatibility, and removable selection mechanism.
Exploring opportunities for more adaptive attention
The research identifies several areas for further investigation, including adaptive token budgets across attention heads and layers, stronger coarse-to-fine guidance, cross-layer selection memory, and uncertainty-aware fallback mechanisms. It also highlights opportunities to explore hybrid sparse-dense training schedules and broader evaluation across retrieval, reasoning, code understanding, memory stability, and challenging long-context scenarios.
Shaping the future of efficient long-context models
Lighthouse Attention offers a distinct approach to long-context pretraining by combining hierarchical token selection with established dense attention infrastructure. The survey highlights its architectural strengths and identifies reproducibility, adaptive selection, uncertainty management, and cross-layer coordination as key areas for continued research toward more efficient and scalable long-context language models.
