Advancing long-context transformer efficiency with PHAROS
Long-context transformer training still faces significant compute and memory challenges because dense attention scales quadratically. PHAROS is a composable augmentation of Lighthouse-style hierarchical attention that makes long-context token selection more adaptive, guided, persistent, and uncertainty-aware, while remaining compatible with efficient attention pipelines such as FlashAttention.
A smarter approach to hierarchical attention selection
Traditional Lighthouse-style attention uses uniform token budgets across attention heads, scores pyramid levels independently, and recomputes token selection across transformer layers. PHAROS addresses these limitations through four coordinated mechanisms that dynamically allocate attention resources, connect information across pyramid levels, preserve relevant selection signals across model depth, and respond to uncertainty during token selection.
Four mechanisms powering PHAROS
PHAROS introduces depth-sounding budget allocation (DBA), which dynamically allocates token budgets based on attention-head entropy, and range light cross-level guidance (RCG), which uses coarse-level information to guide finer-level selection. Tidal momentum selection (TMS) propagates selection evidence across transformer layers to improve consistency, while fog gate confidence fallback (FCF) expands the selection budget when confidence is low. Together, these mechanisms enable a more flexible approach to identifying relevant tokens in long contexts.
Preserving compatibility with efficient attention pipelines
PHAROS is designed as a lightweight enhancement rather than a replacement for existing attention infrastructure. The mechanisms operate around the token-selection process, allowing selected keys and values to continue being processed by FlashAttention. The approach introduces only lightweight per-head parameters for guidance, momentum, and confidence-based fallback.
A framework for future evaluation
PHAROS is currently presented as a proposed methodology and a reproducible evaluation framework rather than an empirically validated performance improvement. Future evaluation should assess training loss, validation perplexity, long-context retrieval accuracy, throughput, memory consumption, and latency. It should also test the contribution of each PHAROS mechanism individually across model scales, datasets, context lengths, and tokenizers.
Rethinking attention for longer contexts
By combining adaptive budgets, cross-level guidance, cross-layer selection persistence, and uncertainty-aware fallback, PHAROS proposes a structured evolution of hierarchical attention. It aims to improve long-context token selection while maintaining compatibility with established efficient-attention architectures and training pipelines.
