AERO: Entropy-Guided Attention for Private LLM Inference
Entropy-guided attention for efficient private LLM inferenceAbstract
Private inference for language models is bottlenecked by nonlinear operations, motivating architectures that reduce expensive nonlinear components while preserving model quality. AERO studies the entropy dynamics induced by reducing transformer nonlinearities and identifies two failure modes: entropy collapse in deeper layers and entropic overload in earlier layers. It introduces entropy-guided attention and hierarchical entropy regularization to stabilize attention behavior while improving the efficiency of private LLM inference.
Related workshop versions
The AERO line grew through several peer-reviewed workshop versions before the current under-review manuscript:
- Regularizing the Entropy Landscape of Self-Attention: Towards a Soft Inductive Bias in LLMs, OPT Workshop at NeurIPS 2025. Workshop · OpenReview
- ReLU’s Revival: On the Entropic Overload in Normalization-Free Large Language Models, ATTRIB Workshop at NeurIPS 2024. Workshop · arXiv · Code
- Entropy-Guided Attention for Private LLMs, PPAI Workshop at AAAI 2025. arXiv · Code · Press release
Nandan Kumar Jha
Ph.D., Electrical and Computer Engineering
I study nonlinear representation dynamics in large language models, focusing on how nonlinearities, architecture, and optimization jointly shape representational geometry, scaling behavior, and usable computational capacity.