Entropy Dynamics

AERO: Entropy-Guided Attention for Private LLM Inference
Develops entropy-guided attention and hierarchical entropy regularization for efficient private LLM inference with reduced nonlinearities.
Nandan Kumar Jha, Brandon Reagen
Cite Code Poster Video Earlier arXiv Press release
AERO: Entropy-Guided Attention for Private LLM Inference
Regularizing the Entropy Landscape of Self-Attention: Towards a Soft Inductive Bias in LLMs
Studies entropy regularization for self-attention as a soft inductive bias in large language models.
Nandan Kumar Jha, Brandon Reagen
Cite Workshop OpenReview