Attention Mechanisms

A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention
Uses random-matrix tools to analyze how multi-head latent attention evolves during training, revealing capacity bottlenecks and representation-geometry shifts.
Nandan Kumar Jha, Brandon Reagen
PDF Cite News
AERO: Entropy-Guided Attention for Private LLM Inference
Develops entropy-guided attention and hierarchical entropy regularization for efficient private LLM inference with reduced nonlinearities.
Nandan Kumar Jha, Brandon Reagen
Cite Code Poster Video Earlier arXiv Press release
AERO: Entropy-Guided Attention for Private LLM Inference
Regularizing the Entropy Landscape of Self-Attention: Towards a Soft Inductive Bias in LLMs
Studies entropy regularization for self-attention as a soft inductive bias in large language models.
Nandan Kumar Jha, Brandon Reagen
Cite Workshop OpenReview