3

Shows that optimizers can determine how much nominal FFN width becomes realized spectral capacity, even when validation loss is matched.

Uses random-matrix tools to analyze how multi-head latent attention evolves during training, revealing capacity bottlenecks and representation-geometry shifts.

Develops entropy-guided attention and hierarchical entropy regularization for efficient private LLM inference with reduced nonlinearities.

Studies entropy regularization for self-attention as a soft inductive bias in large language models.

Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on …

In this paper we demonstrated that how the current trend in private inference myopically optimized the performance only for zero arrival rate; in particular, they have developed the mechanism to mitigate the bottlenecked caused by non-linearity in neural networks. However, in a real-world scenario when inference request comes even with a moderate arrival rate the homomorphic encryption becomes the main bottleneck since we can no longer pre-process it in the offline computation phase.

LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. …

We introduce an information-theoretic framework to characterize the role of nonlinearities in decoder-only language models, laying a …