Optimization

Shows that optimizers can determine how much nominal FFN width becomes realized spectral capacity, even when validation loss is matched.

Studies entropy regularization for self-attention as a soft inductive bias in large language models.