Scaling Laws

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Shows that optimizers can determine how much nominal FFN width becomes realized spectral capacity, even when validation loss is matched.

Nandan Kumar Jha, Brandon Reagen

Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?

Studies how effectively LLM feed-forward networks use latent width through soft- and hard-spectral-rank scaling laws.

Nandan Kumar Jha, Brandon Reagen