arXiv Machine Learning
techCenter
Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationtranslating…
1 min readUnknownarXiv Digital Media
arXiv:2608.26288v1 Announce Type: new
Abstract: Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations. Existing theory either replaces this…