#Adaptive optimizers

Articles (2)

• 🇺🇸 Understanding Transformer Optimization via Gradient Heterogeneity — 19/02/2026 [USA]

  arXiv:2502.00213v4 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite t...

  Related: #Transformers, #Optimization, #Gradient heterogeneity, #Stochastic gradient descent
• 🇺🇸 On Surprising Effectiveness of Masking Updates in Adaptive Optimizers — 18/02/2026 [USA]

  arXiv:2602.15322v1 Announce Type: cross Abstract: Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We chal...

  Related: #Machine learning optimization, #Large language model training, #Regularization techniques, #Curvature-aware algorithms