#Gradient heterogeneity
Latest news articles tagged with "Gradient heterogeneity". Follow the timeline of events, related topics, and entities.
Articles (1)
-
🇺🇸 Understanding Transformer Optimization via Gradient Heterogeneity
[USA]
arXiv:2502.00213v4 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite t...
Related: #Transformers, #Optimization, #Adaptive optimizers, #Stochastic gradient descent