Neural Foundations · 2014
Adam: A Method for Stochastic Optimization
Introduced Adam, an optimizer that maintains per-parameter adaptive learning rates from running estimates of the first and second moments of the gradients, removing much of the manual learning-rate tuning and poor conditioning that hampered SGD on sparse or noisy gradients.
Editorial record
Plain-language summary
Adam keeps exponentially decaying averages of past gradients and squared gradients, applies bias correction to those estimates, and uses them to scale each parameter's step. This gives well-behaved step sizes across parameters with different gradient magnitudes and works with little tuning across many problems. Its combination of low memory cost and robust default hyperparameters made it the default optimizer for training deep networks.
Knowledge graph
Relationships
Antecedents
Depends onEvidence: Direct
Attention Is All You Need
Adam optimizer with warmup schedule
P-001 Sec 5.3
Source record
Provenance
- Record ID
- A-011
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1412.6980
- arXiv:1412.6980
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.