Skip to main content
Modern PyTorch Guide home page
Search...
⌘K
Official Docs
GitHub
GitHub
Search...
Navigation
Optimizers
SGD & Momentum
Foundations
Building Models
Performance
Domains
Production
Advanced
API Reference
Community
Forums
Architecture Patterns
Multi-Layer Perceptrons
Convolutional Neural Networks
RNNs & LSTMs
Gru
Transformers
Attention mechanisms
Autoencoders
Vae
Diffusion Models
Gans
Spatial transformer
Loss Functions
Regression Losses
Classification Losses
Contrastive Losses
Custom Loss Functions
Adversarial
Optimizers
Optimizers Overview
SGD & Momentum
Adam & Variants
Learning Rate Schedulers
Compiled optimizer
Zero redundancy
Training Loops
Basic Training Loop
Validation & Metrics
Checkpointing
Early Stopping
Mixed Precision Training
Gradient accumulation
Grad clipping
Debugging & Visualization
Common Errors
Gradient Issues
Profiling
TensorBoard Integration
Anomaly detection
Gradcheck
Visualizing gradients
Advanced Techniques
Parametrizations
Pruning
Knowledge distillation
Model ensembling
Per sample gradients
Parallel computing
Overview
Device Management
Cpu
Cuda basics
Mps
Xpu
Streams events
On this page
SGD & Momentum
Optimizers
SGD & Momentum
Classic optimizers that still work well
SGD & Momentum
SGD, momentum, Nesterov momentum, weight decay.
Optimizers Overview
Adam & Variants
⌘I
