Skip to main content
Modern PyTorch Guide home page
Search...
⌘K
Official Docs
GitHub
GitHub
Search...
Navigation
Advanced Parallelism
Expert Parallelism
Foundations
Building Models
Performance
Domains
Production
Advanced
API Reference
Community
Forums
Fine-tuning & Adaptation
Transfer Learning
PEFT & LoRA
QLoRA
RLHF
Warmstarting
Custom Operators
Custom Python Operators
C++ Extensions
CUDA Kernels
Triton Kernels
Dispatching
Backend integration
Double backward
Advanced Parallelism
Megatron-LM
DeepSpeed ZeRO
Expert Parallelism
3D Parallelism
Device mesh
Symmetric memory
Research Tools
torch.func
torch.fx
functorch
FlashAttention
Vmap
Jacobian hessian
Extending PyTorch
Overview
Autograd functions
Cpp frontend
Custom backends
Privateuse1
Sparse & Structured Tensors
Sparse tensors
Nested tensors
Masked tensors
Operations
Advanced Features
Complex numbers
Foreach map
Packaging
Hub
Autoload extensions
Torch modes
Low-Level & Internals
Fake tensors
Aten ir
Provenance tracking
Minifier
Logging
On this page
Expert Parallelism
Advanced Parallelism
Expert Parallelism
Mixture of Experts training strategies
Expert Parallelism
MoE architectures, expert routing, DeepSpeed MoE.
DeepSpeed ZeRO
3D Parallelism
⌘I
