Skip to main content

Modern PyTorch Guide home page

Official Docs
GitHub
GitHub

Building Models

Community
Forums

Fine-tuning & Adaptation

Transfer Learning
PEFT & LoRA
QLoRA
RLHF
Warmstarting

Custom Operators

Custom Python Operators
C++ Extensions
CUDA Kernels
Triton Kernels
Dispatching
Backend integration
Double backward

Advanced Parallelism

Megatron-LM
DeepSpeed ZeRO
Expert Parallelism
3D Parallelism
Device mesh
Symmetric memory

Research Tools

torch.func
torch.fx
functorch
FlashAttention
Vmap
Jacobian hessian

Extending PyTorch

Overview
Autograd functions
Cpp frontend
Custom backends
Privateuse1

Sparse & Structured Tensors

Sparse tensors
Nested tensors
Masked tensors
Operations

Advanced Features

Complex numbers
Foreach map
Packaging
Hub
Autoload extensions
Torch modes

Low-Level & Internals

Fake tensors
Aten ir
Provenance tracking
Minifier
Logging

Expert Parallelism

Advanced Parallelism

Expert Parallelism

Mixture of Experts training strategies

Documentation Index
Fetch the complete documentation index at: https://newtorch.aboneda.com/llms.txt
Use this file to discover all available pages before exploring further.

Expert Parallelism

MoE architectures, expert routing, DeepSpeed MoE.

DeepSpeed ZeRO 3D Parallelism

⌘I

x github linkedin

Powered byThis documentation is built and hosted on Mintlify, a developer documentation platform