Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.

Skip to main content

Modern PyTorch Guide home page

Official Docs
GitHub
GitHub

Building Models

Community
Forums

Fine-tuning & Adaptation

Transfer Learning
PEFT & LoRA
QLoRA
RLHF
Warmstarting

Custom Operators

Custom Python Operators
C++ Extensions
CUDA Kernels
Triton Kernels
Dispatching
Backend integration
Double backward

Advanced Parallelism

Megatron-LM
DeepSpeed ZeRO
Expert Parallelism
3D Parallelism
Device mesh
Symmetric memory

Research Tools

torch.func
torch.fx
functorch
FlashAttention
Vmap
Jacobian hessian

Extending PyTorch

Overview
Autograd functions
Cpp frontend
Custom backends
Privateuse1

Sparse & Structured Tensors

Sparse tensors
Nested tensors
Masked tensors
Operations

Advanced Features

Complex numbers
Foreach map
Packaging
Hub
Autoload extensions
Torch modes

Low-Level & Internals

Fake tensors
Aten ir
Provenance tracking
Minifier
Logging

DeepSpeed ZeRO

Advanced Parallelism

DeepSpeed ZeRO

Zero Redundancy Optimizer for massive models

DeepSpeed ZeRO

ZeRO-1, ZeRO-2, ZeRO-3, ZeRO-Infinity.

Expert Parallelism

x github linkedin

Powered byThis documentation is built and hosted on Mintlify, a developer documentation platform