# Modern PyTorch Guide ## Docs - [Backend integration](https://newtorch.aboneda.com/advanced/custom-ops/backend-integration.md) - [C++ Extensions](https://newtorch.aboneda.com/advanced/custom-ops/cpp-extensions.md): High-performance custom operators - [CUDA Kernels](https://newtorch.aboneda.com/advanced/custom-ops/cuda-kernels.md): Write custom GPU kernels - [Dispatching](https://newtorch.aboneda.com/advanced/custom-ops/dispatching.md) - [Double backward](https://newtorch.aboneda.com/advanced/custom-ops/double-backward.md) - [Custom Python Operators](https://newtorch.aboneda.com/advanced/custom-ops/python-ops.md): Extend PyTorch with Python functions - [Triton Kernels](https://newtorch.aboneda.com/advanced/custom-ops/triton.md): Python-based GPU programming - [PEFT & LoRA](https://newtorch.aboneda.com/advanced/finetuning/peft-lora.md): Parameter-efficient fine-tuning techniques - [QLoRA](https://newtorch.aboneda.com/advanced/finetuning/qlora.md): Quantized LoRA for efficient fine-tuning - [RLHF](https://newtorch.aboneda.com/advanced/finetuning/rlhf.md): Reinforcement Learning from Human Feedback - [Transfer Learning](https://newtorch.aboneda.com/advanced/finetuning/transfer-learning.md): Leverage pretrained models for new tasks - [Warmstarting](https://newtorch.aboneda.com/advanced/finetuning/warmstarting.md) - [3D Parallelism](https://newtorch.aboneda.com/advanced/parallelism/3d-parallelism.md): Combining data, tensor, and pipeline parallelism - [Device mesh](https://newtorch.aboneda.com/advanced/parallelism/device-mesh.md) - [Expert Parallelism](https://newtorch.aboneda.com/advanced/parallelism/expert-parallel.md): Mixture of Experts training strategies - [Megatron-LM](https://newtorch.aboneda.com/advanced/parallelism/megatron.md): NVIDIA's large model training framework - [Symmetric memory](https://newtorch.aboneda.com/advanced/parallelism/symmetric-memory.md) - [DeepSpeed ZeRO](https://newtorch.aboneda.com/advanced/parallelism/zero.md): Zero Redundancy Optimizer for massive models - [FlashAttention](https://newtorch.aboneda.com/advanced/research/flash-attention.md): Memory-efficient attention implementation - [functorch](https://newtorch.aboneda.com/advanced/research/functorch.md): Composable function transforms (now torch.func) - [Jacobian hessian](https://newtorch.aboneda.com/advanced/research/jacobian-hessian.md) - [torch.func](https://newtorch.aboneda.com/advanced/research/torch-func.md): Functional transforms for JAX-style programming - [torch.fx](https://newtorch.aboneda.com/advanced/research/torch-fx.md): Symbolic tracing for graph transformations - [Vmap](https://newtorch.aboneda.com/advanced/research/vmap.md) - [Amp](https://newtorch.aboneda.com/api/advanced/amp.md) - [Autograd Profiler](https://newtorch.aboneda.com/api/advanced/autograd-profiler.md): Profile autograd operations - [C++ Extensions](https://newtorch.aboneda.com/api/advanced/cpp-extensions.md): Extend PyTorch with C++ and CUDA - [Custom Operators](https://newtorch.aboneda.com/api/advanced/custom-operators.md): Register custom operators with torch.library - [Hooks](https://newtorch.aboneda.com/api/advanced/hooks.md): Forward hooks, backward hooks, and gradient hooks - [Library](https://newtorch.aboneda.com/api/advanced/library.md) - [Memory Format](https://newtorch.aboneda.com/api/advanced/memory-format.md): channels_last, contiguous - memory layout optimization - [Overrides](https://newtorch.aboneda.com/api/advanced/overrides.md) - [Custom Autograd Functions](https://newtorch.aboneda.com/api/autograd/custom-functions.md): Extend autograd with custom forward/backward - [Autograd Functional API](https://newtorch.aboneda.com/api/autograd/functional-api.md): jacobian, hessian, vjp, jvp functions - [Gradient Computation](https://newtorch.aboneda.com/api/autograd/gradient-computation.md): backward, grad, and automatic differentiation - [Gradient Contexts](https://newtorch.aboneda.com/api/autograd/gradient-contexts.md): no_grad, enable_grad, set_grad_enabled contexts - [Autograd Profiler](https://newtorch.aboneda.com/api/autograd/profiler.md): Profile autograd operations - [AOT Autograd](https://newtorch.aboneda.com/api/compiler/aot-autograd.md): Ahead-of-time autograd compilation - [Compiler Backends](https://newtorch.aboneda.com/api/compiler/backends.md): Available compilation backends - [TorchDynamo](https://newtorch.aboneda.com/api/compiler/dynamo.md): Python bytecode analysis and graph capture - [TorchInductor](https://newtorch.aboneda.com/api/compiler/inductor.md): Code generation backend - [torch.compile](https://newtorch.aboneda.com/api/compiler/torch-compile.md): PyTorch 2.0 compilation API - [Indexing & Slicing](https://newtorch.aboneda.com/api/core/indexing-slicing.md): Advanced indexing, gathering, scattering, and masking operations - [Tensor Attributes](https://newtorch.aboneda.com/api/core/tensor-attributes.md): Shape, dtype, device, requires_grad, and other tensor properties - [Tensor Creation](https://newtorch.aboneda.com/api/core/tensor-creation.md): Create tensors with zeros, ones, arange, linspace, and more - [Tensor Manipulation](https://newtorch.aboneda.com/api/core/tensor-manipulation.md): Reshape, transpose, permute, stack, and manipulate tensors - [Tensor size](https://newtorch.aboneda.com/api/core/tensor-size.md) - [Tensor Views](https://newtorch.aboneda.com/api/core/tensor-views.md): View, as_strided, expand, and memory-efficient tensor operations - [Data Loading](https://newtorch.aboneda.com/api/data/data-loading.md): DataLoader configuration and optimization - [Datasets](https://newtorch.aboneda.com/api/data/datasets.md): Dataset, IterableDataset, and data abstractions - [Samplers](https://newtorch.aboneda.com/api/data/samplers.md): RandomSampler, SequentialSampler, DistributedSampler - [Transforms](https://newtorch.aboneda.com/api/data/transforms.md): Data augmentation and preprocessing transforms - [Accelerator](https://newtorch.aboneda.com/api/device/accelerator.md) - [Backends](https://newtorch.aboneda.com/api/device/backends.md) - [CPU](https://newtorch.aboneda.com/api/device/cpu.md): CPU-specific operations and optimizations - [CUDA](https://newtorch.aboneda.com/api/device/cuda.md): NVIDIA GPU operations and management - [Memory Management](https://newtorch.aboneda.com/api/device/memory-management.md): Memory allocation, caching, and optimization - [MPS](https://newtorch.aboneda.com/api/device/mps.md): Apple Silicon GPU support - [Mtia](https://newtorch.aboneda.com/api/device/mtia.md) - [Streams & Events](https://newtorch.aboneda.com/api/device/streams-events.md): CUDA streams and synchronization events - [XPU](https://newtorch.aboneda.com/api/device/xpu.md): Intel GPU support - [Algorithms](https://newtorch.aboneda.com/api/distributed/algorithms.md) - [Basic Distributed](https://newtorch.aboneda.com/api/distributed/basic-distributed.md): init_process_group and distributed primitives - [Distributed Checkpoint](https://newtorch.aboneda.com/api/distributed/checkpoint.md): Save and load distributed model states - [DistributedDataParallel](https://newtorch.aboneda.com/api/distributed/ddp.md): Multi-GPU training with DDP - [Elastic](https://newtorch.aboneda.com/api/distributed/elastic.md) - [FullyShardedDataParallel](https://newtorch.aboneda.com/api/distributed/fsdp.md): Memory-efficient distributed training with FSDP - [Pipeline Parallelism](https://newtorch.aboneda.com/api/distributed/pipeline-parallel.md): Pipeline model parallelism - [RPC Framework](https://newtorch.aboneda.com/api/distributed/rpc.md): Remote procedure call for distributed training - [Tensor](https://newtorch.aboneda.com/api/distributed/tensor.md) - [Tensor Parallelism](https://newtorch.aboneda.com/api/distributed/tensor-parallel.md): Parallelize tensors across devices - [ONNX Export](https://newtorch.aboneda.com/api/export/onnx.md): Export to ONNX format - [Export Quantization](https://newtorch.aboneda.com/api/export/quantization.md): Quantize models for export - [torch.export](https://newtorch.aboneda.com/api/export/torch-export.md): Export models for deployment - [functionalize](https://newtorch.aboneda.com/api/func/functionalize.md): Convert stateful to functional operations - [grad](https://newtorch.aboneda.com/api/func/grad.md): Functional gradient computation - [jacobian](https://newtorch.aboneda.com/api/func/jacobian.md): Compute Jacobian matrices - [vmap](https://newtorch.aboneda.com/api/func/vmap.md): Vectorized map for batched operations - [Functional Activations](https://newtorch.aboneda.com/api/functional/activation-functions.md): Functional activation functions - [Functional Convolutions](https://newtorch.aboneda.com/api/functional/convolution-ops.md): Functional convolution operations - [Functional Loss Functions](https://newtorch.aboneda.com/api/functional/loss-functions.md): Functional loss functions - [NN Functional Overview](https://newtorch.aboneda.com/api/functional/nn-functional.md): torch.nn.functional module overview - [Functional Normalization](https://newtorch.aboneda.com/api/functional/normalization-ops.md): Functional normalization operations - [Functional Pooling](https://newtorch.aboneda.com/api/functional/pooling-ops.md): Functional pooling operations - [JIT Optimization](https://newtorch.aboneda.com/api/jit/optimization.md): Optimize JIT compiled models - [Scripting](https://newtorch.aboneda.com/api/jit/scripting.md): torch.jit.script for model compilation - [TorchScript](https://newtorch.aboneda.com/api/jit/torchscript.md): TorchScript overview and usage - [Tracing](https://newtorch.aboneda.com/api/jit/tracing.md): torch.jit.trace for model compilation - [Classification Losses](https://newtorch.aboneda.com/api/loss/classification-losses.md): CrossEntropyLoss, NLLLoss, BCELoss for classification - [Embedding Losses](https://newtorch.aboneda.com/api/loss/embedding-losses.md): TripletMarginLoss, CosineEmbeddingLoss for metric learning - [Other Losses](https://newtorch.aboneda.com/api/loss/other-losses.md): CTCLoss, PoissonNLLLoss, and specialized losses - [Regression Losses](https://newtorch.aboneda.com/api/loss/regression-losses.md): MSELoss, L1Loss, SmoothL1Loss, HuberLoss for regression tasks - [Bitwise Operations](https://newtorch.aboneda.com/api/math/bitwise-ops.md): Bitwise AND, OR, XOR, NOT, and shift operations - [Comparison Operations](https://newtorch.aboneda.com/api/math/comparison-ops.md): Equal, greater than, less than, allclose, and comparison functions - [Logical Operations](https://newtorch.aboneda.com/api/math/logical-ops.md): AND, OR, NOT, XOR operations on boolean tensors - [Other ops](https://newtorch.aboneda.com/api/math/other-ops.md) - [Pointwise Operations](https://newtorch.aboneda.com/api/math/pointwise-ops.md): Element-wise math: add, mul, sin, exp, log, and 100+ operations - [Reduction Operations](https://newtorch.aboneda.com/api/math/reduction-ops.md): Sum, mean, std, max, min, argmax, and statistical operations - [Spectral Operations](https://newtorch.aboneda.com/api/math/spectral-ops.md): STFT, iSTFT, and spectral analysis functions - [Nested Operations](https://newtorch.aboneda.com/api/nested/nested-ops.md): Operations on nested tensors - [Nested Tensors](https://newtorch.aboneda.com/api/nested/nested-tensors.md): Tensors with varying dimensions for irregular data - [Activation Functions](https://newtorch.aboneda.com/api/nn/activation-functions.md): ReLU, GELU, Sigmoid, Tanh, Softmax, and 25+ activations - [Attention layers](https://newtorch.aboneda.com/api/nn/attention-layers.md) - [Containers](https://newtorch.aboneda.com/api/nn/containers.md): Module, Sequential, ModuleList, ModuleDict for building networks - [Convolution Layers](https://newtorch.aboneda.com/api/nn/convolution-layers.md): Conv1d, Conv2d, Conv3d, and transposed convolutions - [Distance Functions](https://newtorch.aboneda.com/api/nn/distance-functions.md): CosineSimilarity, PairwiseDistance for similarity metrics - [Dropout Layers](https://newtorch.aboneda.com/api/nn/dropout-layers.md): Dropout, AlphaDropout for regularization - [Embedding Layers](https://newtorch.aboneda.com/api/nn/embedding-layers.md): Embedding, EmbeddingBag for discrete inputs - [Init](https://newtorch.aboneda.com/api/nn/init.md) - [Linear Layers](https://newtorch.aboneda.com/api/nn/linear-layers.md): Linear, Bilinear, and Identity layers - [Normalization Layers](https://newtorch.aboneda.com/api/nn/normalization-layers.md): BatchNorm, LayerNorm, GroupNorm, InstanceNorm for training stability - [Padding Layers](https://newtorch.aboneda.com/api/nn/padding-layers.md): ReflectionPad, ReplicationPad, ZeroPad, ConstantPad - [Pooling Layers](https://newtorch.aboneda.com/api/nn/pooling-layers.md): MaxPool, AvgPool, AdaptivePool for spatial downsampling - [Recurrent Layers](https://newtorch.aboneda.com/api/nn/recurrent-layers.md): RNN, LSTM, GRU for sequence modeling - [Transformer Layers](https://newtorch.aboneda.com/api/nn/transformer-layers.md): Transformer, MultiheadAttention, and encoder/decoder layers - [Upsampling Layers](https://newtorch.aboneda.com/api/nn/upsampling-layers.md): Upsample, PixelShuffle, PixelUnshuffle for resolution increase - [Vision Layers](https://newtorch.aboneda.com/api/nn/vision-layers.md): Vision-specific layers and transformations - [Learning Rate Schedulers](https://newtorch.aboneda.com/api/optim/lr-schedulers.md): StepLR, CosineAnnealing, OneCycleLR schedulers - [Optimizer API](https://newtorch.aboneda.com/api/optim/optimizer-api.md): Optimizer base class and interface - [Optimizers](https://newtorch.aboneda.com/api/optim/optimizers.md): SGD, Adam, AdamW, and all optimization algorithms - [Anomaly Detection](https://newtorch.aboneda.com/api/profiling/anomaly-detection.md): Detect NaN and other anomalies - [Bottleneck](https://newtorch.aboneda.com/api/profiling/bottleneck.md): Identify performance bottlenecks - [Debugging](https://newtorch.aboneda.com/api/profiling/debugging.md): Debug PyTorch programs - [Flop counter](https://newtorch.aboneda.com/api/profiling/flop-counter.md) - [Monitor](https://newtorch.aboneda.com/api/profiling/monitor.md) - [Profiler](https://newtorch.aboneda.com/api/profiling/profiler.md): PyTorch profiler for performance analysis - [Fake Quantize](https://newtorch.aboneda.com/api/quantization/fake-quantize.md): Quantization-aware training - [Observers](https://newtorch.aboneda.com/api/quantization/observers.md): Quantization observers for calibration - [QConfig](https://newtorch.aboneda.com/api/quantization/qconfig.md): Quantization configuration - [Quantization API](https://newtorch.aboneda.com/api/quantization/quantization-api.md): Model quantization for efficiency - [Quantized Modules](https://newtorch.aboneda.com/api/quantization/quantized-modules.md): Quantized neural network modules - [Distributions](https://newtorch.aboneda.com/api/random/distributions.md): Probability distributions for sampling - [Random Sampling](https://newtorch.aboneda.com/api/random/random-sampling.md): rand, randn, randint, and random generation - [Random Seed](https://newtorch.aboneda.com/api/random/random-seed.md): Seed management and reproducibility - [Checkpointing](https://newtorch.aboneda.com/api/serialization/checkpoint.md): Model checkpointing best practices - [Save & Load](https://newtorch.aboneda.com/api/serialization/save-load.md): torch.save and torch.load for persistence - [State Dict](https://newtorch.aboneda.com/api/serialization/state-dict.md): Working with model state dictionaries - [Storage](https://newtorch.aboneda.com/api/serialization/storage.md) - [FFT](https://newtorch.aboneda.com/api/signal/fft.md): Fast Fourier Transform operations - [Signal ops](https://newtorch.aboneda.com/api/signal/signal-ops.md) - [Spectral Analysis](https://newtorch.aboneda.com/api/signal/spectral.md): Spectral analysis functions - [Window Functions](https://newtorch.aboneda.com/api/signal/windows.md): Windowing functions for signal processing - [Sparse Creation](https://newtorch.aboneda.com/api/sparse/sparse-creation.md): Create sparse tensors - [Sparse Formats](https://newtorch.aboneda.com/api/sparse/sparse-formats.md): COO, CSR, CSC, BSR sparse formats - [Sparse Operations](https://newtorch.aboneda.com/api/sparse/sparse-ops.md): Operations on sparse tensors - [Bessel Functions](https://newtorch.aboneda.com/api/special/bessel-functions.md): i0, i1 - modified Bessel functions - [Error Functions](https://newtorch.aboneda.com/api/special/error-functions.md): erf, erfc, erfinv - error function family - [Gamma Functions](https://newtorch.aboneda.com/api/special/gamma-functions.md): gamma, gammaln, digamma, polygamma - [Other Special Functions](https://newtorch.aboneda.com/api/special/other-functions.md): expit, logit, xlog1py, xlogy - [Test Assertions](https://newtorch.aboneda.com/api/testing/assertions.md): assert_close, assert_equal - testing utilities - [Gradient Checking](https://newtorch.aboneda.com/api/testing/gradcheck.md): gradcheck, gradgradcheck - verify gradient computation - [Testing Utils](https://newtorch.aboneda.com/api/testing/utils.md): Utilities for writing PyTorch tests - [Benchmark Utils](https://newtorch.aboneda.com/api/utils/benchmark.md): Timer, Compare - microbenchmarking utilities - [Checkpointing Utils](https://newtorch.aboneda.com/api/utils/checkpoint.md): Gradient checkpointing for memory efficiency - [Collect env](https://newtorch.aboneda.com/api/utils/collect-env.md) - [C++ Extensions](https://newtorch.aboneda.com/api/utils/cpp-extension.md): Build custom C++/CUDA operators - [Data utils](https://newtorch.aboneda.com/api/utils/data-utils.md) - [Deterministic](https://newtorch.aboneda.com/api/utils/deterministic.md) - [Dlpack](https://newtorch.aboneda.com/api/utils/dlpack.md) - [Hipify](https://newtorch.aboneda.com/api/utils/hipify.md) - [Jit utils](https://newtorch.aboneda.com/api/utils/jit-utils.md) - [Mobile Optimizer](https://newtorch.aboneda.com/api/utils/mobile-optimizer.md): Optimize models for mobile deployment - [Model zoo](https://newtorch.aboneda.com/api/utils/model-zoo.md) - [Module tracker](https://newtorch.aboneda.com/api/utils/module-tracker.md) - [TensorBoard](https://newtorch.aboneda.com/api/utils/tensorboard.md): Visualization and experiment tracking - [Attention mechanisms](https://newtorch.aboneda.com/building/architectures/attention-mechanisms.md) - [Autoencoders](https://newtorch.aboneda.com/building/architectures/autoencoders.md): Unsupervised learning with encoder-decoder pairs - [Convolutional Neural Networks](https://newtorch.aboneda.com/building/architectures/cnn.md): CNNs for image and spatial data processing - [Diffusion Models](https://newtorch.aboneda.com/building/architectures/diffusion.md): State-of-the-art generative models - [Gans](https://newtorch.aboneda.com/building/architectures/gans.md) - [Gru](https://newtorch.aboneda.com/building/architectures/gru.md) - [Multi-Layer Perceptrons](https://newtorch.aboneda.com/building/architectures/mlp.md): Fully connected networks - the simplest neural architecture - [RNNs & LSTMs](https://newtorch.aboneda.com/building/architectures/rnn-lstm.md): Recurrent networks for sequential data - [Spatial transformer](https://newtorch.aboneda.com/building/architectures/spatial-transformer.md) - [Transformers](https://newtorch.aboneda.com/building/architectures/transformer.md): Attention is all you need - the dominant architecture - [Vae](https://newtorch.aboneda.com/building/architectures/vae.md) - [Anomaly detection](https://newtorch.aboneda.com/building/debugging/anomaly-detection.md) - [Common Errors](https://newtorch.aboneda.com/building/debugging/common-errors.md): Debugging typical PyTorch mistakes - [Gradcheck](https://newtorch.aboneda.com/building/debugging/gradcheck.md) - [Gradient Issues](https://newtorch.aboneda.com/building/debugging/gradient-issues.md): Vanishing, exploding gradients and solutions - [Profiling](https://newtorch.aboneda.com/building/debugging/profiling.md): Find bottlenecks in your training code - [TensorBoard Integration](https://newtorch.aboneda.com/building/debugging/tensorboard.md): Visualize training with TensorBoard - [Visualizing gradients](https://newtorch.aboneda.com/building/debugging/visualizing-gradients.md) - [Adversarial](https://newtorch.aboneda.com/building/losses/adversarial.md) - [Classification Losses](https://newtorch.aboneda.com/building/losses/classification.md): Cross-entropy, BCE, focal loss for classification - [Contrastive Losses](https://newtorch.aboneda.com/building/losses/contrastive.md): Triplet loss, InfoNCE, and self-supervised objectives - [Custom Loss Functions](https://newtorch.aboneda.com/building/losses/custom.md): Building your own differentiable loss functions - [Regression Losses](https://newtorch.aboneda.com/building/losses/regression.md): MSE, MAE, Huber, and other regression objectives - [Adam & Variants](https://newtorch.aboneda.com/building/optimizers/adam-variants.md): AdamW, AdaFactor, LAMB, and modern optimizers - [Compiled optimizer](https://newtorch.aboneda.com/building/optimizers/compiled-optimizer.md) - [Learning Rate Schedulers](https://newtorch.aboneda.com/building/optimizers/lr-schedulers.md): Dynamic learning rate adjustments - [Optimizers Overview](https://newtorch.aboneda.com/building/optimizers/overview.md): How gradient descent variants work - [SGD & Momentum](https://newtorch.aboneda.com/building/optimizers/sgd-momentum.md): Classic optimizers that still work well - [Zero redundancy](https://newtorch.aboneda.com/building/optimizers/zero-redundancy.md) - [Overview](https://newtorch.aboneda.com/building/parallel-computing/overview.md): An overview of Data Parallelism and Model Parallelism in PyTorch. - [Basic Training Loop](https://newtorch.aboneda.com/building/training/basic-loop.md): The fundamental train-evaluate-repeat cycle - [Checkpointing](https://newtorch.aboneda.com/building/training/checkpointing.md): Saving and loading model state - [Early Stopping](https://newtorch.aboneda.com/building/training/early-stopping.md): Prevent overfitting by stopping at the right time - [Grad clipping](https://newtorch.aboneda.com/building/training/grad-clipping.md) - [Gradient accumulation](https://newtorch.aboneda.com/building/training/gradient-accumulation.md) - [Mixed Precision Training](https://newtorch.aboneda.com/building/training/mixed-precision.md): FP16/BF16 training for faster computation - [Validation & Metrics](https://newtorch.aboneda.com/building/training/validation.md): Evaluating model performance during training - [Audio Classification](https://newtorch.aboneda.com/domains/audio/audio-classification.md): Classifying sounds and music - [Speech Recognition](https://newtorch.aboneda.com/domains/audio/speech-recognition.md): Automatic speech recognition (ASR) - [TorchAudio](https://newtorch.aboneda.com/domains/audio/torchaudio.md): Audio processing with PyTorch - [Dcgan](https://newtorch.aboneda.com/domains/generative/dcgan.md) - [HuggingFace Diffusers](https://newtorch.aboneda.com/domains/generative/diffusers.md): State-of-the-art diffusion models library - [Diffusion Basics](https://newtorch.aboneda.com/domains/generative/diffusion-basics.md): Understanding diffusion models from scratch - [Generative Adversarial Networks](https://newtorch.aboneda.com/domains/generative/gans.md): Generator vs discriminator training - [Variational Autoencoders](https://newtorch.aboneda.com/domains/generative/vae.md): Latent variable generative models - [Attention mechanisms](https://newtorch.aboneda.com/domains/nlp/attention-mechanisms.md) - [Language Modeling](https://newtorch.aboneda.com/domains/nlp/language-modeling.md): Training models to predict text - [LLM Inference](https://newtorch.aboneda.com/domains/nlp/llm-inference.md): Efficient inference for large language models - [Nlp from scratch](https://newtorch.aboneda.com/domains/nlp/nlp-from-scratch.md) - [Text Classification](https://newtorch.aboneda.com/domains/nlp/text-classification.md): Sentiment, topic, and intent classification - [Tokenizers](https://newtorch.aboneda.com/domains/nlp/tokenizers.md): Text preprocessing for NLP models - [HuggingFace Transformers](https://newtorch.aboneda.com/domains/nlp/transformers-hf.md): The go-to library for NLP and LLMs - [Deep Q-Networks](https://newtorch.aboneda.com/domains/rl/dqn.md): Value-based reinforcement learning - [RL Environments](https://newtorch.aboneda.com/domains/rl/environments.md): Gym, MuJoCo, and custom environments - [Mario agent](https://newtorch.aboneda.com/domains/rl/mario-agent.md) - [Policy Gradient Methods](https://newtorch.aboneda.com/domains/rl/policy-gradient.md): REINFORCE, PPO, A2C algorithms - [Ppo](https://newtorch.aboneda.com/domains/rl/ppo.md) - [TorchRL](https://newtorch.aboneda.com/domains/rl/torchrl.md): PyTorch's reinforcement learning library - [Adversarial examples](https://newtorch.aboneda.com/domains/vision/adversarial-examples.md) - [Image Classification](https://newtorch.aboneda.com/domains/vision/image-classification.md): Classifying images into categories - [Object Detection](https://newtorch.aboneda.com/domains/vision/object-detection.md): Detecting and localizing objects in images - [Image Segmentation](https://newtorch.aboneda.com/domains/vision/segmentation.md): Pixel-level classification - [timm (PyTorch Image Models)](https://newtorch.aboneda.com/domains/vision/timm.md): State-of-the-art vision models collection - [TorchVision](https://newtorch.aboneda.com/domains/vision/torchvision.md): PyTorch's official computer vision library - [Transfer learning](https://newtorch.aboneda.com/domains/vision/transfer-learning.md) - [Autograd Basics](https://newtorch.aboneda.com/foundations/autograd/basics.md): Automatic differentiation fundamentals in PyTorch - [Computational Graph](https://newtorch.aboneda.com/foundations/autograd/computational-graph.md): Understanding the dynamic computation graph - [Custom Autograd Functions](https://newtorch.aboneda.com/foundations/autograd/custom-functions.md): Define your own forward and backward passes - [Forward mode](https://newtorch.aboneda.com/foundations/autograd/forward-mode.md) - [Gradients Explained](https://newtorch.aboneda.com/foundations/autograd/gradients-explained.md): retain_grad, leaf tensors, gradient accumulation deep dive - [Requires grad](https://newtorch.aboneda.com/foundations/autograd/requires-grad.md) - [DataLoaders](https://newtorch.aboneda.com/foundations/data/dataloaders.md): Batching, shuffling, and parallel data loading - [Datasets](https://newtorch.aboneda.com/foundations/data/datasets.md): torch.utils.data.Dataset for custom data loading - [Pin memory](https://newtorch.aboneda.com/foundations/data/pin-memory.md) - [Samplers](https://newtorch.aboneda.com/foundations/data/samplers.md): Control how data is sampled from datasets - [Transforms](https://newtorch.aboneda.com/foundations/data/transforms.md): Data augmentation and preprocessing pipelines - [Webdataset](https://newtorch.aboneda.com/foundations/data/webdataset.md) - [Overview](https://newtorch.aboneda.com/foundations/etl/overview.md): An overview of the ETL process in PyTorch. - [Forward Pass](https://newtorch.aboneda.com/foundations/nn/forward-pass.md): How data flows through your network - [Hooks](https://newtorch.aboneda.com/foundations/nn/hooks.md): Inspect and modify activations and gradients - [Init strategies](https://newtorch.aboneda.com/foundations/nn/init-strategies.md) - [nn.Module Basics](https://newtorch.aboneda.com/foundations/nn/module-basics.md): The building block of all neural networks in PyTorch - [Parameters & Buffers](https://newtorch.aboneda.com/foundations/nn/parameters-buffers.md): Trainable parameters vs persistent buffers - [State dict](https://newtorch.aboneda.com/foundations/nn/state-dict.md) - [Broadcasting](https://newtorch.aboneda.com/foundations/tensors/broadcasting.md): How PyTorch automatically expands tensors for operations - [Tensor Creation](https://newtorch.aboneda.com/foundations/tensors/creation.md): torch.tensor, zeros, ones, rand, arange, linspace and more - [Indexing & Slicing](https://newtorch.aboneda.com/foundations/tensors/indexing-slicing.md): Access and modify tensor elements with advanced indexing - [Introduction to Tensors](https://newtorch.aboneda.com/foundations/tensors/introduction.md): What tensors are and why they're the foundation of PyTorch - [Named tensors](https://newtorch.aboneda.com/foundations/tensors/named-tensors.md) - [Notes](https://newtorch.aboneda.com/foundations/tensors/notes.md) - [Tensor Operations](https://newtorch.aboneda.com/foundations/tensors/operations.md): Mathematical operations, reshaping, concatenation, and more - [Tensor attributes](https://newtorch.aboneda.com/foundations/tensors/tensor-attributes.md) - [Views strides](https://newtorch.aboneda.com/foundations/tensors/views-strides.md) - [Overview](https://newtorch.aboneda.com/index.md): Comprehensive guide to mastering all PyTorch APIs with examples and best practices - [Aot autograd](https://newtorch.aboneda.com/performance/compile/aot-autograd.md) - [Compiler Backends](https://newtorch.aboneda.com/performance/compile/backends.md): inductor, cudagraphs, and other backends - [Cache control](https://newtorch.aboneda.com/performance/compile/cache-control.md) - [Compiled autograd](https://newtorch.aboneda.com/performance/compile/compiled-autograd.md) - [Dynamic Shapes](https://newtorch.aboneda.com/performance/compile/dynamic-shapes.md): Handling variable input sizes - [Graph Breaks](https://newtorch.aboneda.com/performance/compile/graph-breaks.md): What breaks compilation and how to fix it - [Inductor](https://newtorch.aboneda.com/performance/compile/inductor.md) - [Introduction to torch.compile](https://newtorch.aboneda.com/performance/compile/introduction.md): PyTorch 2.0's game-changing JIT compiler - [Regional compilation](https://newtorch.aboneda.com/performance/compile/regional-compilation.md) - [Troubleshooting](https://newtorch.aboneda.com/performance/compile/troubleshooting.md) - [Comm hooks](https://newtorch.aboneda.com/performance/distributed/comm-hooks.md) - [Context parallel](https://newtorch.aboneda.com/performance/distributed/context-parallel.md) - [DistributedDataParallel (DDP)](https://newtorch.aboneda.com/performance/distributed/ddp.md): The standard for multi-GPU training - [DeepSpeed Integration](https://newtorch.aboneda.com/performance/distributed/deepspeed.md): Microsoft's optimization library for PyTorch - [Fully Sharded Data Parallel (FSDP)](https://newtorch.aboneda.com/performance/distributed/fsdp.md): Shard model parameters across GPUs - [Fsdp2](https://newtorch.aboneda.com/performance/distributed/fsdp2.md) - [Join context](https://newtorch.aboneda.com/performance/distributed/join-context.md) - [Monarch](https://newtorch.aboneda.com/performance/distributed/monarch.md) - [Distributed Training Overview](https://newtorch.aboneda.com/performance/distributed/overview.md): Scale training across multiple GPUs and nodes - [Pipeline Parallelism](https://newtorch.aboneda.com/performance/distributed/pipeline-parallel.md): Split model layers across GPUs - [Rpc framework](https://newtorch.aboneda.com/performance/distributed/rpc-framework.md) - [Tensor Parallelism](https://newtorch.aboneda.com/performance/distributed/tensor-parallel.md): Split individual layers across GPUs - [Zero redundancy](https://newtorch.aboneda.com/performance/distributed/zero-redundancy.md) - [Amp](https://newtorch.aboneda.com/performance/hardware/amp.md) - [Channels last](https://newtorch.aboneda.com/performance/hardware/channels-last.md) - [Cuda semantics](https://newtorch.aboneda.com/performance/hardware/cuda-semantics.md) - [Hip rocm](https://newtorch.aboneda.com/performance/hardware/hip-rocm.md) - [Intel gpu](https://newtorch.aboneda.com/performance/hardware/intel-gpu.md) - [GPU Memory Management](https://newtorch.aboneda.com/performance/hardware/memory-management.md): Optimize memory usage on GPU - [Mps backend](https://newtorch.aboneda.com/performance/hardware/mps-backend.md) - [Multi-GPU Setup](https://newtorch.aboneda.com/performance/hardware/multi-gpu.md): Using multiple GPUs effectively - [NVLink & InfiniBand](https://newtorch.aboneda.com/performance/hardware/nvlink-infiniband.md): High-bandwidth GPU interconnects - [Activation checkpointing](https://newtorch.aboneda.com/performance/memory/activation-checkpointing.md) - [Gradient Checkpointing](https://newtorch.aboneda.com/performance/memory/gradient-checkpointing.md): Trade compute for memory - [Mixed Precision Deep Dive](https://newtorch.aboneda.com/performance/memory/mixed-precision.md): FP16, BF16, and precision considerations - [CPU/Disk Offloading](https://newtorch.aboneda.com/performance/memory/offloading.md): Extend GPU memory with CPU/NVMe - [Backends](https://newtorch.aboneda.com/performance/quantization/backends.md) - [Dynamic Quantization](https://newtorch.aboneda.com/performance/quantization/dynamic.md): Quantize weights, compute in FP32 - [Fx graph mode](https://newtorch.aboneda.com/performance/quantization/fx-graph-mode.md) - [Quantization Overview](https://newtorch.aboneda.com/performance/quantization/overview.md): Reduce model size and increase inference speed - [Post-Training Quantization](https://newtorch.aboneda.com/performance/quantization/post-training.md): Quantize after training without retraining - [Quantization-Aware Training](https://newtorch.aboneda.com/performance/quantization/qat.md): Train with quantization in mind - [Study Plan 1](https://newtorch.aboneda.com/plan.md) - [HuggingFace Accelerate](https://newtorch.aboneda.com/production/ecosystem/accelerate.md): Simplify distributed training setup - [Ax nas](https://newtorch.aboneda.com/production/ecosystem/ax-nas.md) - [HuggingFace Datasets](https://newtorch.aboneda.com/production/ecosystem/datasets-hf.md): Efficient data loading for ML - [HuggingFace Hub](https://newtorch.aboneda.com/production/ecosystem/huggingface-hub.md): Share and discover models and datasets - [HuggingFace Optimum](https://newtorch.aboneda.com/production/ecosystem/optimum.md): Hardware-optimized inference - [Peft](https://newtorch.aboneda.com/production/ecosystem/peft.md) - [Ray tune](https://newtorch.aboneda.com/production/ecosystem/ray-tune.md) - [AOT Inductor](https://newtorch.aboneda.com/production/export/aot-inductor.md): Ahead-of-time compilation for deployment - [ONNX Export](https://newtorch.aboneda.com/production/export/onnx.md): Export to Open Neural Network Exchange format - [Onnx extending](https://newtorch.aboneda.com/production/export/onnx-extending.md) - [Pt2 archive](https://newtorch.aboneda.com/production/export/pt2-archive.md) - [torch.export](https://newtorch.aboneda.com/production/export/torch-export.md): PyTorch 2.0's new export mechanism - [TorchScript](https://newtorch.aboneda.com/production/export/torchscript.md): Serialize models for production deployment - [Async checkpointing](https://newtorch.aboneda.com/production/mlops/async-checkpointing.md) - [Checkpoint dcp](https://newtorch.aboneda.com/production/mlops/checkpoint-dcp.md) - [CI/CD for ML](https://newtorch.aboneda.com/production/mlops/ci-cd.md): Continuous integration for ML projects - [Docker for ML](https://newtorch.aboneda.com/production/mlops/docker.md): Containerize your training and inference - [Kubernetes for ML](https://newtorch.aboneda.com/production/mlops/kubernetes.md): Orchestrate ML workloads at scale - [Large scale deployments](https://newtorch.aboneda.com/production/mlops/large-scale-deployments.md) - [Reproducibility](https://newtorch.aboneda.com/production/mlops/reproducibility.md): Ensure consistent results across runs - [Libtorch abi](https://newtorch.aboneda.com/production/serving/libtorch-abi.md) - [Mobile optimization](https://newtorch.aboneda.com/production/serving/mobile-optimization.md) - [Raspberry pi](https://newtorch.aboneda.com/production/serving/raspberry-pi.md) - [TensorRT](https://newtorch.aboneda.com/production/serving/tensorrt.md): NVIDIA's high-performance inference optimizer - [TorchServe](https://newtorch.aboneda.com/production/serving/torchserve.md): PyTorch's official model serving framework - [Triton Inference Server](https://newtorch.aboneda.com/production/serving/triton.md): NVIDIA's production inference platform - [vLLM](https://newtorch.aboneda.com/production/serving/vllm.md): High-throughput LLM inference engine - [MLflow](https://newtorch.aboneda.com/production/tracking/mlflow.md): Open-source ML lifecycle management - [TensorBoard](https://newtorch.aboneda.com/production/tracking/tensorboard.md): Visualization toolkit for training - [Weights & Biases](https://newtorch.aboneda.com/production/tracking/wandb.md): Experiment tracking and MLOps platform - [Study Plan 2](https://newtorch.aboneda.com/study-guide.md) ## OpenAPI Specs - [openapi](https://newtorch.aboneda.com/api-reference/openapi.json) ## Optional - [Community](https://pytorch.org/get-started/community) - [Forums](https://discuss.pytorch.org)