Your One-Page PyTorch Training Pipeline Cheat Sheet.

The entire engine of Deep Learning works by making tiny, continuous adjustments to a model’s weights.

This is my PyTorch training pipeline cheat sheet. If you spot any mistakes or have suggestions for improvement, feel free to leave a comment. I hope you find it helpful!

1. TENSOR

In PyTorch, a tensor is the fundamental data structure used to store and manipulate numerical data, acting as the primary building block for neural networks.

Input:

pythondata= [[1,2,3],[4,5,6]] x = torch.tensor(data)

Output:

pythontensor([[1, 2, 3], [4, 5, 6]])

Q. What’s Inside a Tensor?

Three things. Shape,** DType**, and** Device.**

• **Shape: **A tuple describing the dimensions (90% of error in PyTorch will be shape mismatch)

• **Dtype: **Data type of numbers.

• **Device: **Where the tensor lives. CPU or CUDA (GPU)

If your code breaks, it’s usually because one of these three doesn’t match.

Input:

python#attributes of tensor

tensor = torch.randn(2,3) print(“shape:”, tensor.shape) print(“datatype:”, tensor.dtype) print(“device:”, tensor.device)

Output:

pythonshape: torch.Size([2, 3]) datatype: torch.float32 device: cpu

By default, a tensor is just data. To make it a learnable parameter, you need to set requires_grad=True. Once enabled, PyTorch begins building a computation graph, recording every operation performed on that tensor so it can automatically compute gradients during backpropagation.

Input:

python#data vs parameter

#standard data tensor x_data = torch.tensor([[1.,2.],[3.,4.]])

#parameter tensor w = torch.tensor([[1.0],[2.0]], requires_grad=True)

print(“Data tensor requires_grad:”,x_data.requires_grad) print(“Parameter tensor requires_grad:”, w.requires_grad)

Output:

pythonData tensor requires_grad: False Parameter tensor requires_grad: True

Q. what is a paramater?

A special tensor that** requires_grad = True** by default, auto-registers with the model, handles all bookkeeping.

2. Models

A model is a** neural network architecture **defined as a Python object that inherits from the torch.nn.Module base class.

• A model is just an nn.Module containing layers.

• A layer is just a container for parameters that perform a mathematical operation.

• Learning is just updating parameters with zero_grad, backward, step.

• Model parameters (weights, biases) MUST be a float type (float 32)

Q. Why float32?

Weights need to change by tiny amounts. That’s impossible with integers. Floating point numbers allow the model to make microscopic improvements every iteration.

The simplest neural network is

y = X @ W + b

where

• y -> prediction

• X -> Input

• W -> Weights

• b -> Bias

Prediction = Input × Knowledge + Adjustment

3. Matrix Multiplication

X @ W

It is the heart of deep learning. Almost every modern neural network is built from repeated matrix multiplications.

Use the @ operator when building a linear layer. In PyTorch, @ performs matrix multiplication, which is the core operation behind every nn.Linear layer.

RULE: M1 COL = M2 ROW

Input:

python#Matrix Multiplication

#shape (2,3) m1 = torch.tensor([[1,2,3], [4,5,6]])

#shape {3,2) m2 = torch.tensor([[7,8], [9,10], [11,12]])

#shape (2,2) matrix_product = m1 @ m2 print(matrix_product)

Output:

pythontensor([[ 58, 64], [139, 154]])

Don’t confuse ‘@’ with ’ * ’…@ performs matrix multiplication, while * multiplies corresponding elements one by one.

4. AUTOGRAD (Automatic Differentiation)

torch.autograd is PyTorch’s automatic differentiation engine that powers neural network training. It dynamically builds a directed acyclic graph (DAG) during the forward pass to track tensor operations, allowing you to automatically compute gradients via backpropagation.

auto grad tell** “travel back from loss and calculate gradients for all parametes with requires_grad=True”.**

Input:

python#y=2x+1 #batch of data has 10 points N=10 #each data point has 1 input feature and 1 output value D_in = 1 D_out = 1 #create out input data X X = torch.randn(N, D_in) #create our true target labels y by using the “true” W,B #the “true” W is 2.0, the “true” b is 1.0 true_w = torch.tensor([[2.0]]) true_b = torch.tensor([[1.0]]) y_true = X @ true_w + true_b + torch.randn(N, D_out) * 1.0 #added littile noise

W = torch.randn(D_in, D_out, requires_grad = True) b = torch.randn(1, requires_grad = True) print(“Initial Weight W:\n”,W) print(“Initial bias b:\n”,b)

y_hat = X @ W + b

print(y_hat) #error error = y_hat - y_true squared_error = error ** 2 loss = squared_error.mean()

print(“Loss:”,loss,“\n”) loss.backward() print(“gradient for w:\n”, W.grad) print(“gradient for b:\n”, b.grad)

Output:

pythonInitial Weight W: tensor([[-0.5669]], requires_grad=True) Initial bias b: tensor([0.0011], requires_grad=True) tensor([[ 0.0604], [ 0.5645], [ 0.6193], [-0.2863], [ 1.2082], [ 0.6468], [ 0.5936], [-0.2940], [ 0.6108], [ 0.8328]], grad_fn=) Loss: tensor(6.5599, grad_fn=)

gradient for w: tensor([[-5.0230]]) gradient for b: tensor([1.8041])

**5. Backpropagation **(Backward Pass)

pythonloss.backward()

• ∂L/∂w = Gradient of the loss w.r.t. weight w

• ∂L/∂b = Gradient of the loss w.r.t. bias b

Computed automatically with loss.backward().

6. Gradient Descent (Optimization)

After computing the gradients, we update the model’s parameters to reduce the loss.

The update rule is:

θt+1=θt−η∇θL\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L

Where:

• θ (theta) = model parameters (weights & biases)

• η (eta)= learning rate

• ∇θL = gradient of the loss with respect to the parameters (w.grad,b.grad)

New Weight = Old Weight - Learning Rate × Gradient

Input:

python#training (gradient descent) #Hyperparameters learning_rate, epochs = 0.01, 100 #re-initialize parameters W,b = torch.randn(1,1, requires_grad=True), torch.randn(1, requires_grad=True) #Training Loop for epoch in range(epochs): #forward pass and loss y_hat = X @ W + b loss = torch.mean((y_hat - y_true)**2) #backward pass loss.backward() #update parameters with torch.no_grad(): W -= learning_rate * W.grad; b -= learning_rate * b.grad #zero gradients (reset for new learning cycle) W.grad.zero_(); b.grad.zero_() print(W.grad,“\n”,b.grad)

7. nn.Module

Instead of writing everything manually PyTorch provides.

• nn.Linear

• nn.ReLU

• nn.GELU

• nn.softmax

• nn.Embedding

• nn.layernorm

• nn.dropout

These are simply reusable building blocks.

Input:

python#NN.LINEAR #input has 1 feature ans the putput has 1 value D_in = 1 D_out = 1

linear_layer = torch.nn.Linear(in_features=D_in, out_features=D_out)

print(“Layer’s weight (W):” ,linear_layer.weight) print(“Layer’s bias (b):”, linear_layer.bias)

#forward pass y_hat_nn = linear_layer(X) print(“Output of nn.Liner (first 3):\n”, y_hat_nn[:3])

output:

pythonLayer’s weight (W): Parameter containing: tensor([[0.5193]], requires_grad=True) Layer’s bias (b): Parameter containing: tensor([-0.2911], requires_grad=True) Output of nn.Liner (first 3): tensor([[-0.3454], [-0.8072], [-0.8574]], grad_fn=)

Input:

python#NN.RELU (RECTIFIED LINEAR UNIT) #Rule: if an input is negative make it zero ‘ReLU(x)=max(0,x)’

relu = torch.nn.ReLU() sample_data = torch.tensor([-2.0,-5.0,0.0,0.5,2.0]) activated_data = relu(sample_data)

print(“Original Data:”, sample_data) print(“Data after ReLU:”, activated_data)

output:

pythonOriginal Data: tensor([-2.0000, -5.0000, 0.0000, 0.5000, 2.0000]) Data after ReLU: tensor([0.0000, 0.0000, 0.0000, 0.5000, 2.0000])

Input:

python#NN.GELU (GAUSSIAN ERROR LINEAR UNIT) #modern standard for transformers (GPT,Llama). A smoother, gently curving version of ReLU gelu = torch.nn.GELU() sample_data = torch.tensor([-2.0,-0.5,0.0,0.5,2.0]) activated_data = gelu(sample_data)

print(“Original Data:”, sample_data) print(“Data after gelu:”,activated_data)

output:

pythonOriginal Data: tensor([-2.0000, -0.5000, 0.0000, 0.5000, 2.0000]) Data after gelu: tensor([-0.0455, -0.1543, 0.0000, 0.3457, 1.9545])

Input:

python#NN.SOFTMAX #used on final output layer for classification #converts logits -> probability distribution

softmax = torch.nn.Softmax(dim=-1) logits = torch.tensor([[1.0,3.0,0.5,1.5],[-1.0,2.0,1.0,0.0]]) prob = softmax(logits) print(“output prob:”, prob) print(“sum of prob:”, prob[0].sum())

output:

pythonoutput prob: tensor([[0.0939, 0.6942, 0.0570, 0.1549], [0.0321, 0.6439, 0.2369, 0.0871]]) sum of prob: tensor(1.)

Input:

python#NN.EMBEDDING #turns word -> numbers

vocab_size = 10 embedding_dim = 3 embedding_layer = torch.nn.Embedding(vocab_size, embedding_dim)

input_ids = torch.tensor([[1,5,0,8]]) word_vector = embedding_layer(input_ids) word_vector

output:

pythontensor([[[-0.3875, -1.7026, -0.8399], [-0.3027, -0.0060, 0.7160], [-2.0736, 0.3490, -0.0605], [ 1.3206, 0.8779, 0.4340]]], grad_fn=)

Input:

python#nn.layernorm #prevents values from exploding/vanishing, it rescales to stable range norm_layer = torch.nn.LayerNorm(normalized_shape=3) input_features = torch.tensor([[1.0,2.0,3.0],[4.0,5.0,6.0]]) normalized_features = norm_layer(input_features)

print(“Mean (should be ~0):”, normalized_features.mean(dim=-1)) print(“std dev (should be ~1):”, normalized_features.std(dim=-1))

output:

pythonMean (should be ~0): tensor([0., 0.], grad_fn=) std dev (should be ~1): tensor([1.2247, 1.2247], grad_fn=)

Input:

python#NN.DROPOUT #prevent overfitting, randomly zero neurons during training forces network robustness

dropout_layer = torch.nn.Dropout(p=0.5) input_tensor = torch.ones(1,10)

dropout_layer.train() output_during_train = dropout_layer(input_tensor)

dropout_layer.eval() output_during_eval = dropout_layer(input_tensor)

8. The Rule of Three

• optimizer.zero_grad()

• loss.backward()

• optimizer.step()

9. DEMO TOY MODEL

pythonimport torch import torch.nn as nn

print(torch.cuda.is_available())

class LinearRegressionModel(nn.Module): def init(self,in_features,out_features): super().init() #inside the constructor, we define the layer self.linear_layer = nn.Linear(in_features, out_features)

def forward(self, x):
    #in the forward pass, we connect the layers
    return self.linear_layer(x)

model = LinearRegressionModel(in_features=1, out_features=1) print(“Model Architecture:”) print(model)

#torch.optim

import torch.optim as optim

#hyperaramater learning_rate = 0.01 optimizer = optim.Adam(model.parameters(), lr=learning_rate) loss_fn = nn.MSELoss()

epochs = 100

for epoch in range(epochs): #forward pass y_hat = model(X)

loss = loss_fn(y_hat,y_true)

optimizer.zero_grad()
loss.backward()
optimizer.step()

if epoch % 10 ==0:
    print("Epoch:",epoch,"Loss:",loss.item())

output:

TRUE Model Architecture: LinearRegressionModel( (linear_layer): Linear(in_features=1, out_features=1, bias=True) ) Epoch: 0 Loss: 11.082582473754883 Epoch: 10 Loss: 9.987239837646484 Epoch: 20 Loss: 8.982141494750977 Epoch: 30 Loss: 8.073999404907227 Epoch: 40 Loss: 7.263952732086182 Epoch: 50 Loss: 6.5487494468688965 Epoch: 60 Loss: 5.9224066734313965 Epoch: 70 Loss: 5.377523899078369 Epoch: 80 Loss: 4.90612268447876 Epoch: 90 Loss: 4.500123500823975

Hope this helps you understand PyTorch a little better. Happy building!