Math for Gradient Descent — Worked Examples with Numbers

Part 1: Derivatives, Partial Derivatives, Chain Rule, and Full Gradient Descent with actual numbers.

1. Basic Derivative: f(x) = x²

Rule: f'(x) = 2x

f(3) = 3² = 9
f'(3) = 2×3 = 6
Meaning: at x=3, if x increases by 0.01, f(x) increases by ~0.06

Verify numerically:
  (f(3.0001) - f(3)) / 0.0001 = 6.0001 ≈ 6 ✓

2. Key Derivative Rules

FunctionDerivativeExamplef(x) = cf'(x) = 0f(x) = 5 → f'(x) = 0f(x) = x^nf'(x) = n·x^(n-1)f(x) = x³ → f'(x) = 3x²f(x) = a·xf'(x) = af(x) = 3x → f'(x) = 3f(x) = e^xf'(x) = e^xstays the same!f(x) = ln(x)f'(x) = 1/xsigmoid σ(x)σ'(x) = σ(x)·(1-σ(x))key for neural networks!

3. Squared Error — THE Key Function in ML

L = (prediction - actual)²
L'(prediction) = 2×(prediction - actual)

Worked example:
  prediction = 7, actual = 10
  Loss = (7 - 10)² = (-3)² = 9
  Gradient = 2×(7 - 10) = 2×(-3) = -6
  
  Negative gradient → prediction should INCREASE to reduce loss ✓

4. Partial Derivatives — Multiple Variables

f(x, y) = x²·y + 3y²

f(2, 3) = 2²×3 + 3×3² = 12 + 27 = 39

∂f/∂x = 2xy = 2×2×3 = 12  (treat y as constant)
∂f/∂y = x² + 6y = 4 + 18 = 22  (treat x as constant)

Meaning:
  If x increases by 0.01 (y fixed): f increases by ~0.12
  If y increases by 0.01 (x fixed): f increases by ~0.22

5. ML Model Partial Derivatives

Model: ŷ = w×x + b
Loss:  L = (ŷ - y)²

w = 2.0, b = 1.0, x = 3.0, y_true = 10.0

Forward: ŷ = 2.0×3.0 + 1.0 = 7.0
Loss:    L = (7.0 - 10.0)² = 9.0

∂L/∂w = 2×(ŷ-y)×x = 2×(-3)×3 = -18
∂L/∂b = 2×(ŷ-y)×1 = 2×(-3) = -6

Both negative → w and b should INCREASE
(makes sense: ŷ=7 is too low, need bigger w and b)

6. Chain Rule — Nested Functions

y = (3x + 2)²

Outer: u² → derivative = 2u
Inner: 3x+2 → derivative = 3
Chain: dy/dx = 2(3x+2) × 3 = 6(3x+2)

At x=1:
  y = (3×1+2)² = 25
  dy/dx = 6×(3×1+2) = 6×5 = 30
  Numerical check: 30.00 ✓

7. Sigmoid Derivative (Key for Neural Networks)

σ(x) = 1/(1+e^(-x))
σ'(x) = σ(x) × (1 - σ(x))

x=-2: σ(-2) = 0.1192, σ'(-2) = 0.1050
x= 0: σ(0)  = 0.5000, σ'(0)  = 0.2500  ← maximum!
x= 1: σ(1)  = 0.7311, σ'(1)  = 0.1966
x= 3: σ(3)  = 0.9526, σ'(3)  = 0.0452  ← vanishing!

σ'(0) = 0.25 (max), σ'(±∞) → 0 (vanishing gradient)
This is why deep networks use ReLU instead of sigmoid

8. Full Gradient Descent — Step by Step

Task: Learn y = 3x + 1
Data: (1,4), (2,7), (3,10), (4,13)
Model: ŷ = w×x + b (find w=3, b=1)

Initial: w = 0.5, b = 0.0, learning_rate = 0.01

Epoch  w        b        Loss       ∂L/∂w      ∂L/∂b
------------------------------------------------------
0      0.5000   0.0000   60.3750    -42.5000   -14.5000
1      0.9250   0.1450   41.8938    -35.4000   -12.0850
2      1.2790   0.2659   29.0701    -29.4857   -10.0733
3      1.5739   0.3666   20.1720    -24.5592    -8.3975
4      1.8194   0.4506   13.9978    -20.4555    -7.0016
5      2.0240   0.5206    9.7137    -17.0371    -5.8388
...
Final: w = 3.0246 (target: 3.0) ✓
       b = 0.9277 (target: 1.0) ✓

Prediction for x=5: ŷ = 3.0246×5 + 0.9277 = 16.05 (actual: 16) ✓

Python Code (Part 1)

import math

# === Derivative verification ===
def f(x): return x ** 2
def f_derivative(x): return 2 * x

x = 3
h = 0.0001
numerical = (f(x + h) - f(x)) / h  # 6.0001
analytical = f_derivative(x)         # 6.0

# === Gradient Descent: Learn y = 3x + 1 ===
data = [(1, 4), (2, 7), (3, 10), (4, 13)]
w, b = 0.5, 0.0
lr = 0.01

for epoch in range(200):
    dw, db = 0, 0
    for x, y in data:
        y_pred = w * x + b
        error = y_pred - y
        dw += 2 * error * x / len(data)
        db += 2 * error / len(data)
    w -= lr * dw
    b -= lr * db

print(f"Learned: w={w:.4f}, b={b:.4f}")  # w≈3.0, b≈1.0

The Key Insight

Gradient Descent in 4 lines of math:

  1. Predict:  ŷ = w·x + b
  2. Loss:     L = (ŷ - y)²
  3. Gradient: ∂L/∂w = 2(ŷ-y)·x,  ∂L/∂b = 2(ŷ-y)
  4. Update:   w = w - lr × ∂L/∂w,  b = b - lr × ∂L/∂b

  Repeat until loss is small. That's ALL gradient descent is.