Chang-Woo | Math for AI Part 2 — Backpropagation & Code

Neural Network Backpropagation — Worked Example

Part 2: Complete forward pass and backward pass with actual numbers, AND gate training, ReLU, and integration.

1. Single Neuron — Complete Math

Architecture:
  [x1=1.0] ──w1=0.5──┐
                      ├──→ [Σ + bias] ──→ [sigmoid] ──→ ŷ
  [x2=2.0] ──w2=0.3──┘        ↑
                           b=-0.1

Target: y = 1.0

2. Forward Pass

Step 1: z = w1×x1 + w2×x2 + b
       z = 0.5×1.0 + 0.3×2.0 + (-0.1)
       z = 0.5 + 0.6 - 0.1 = 1.0

Step 2: ŷ = sigmoid(1.0) = 1/(1+e^(-1)) = 0.7311

Step 3: Loss = -[y·ln(ŷ) + (1-y)·ln(1-ŷ)]
       Loss = -[1×ln(0.7311) + 0×ln(0.2689)]
       Loss = -ln(0.7311) = 0.3133

3. Backward Pass (Chain Rule!)

Goal: ∂Loss/∂w1 = ∂Loss/∂ŷ × ∂ŷ/∂z × ∂z/∂w1

Step 1: ∂Loss/∂ŷ = -y/ŷ = -1/0.7311 = -1.3679

Step 2: ∂ŷ/∂z = σ(z)×(1-σ(z)) = 0.7311×0.2689 = 0.1966

Step 3: ∂z/∂w1 = x1 = 1.0
        ∂z/∂w2 = x2 = 2.0
        ∂z/∂b  = 1

FULL CHAIN:
  ∂Loss/∂w1 = -1.3679 × 0.1966 × 1.0 = -0.2689
  ∂Loss/∂w2 = -1.3679 × 0.1966 × 2.0 = -0.5379
  ∂Loss/∂b  = -1.3679 × 0.1966 × 1   = -0.2689

4. Weight Update

learning_rate = 0.1

w1: 0.5000 - 0.1×(-0.2689) = 0.5269  (increased ✓)
w2: 0.3000 - 0.1×(-0.5379) = 0.3538  (increased ✓)
b:  -0.1000 - 0.1×(-0.2689) = -0.0731 (increased ✓)

Loss before: 0.3133
Loss after:  0.2724  ← decreased! ✓

5. Training AND Gate — Watch It Learn

AND gate: only [1,1] → 1, everything else → 0

Epoch   w1       w2       b        Loss     Predictions [0,0] [0,1] [1,0] [1,1]
---------------------------------------------------------------------------------
0       0.2544   0.2261   -0.3268  3.0889   [0.50, 0.46, 0.41, 0.29]
1       0.4160   0.3657   -0.5863  2.7810   [0.42, 0.42, 0.38, 0.30]
3       0.7201   0.6373   -1.0094  2.3591   [0.31, 0.39, 0.36, 0.35]
5       0.9838   0.8787   -1.3668  2.0606   [0.23, 0.37, 0.35, 0.40]
9       1.4081   1.2782   -1.9666  1.6533   [0.14, 0.33, 0.32, 0.49]

...after 1000 epochs...
Final: w1=8.88, w2=8.87, b=-13.47

  [0,0] → 0.0000 (target: 0) ✓
  [0,1] → 0.0099 (target: 0) ✓
  [1,0] → 0.0100 (target: 0) ✓
  [1,1] → 0.9863 (target: 1) ✓

The neuron learned: needs BOTH inputs high AND large negative bias
to overcome. Only [1,1] produces z > 0 → sigmoid > 0.5

6. ReLU — Simpler Derivative

f(x) = max(0, x)
f'(x) = 1 if x > 0, else 0

x:     -3   -1    0    1    3    5
f(x):   0    0    0    1    3    5
f'(x):  0    0    0    1    1    1

Simple! Gradient is 1 (pass through) or 0 (dead)
No vanishing gradient for positive values → trains faster than sigmoid
This is why modern networks use ReLU!

7. Integration in AI

∫₀³ 2x dx = [x²]₀³ = 9 - 0 = 9

Used in AI for:
  - Probability: P(a < X < b) = ∫ₐᵇ p(x) dx
  - Expected value: E[X] = ∫ x·p(x) dx
  - Normalization: ensure ∫ p(x) dx = 1
  - Softmax denominator

Example: Normal distribution
  P(-1 < X < 1) = ∫₋₁¹ N(0,1) dx ≈ 0.6827 (68.3%)
  (the famous 68-95-99.7 rule)

8. Python Code — Single Neuron Training

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# AND gate data
train_data = [([0,0], 0), ([0,1], 0), ([1,0], 0), ([1,1], 1)]
w1, w2, b = 0.1, 0.1, 0.0
lr = 0.5

for epoch in range(1000):
    for inputs, target in train_data:
        x1, x2 = inputs
        # Forward
        z = w1*x1 + w2*x2 + b
        y_pred = sigmoid(z)
        # Backward (simplified)
        dz = y_pred - target
        w1 -= lr * dz * x1
        w2 -= lr * dz * x2
        b  -= lr * dz

# Test
for inputs, target in train_data:
    z = w1*inputs[0] + w2*inputs[1] + b
    print(f"{inputs} → {sigmoid(z):.4f} (target: {target})")

9. Math → AI Connection Summary

┌──────────────────┬───────────────────────────────────────┐
│ Math Concept     │ Where It's Used in AI                 │
├──────────────────┼───────────────────────────────────────┤
│ Derivative       │ How much does loss change per weight? │
│ Partial Deriv.   │ Gradient for each weight separately   │
│ Chain Rule       │ Backpropagation through layers        │
│ Gradient Vector  │ Direction to update all weights       │
│ Sigmoid/ReLU     │ Activation functions + derivatives    │
│ Integration      │ Probability, normalization            │
│ Matrix multiply  │ Forward pass through layers           │
└──────────────────┴───────────────────────────────────────┘

The ENTIRE training process:
  1. Forward: matrix multiply + activation
  2. Loss: measure error
  3. Backward: chain rule (derivatives)
  4. Update: subtract gradient × learning_rate
  5. Repeat millions of times → that's why we need GPUs!

10. Study Priority

PriorityTopicTimeUse in AI🔴 MustDerivatives (basic rules)2-3 hrsComputing gradients🔴 MustPartial derivatives1-2 hrsMulti-variable loss🔴 MustChain rule2-3 hrsBackpropagation🟡 ImportantVectors & dot product2 hrsWeight updates🟡 ImportantMatrix multiplication2-3 hrsNeural network layers🟢 HelpfulIntegration2 hrsProbability/distributions🟢 HelpfulExponentials & logarithms1 hrCross-entropy, softmax

#python (6) #sigmoid (2) #math (5) #integration (2) #ai (14) #backpropagation (1) #neural network (1) #chain rule (2) #relu (1)

List