Hello World

Be Happy!

TP vs PP — Multi-Node Mac M5 LLM Strategy


TP vs PP for Multi-Node LLM Inference

When running large language models across multiple Apple Silicon Macs, choosing the right parallelism strategy is critical. Here's how Tensor Parallelism (TP) and Pipeline Parallelism (PP) compare.

How TP Works (Tensor Parallelism)

Splits each layer's computation across multiple devices. Every device computes part of every layer, then they synchronize (all-reduce) after each layer.

┌─────────────────────────────────────────────────────────┐
│              Tensor Parallelism (TP) Flow                │
└─────────────────────────────────────────────────────────┘

              Input Token
                  │
                  ▼
    ┌─────────────────────────────┐
    │         Layer 1             │
    │  ┌───────┐    ┌───────┐    │
    │  │ Mac 1 │◄──►│ Mac 2 │    │  ← All-Reduce sync
    │  │(half) │    │(half) │    │
    │  └───────┘    └───────┘    │
    └─────────────────────────────┘
                  │
                  ▼  (sync needed!)
    ┌─────────────────────────────┐
    │         Layer 2             │
    │  ┌───────┐    ┌───────┐    │
    │  │ Mac 1 │◄──►│ Mac 2 │    │  ← All-Reduce sync
    │  │(half) │    │(half) │    │
    │  └───────┘    └───────┘    │
    └─────────────────────────────┘
                  │
                  ▼  (sync needed!)
    ┌─────────────────────────────┐
    │         Layer N             │
    │  ┌───────┐    ┌───────┐    │
    │  │ Mac 1 │◄──►│ Mac 2 │    │  ← All-Reduce sync
    │  │(half) │    │(half) │    │
    │  └───────┘    └───────┘    │
    └─────────────────────────────┘
                  │
                  ▼
              Output

⚠️  N layers = N sync points = HEAVY network traffic
⚠️  Needs ~900 Gbps+ (NVLink) to be efficient
✗  Thunderbolt 5 (120 Gbps) = bottleneck
  • Requires heavy inter-device communication
  • Needs very fast interconnect (NVLink-class: ~900 Gbps)
  • Best within a single machine with shared memory

How PP Works (Pipeline Parallelism)

Assigns different groups of layers to different devices. Data flows sequentially through the pipeline — each device handles its chunk independently.

┌─────────────────────────────────────────────────────────┐
│            Pipeline Parallelism (PP) Flow                │
└─────────────────────────────────────────────────────────┘

              Input Token
                  │
                  ▼
    ┌─────────────────────────────┐
    │           Mac 1             │
    │  ┌───────────────────────┐  │
    │  │  Layer 1              │  │
    │  │  Layer 2              │  │
    │  │  Layer 3              │  │  No network needed
    │  │  ...                  │  │  (all in unified memory)
    │  │  Layer N/2            │  │
    │  └───────────────────────┘  │
    └─────────────────────────────┘
                  │
                  ▼  (one transfer: activations ~few MB)
         ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
         Thunderbolt 5 / Network
         ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                  │
                  ▼
    ┌─────────────────────────────┐
    │           Mac 2             │
    │  ┌───────────────────────┐  │
    │  │  Layer N/2 + 1        │  │
    │  │  Layer N/2 + 2        │  │
    │  │  ...                  │  │  No network needed
    │  │  Layer N              │  │  (all in unified memory)
    │  └───────────────────────┘  │
    └─────────────────────────────┘
                  │
                  ▼
              Output

✓  Only 1 transfer between machines (not N)
✓  120 Gbps Thunderbolt 5 is sufficient
✓  Each Mac works independently on its layers
  • Minimal inter-device communication (only passes activations between stages)
  • Tolerates higher latency and lower bandwidth
  • Best for multiple machines connected over network

Side-by-Side: Network Traffic Comparison

┌────────────────────────────────────────────────────────────────┐
│           Network Transfers per Token Generation               │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  TP (80 layers):  ████████████████████████████████  80 syncs   │
│                   (each = all-reduce, ~100s MB)                 │
│                                                                │
│  PP (80 layers):  █                                 1 transfer  │
│                   (activations only, ~few MB)                   │
│                                                                │
├────────────────────────────────────────────────────────────────┤
│  TP needs: 900+ Gbps (NVLink)                                  │
│  PP needs: 10-120 Gbps (Ethernet/Thunderbolt) ✓                │
└────────────────────────────────────────────────────────────────┘

Comparison Table

FactorTensor Parallelism (TP)Pipeline Parallelism (PP)How it worksSplits each layer across devicesPuts different layers on different devicesCommunication patternAll-reduce after every layer (heavy)Pass activations between stages (light)Bandwidth requirementVery high (~900 Gbps+ ideal)Low (10–100 Gbps sufficient)Latency sensitivityVery sensitiveTolerantIdle time (bubbles)MinimalSome pipeline bubbles during trainingBest interconnectNVLink, unified memoryThunderbolt, EthernetBest forSingle machine, fast linkMultiple machines over network

Why PP is Better for Multi-Mac Setups

The bottleneck is interconnect speed:

InterconnectBandwidthGood for TP?NVIDIA NVLink~900 GbpsYes ✓Apple Unified Memory (within chip)~800–1228 GB/sYes ✓Thunderbolt 5 (Mac-to-Mac)~120 GbpsNo ✗ (7× too slow)10GbE (Mac-to-Mac)~10 GbpsNo ✗ (90× too slow)

TP requires all-reduce after every layer. With Thunderbolt 5 at 120 Gbps (vs NVLink at 900 Gbps), TP would spend most time waiting for data transfer, destroying performance.

PP only sends activations once between stages — the network bandwidth is sufficient.

Practical Setup: 2× M5 Ultra Running a 140B Model

┌──────────────────────────────────────────────────┐
│          2× M5 Ultra — 140B Model (PP)           │
└──────────────────────────────────────────────────┘

┌────────────────────────┐     ┌────────────────────────┐
│   Mac Studio #1        │     │   Mac Studio #2        │
│   M5 Ultra, 256 GB     │     │   M5 Ultra, 256 GB     │
│                        │     │                        │
│   Layers 0–39          │     │   Layers 40–79         │
│   (~70 GB loaded)      │     │   (~70 GB loaded)      │
│                        │ TB5 │                        │
│   [Input] ──► [Proc] ──┼─────┼──► [Proc] ──► [Output]│
│                        │120Gb│                        │
└────────────────────────┘     └────────────────────────┘

  Transfer: ~few MB activations per token (fast enough!)

Each Mac independently processes its layer group. Only intermediate activations (~few MB per token) pass between machines.

Recommended Strategy by Use Case

Use CaseStrategyWhyWithin single M5 UltraTPUnified memory = zero transfer costBetween 2+ Macs (inference)PPLow network demand, tolerates latencyBetween 2+ Macs (training)PP + DP (data parallelism)PP for model split, DP for batch splitSingle RTX 6000 AdaNeither neededSingle device, 48 GB fits many modelsMulti-GPU NVIDIA (NVLink)TPNVLink fast enough for all-reduce

Key Takeaways

  1. Use PP between Macs — network is too slow for TP's constant synchronization.
  2. Use TP within a single Mac — unified memory has no transfer overhead.
  3. PP is ideal for inference — latency is acceptable, throughput is good.
  4. PP has bubble overhead for training — combine with Data Parallelism (DP) to mitigate.
  5. Apple's advantage: huge memory (128–512 GB per machine) means fewer nodes needed vs NVIDIA's 48 GB per card.
  6. Thunderbolt 5 (120 Gbps) is sufficient for PP but insufficient for TP. This is the key constraint for multi-Mac setups.
#tensor parallelism (1) #pipeline parallelism (1) #llm (1) #apple silicon (2) #m5 (1) #distributed inference (1) #multi-node (1)
List

Apple GPU vs NVIDIA RTX 6000 Ada — Comparison

x1wins@changwoo.net

Apple M-series GPU v...... #gpu (1) #nvidia (1) #comparison (1) #hardware (1) #m5 pro (1) #m5 max (1) #alu (1) #cuda (1) #apple silicon (2)