Hello World

Be Happy!

TP vs PP — Multi-Node Mac M5 LLM Strategy


TP vs PP for Multi-Node LLM Inference

When running large language models across multiple Apple Silicon Macs, choosing the right parallelism strategy is critical. Here's how Tensor Parallelism (TP) and Pipeline Parallelism (PP) compare.

What is TP (Tensor Parallelism)?

Splits each layer's computation across multiple devices. Every device computes part of every layer, then they synchronize (all-reduce) after each layer.

  • Requires heavy inter-device communication
  • Needs very fast interconnect (NVLink-class: ~900 Gbps)
  • Best within a single machine with shared memory

What is PP (Pipeline Parallelism)?

Assigns different groups of layers to different devices. Data flows sequentially through the pipeline — each device handles its chunk independently.

  • Minimal inter-device communication (only passes activations between stages)
  • Tolerates higher latency and lower bandwidth
  • Best for multiple machines connected over network

Comparison Table

FactorTensor Parallelism (TP)Pipeline Parallelism (PP)How it worksSplits each layer across devicesPuts different layers on different devicesCommunication patternAll-reduce after every layer (heavy)Pass activations between stages (light)Bandwidth requirementVery high (~900 Gbps+ ideal)Low (10–100 Gbps sufficient)Latency sensitivityVery sensitiveTolerantIdle time (bubbles)MinimalSome pipeline bubbles during trainingBest interconnectNVLink, unified memoryThunderbolt, EthernetBest forSingle machine, fast linkMultiple machines over network

Why PP is Better for Multi-Mac Setups

The bottleneck is interconnect speed:

InterconnectBandwidthGood for TP?NVIDIA NVLink~900 GbpsYes ✓Apple Unified Memory (within chip)~800–1228 GB/sYes ✓Thunderbolt 5 (Mac-to-Mac)~120 GbpsNo ✗ (7× too slow)10GbE (Mac-to-Mac)~10 GbpsNo ✗ (90× too slow)

TP requires all-reduce after every layer. With Thunderbolt 5 at 120 Gbps (vs NVLink at 900 Gbps), TP would spend most time waiting for data transfer, destroying performance.

PP only sends activations once between stages — the network bandwidth is sufficient.

Practical Setup: 2× M5 Ultra Running a 140B Model

Mac 1 (M5 Ultra, 256 GB): Layers 0–39  → ~70 GB in memory
Mac 2 (M5 Ultra, 256 GB): Layers 40–79 → ~70 GB in memory
Connection: Thunderbolt 5 (120 Gbps)
Strategy: Pipeline Parallelism

Each Mac independently processes its layer group. Only intermediate activations (~few MB per token) pass between machines.

Recommended Strategy by Use Case

Use CaseStrategyWhyWithin single M5 UltraTPUnified memory = zero transfer costBetween 2+ Macs (inference)PPLow network demand, tolerates latencyBetween 2+ Macs (training)PP + DP (data parallelism)PP for model split, DP for batch splitSingle RTX 6000 AdaNeither neededSingle device, 48 GB fits many modelsMulti-GPU NVIDIA (NVLink)TPNVLink fast enough for all-reduce

Key Takeaways

  1. Use PP between Macs — network is too slow for TP's constant synchronization.
  2. Use TP within a single Mac — unified memory has no transfer overhead.
  3. PP is ideal for inference — latency is acceptable, throughput is good.
  4. PP has bubble overhead for training — combine with Data Parallelism (DP) to mitigate.
  5. Apple's advantage: huge memory (128–512 GB per machine) means fewer nodes needed vs NVIDIA's 48 GB per card.
  6. Thunderbolt 5 (120 Gbps) is sufficient for PP but insufficient for TP. This is the key constraint for multi-Mac setups.
#tensor parallelism (1) #pipeline parallelism (1) #llm (1) #apple silicon (2) #m5 (1) #distributed inference (1) #multi-node (1)
List

Apple GPU vs NVIDIA RTX 6000 Ada — Comparison

x1wins@changwoo.net

Apple M-series GPU v...... #gpu (1) #nvidia (1) #comparison (1) #hardware (1) #m5 pro (1) #m5 max (1) #alu (1) #cuda (1) #apple silicon (2)