Multi-GPU training uses multiple GPUs in one machine. Fast interconnect (NVLink) enables rapid communication.
Multi-node training spans multiple machines. Network becomes the bottleneck. Requires careful optimization.
Start with multi-GPU on a single node. Move to multi-node only when you've exhausted single-node options. The complexity increases substantially.