Distributed Systems

Primary motivation for parallel execution in ML training

• To reduce model complexity

• To speed up the model training process for large datasets

• To minimize GPU memory usage

• To simplify hyperparameter tuning

Training ResNet-50 on ImageNet-1K, single GPU

• Minutes to hours

• Hours to one day

• Several days to two weeks

• Over a month

Most popular form of in-parallel training

• Model Parallelism

• Pipeline Parallelism

• Data Parallelism

• Hybrid Parallelism

Key characteristic ImageNet vs CIFAR-10

• Larger dataset size

• More classes

• Higher image resolution

• Greater label complexity

Primary goal of parallel execution

• To enable larger models

• To reduce data storage costs

• To speed up the training process

• To improve model accuracy

Fundamental cause of long training times on single node

• Limited CPU cores

• GPU clock speed

• Mismatch between data loading bandwidth and training bandwidth

• Small batch sizes

Effect of higher resolution (ImageNet-1K)

• Faster convergence

• Larger activations → more memory → fewer images per batch

• Reduced overfitting

• Higher gradient precision

Why single-node training is "too slow"

• CPU overheating

• Bandwidth mismatch

• Insufficient RAM

• Software inefficiencies

Direct consequence of high-resolution activations

• Increased model accuracy

• Fewer images can be trained at once

• Faster backward passes

• Reduced need for synchronization

Effect of smaller activations (CIFAR-10)

• Lower model capacity

• Allows more input images at once → higher training bandwidth

• Increased regularization

• Slower convergence