Generative AI and LLMs Quiz

What is the approximate GPU RAM needed to store 1B parameters?

• 4GB

• 24GB

• 32GB

• 16GB

What does DDP stand for in the context of model training?

• Distributed Data Parallel

• Data Distribution Protocol

• Dynamic Data Processing

• Distributed Data Processing

What is the main advantage of Fully Sharded Data Parallel (FSDP)?

• Reduces overall GPU memory utilization

• Increases model accuracy

• Speeds up training time

• Simplifies model architecture

What is the purpose of the All-Reduce Operation in DDP?

• To synchronize gradients across all GPUs

• To update model parameters

• To split input data into mini-batches

• To perform forward and backward passes

What is the communication protocol commonly used in multi-GPU systems?

• NCCL

• MPI

• HTTP

• TCP/IP

What is the main difference between DDP and FSDP?

• Memory usage and communication overhead

• Training speed

• Model accuracy

• Ease of implementation

What does FSDP do to minimize memory usage?

• Shards both model parameters and optimizer states

• Uses a single GPU for training

• Increases model size

• Reduces the number of parameters

What is the purpose of checkpointing in FSDP?

• To store model shards for reconstruction

• To increase training speed

• To reduce memory usage

• To synchronize gradients

What is the main computational challenge as model sizes get larger?

• Need to split the model across multiple GPUs

• Increased training time

• Higher accuracy requirements

• More complex model architectures

What is the role of data sharding in DDP?

• To split input data into smaller subsets

• To increase model size

• To reduce training time

• To synchronize gradients