ML Chapter 06

What does SGD stand for in neural network training?

• Soft Gradient Descent
• Stochastic Gradient Descent
• Strong Graph Derivative
• Semi-Gain Depth

What is the main idea behind Stochastic Gradient Descent (SGD)?

• Using the full dataset for every update
• Adding randomness to initialization
• Updating weights using small random batches
• Freezing weights during training

What problem does the Momentum method solve in SGD?

• Overfitting
• Vanishing gradient
• Oscillations in gradient updates
• Data imbalance

How does AdaGrad adjust the learning rate?

• Keeps it constant
• Increases it exponentially
• Adapts it for each parameter based on past gradients
• Resets it every epoch

What is the main feature of the Adam optimizer?

• Ignores momentum
• Uses only recent gradients
• Combines Momentum and AdaGrad
• Requires no tuning

Why is initializing weights with a standard deviation of 0.01 sometimes problematic?

• It slows down learning
• It may cause vanishing gradients
• It improves generalization
• It speeds up convergence too much

What is the Xavier initialization designed for?

• ReLU activations
• Linear regression
• Layers with sigmoid/tanh activations
• Binary classification

What is the He initialization suitable for?

• Layers using ReLU
• Normalization layers
• Softmax outputs
• Convolutional outputs

What is the main purpose of Batch Normalization?

• Compress layers
• Add noise to inputs
• Stabilize and accelerate training
• Randomize inputs

What does weight decay help prevent?

• Underfitting
• Oscillation
• Overfitting
• Vanishing gradient