Choosing the learning rate

What is the impact of a learning rate that is too small in gradient descent?

• Faster convergence
• Slower convergence
• No convergence
• Increased likelihood of overshooting the minimum

Answer explanation

A learning rate that is too small may lead to slower convergence, as the algorithm takes small steps towards the minimum.

How does a learning rate that is too large affect gradient descent?

• Slower convergence
• Increased likelihood of overshooting the minimum
• No impact on convergence
• Smoother cost function curve

Answer explanation

A learning rate that is too large may result in overshooting the minimum, causing the algorithm to oscillate or even diverge.

What does it mean if the cost function consistently increases after each iteration during gradient descent?

• Learning rate is optimal
• Learning rate is too small
• Learning rate is too large
• Code is broken or has a bug

Answer explanation

Consistently increasing cost after each iteration may indicate a learning rate that is too large or a possible bug in the code.

How can you debug the implementation of gradient descent if the cost function doesn't decrease on every iteration?

• Increase the learning rate
• Set the learning rate to a very small number
• Use a constant learning rate
• Ignore the issue, it's normal behavior

Answer explanation

One debugging approach is to set the learning rate to a very small number and observe if the cost decreases on every iteration.

What trade-off should be considered when choosing the learning rate in gradient descent?

• Trade-off between cost and accuracy
• Trade-off between time and space complexity
• Trade-off between learning rate and convergence speed
• Trade-off between bias and variance

Answer explanation

The trade-off to consider is between the learning rate and convergence speed. A small learning rate may lead to slow convergence, while a large learning rate may result in overshooting and divergence.

How do you choose a good learning rate for gradient descent in practice?

• Always use a very small learning rate
• Randomly pick a learning rate within a given range
• Try a range of values and pick the one that decreases the cost rapidly and consistently
• Stick to a fixed learning rate for all models

Answer explanation

It is recommended to try a range of values for the learning rate and observe the cost function's behavior.

What is the purpose of trying a range of learning rates, including values that are three times larger than the previous one?

• To confuse the model
• To speed up convergence
• To find the smallest learning rate
• To explore a range of possible values and find an optimal learning rate

Answer explanation

Trying a range of learning rates helps identify the optimal rate that decreases the cost function rapidly and consistently.

Why is it important to run gradient descent with different learning rates for only a handful of iterations during the selection process?

• To save computational resources
• To prevent overfitting
• To quickly identify the best learning rate
• To avoid convergence issues

Answer explanation

Running gradient descent with different learning rates for only a few iterations helps quickly identify if a learning rate is suitable or if there are convergence issues.

In the context of gradient descent, why is it crucial to use the minus sign in the update formula?

• It doesn't matter, either plus or minus sign can be used
• To decrease the cost function
• It has no impact on the algorithm
• To move the cost function further from the global minimum

Answer explanation

Using the minus sign in the update formula is crucial for moving the parameter toward the global minimum, ensuring the correct direction for convergence.

How does feature scaling relate to choosing an appropriate learning rate in gradient descent?

• They are unrelated concepts
• Feature scaling has no impact on learning rate
• Feature scaling can affect convergence speed and stability, influencing the choice of the learning rate
• Feature scaling is an alternative to learning rate adjustment

Answer explanation

Feature scaling can affect convergence speed and stability, influencing the choice of the learning rate. It is important to scale features to a similar range to avoid issues where one feature dominates the learning process.