Neural Networks Quiz

A simple Multi-Layer Perceptron (MLP) without any non-linear activation functions is mathematically equivalent to:

• A universal function approximator.

• A single, wider linear layer.

• A model incapable of learning.

• A support vector machine with a linear kernel.

The primary motivation for using the ReLU activation function over Sigmoid in deep hidden layers is to:

• Ensure the output is always positive.

• Confine the activation values between 0 and 1.

• Mitigate the vanishing gradient problem.

• Make the network more computationally expensive but more accurate.

The "dying ReLU" problem is characterized by:

• A neuron's weights being updated such that its pre-activation input is consistently negative, causing its output and gradient to be zero.

• The network becoming too deep, causing all ReLU activations to eventually become zero.

• The learning rate being too low, preventing weights from being updated.

• A neuron's weights exploding to infinity.

Why is breaking symmetry by initializing weights randomly (e.g., using Xavier or He initialization) crucial for training?

• It guarantees faster convergence to the global minimum.

• It prevents all neurons in a layer from learning the same features, as they would with zero initialization.

• It acts as a form of L2 regularization.

• It ensures the initial loss of the network is exactly 1.0.

What is the primary purpose of the bias term in a neuron?

• To scale the output of the activation function.

• To act as a learnable offset, allowing the activation function to be shifted left or right.

• To prevent the weights from becoming zero during training.

• To control the learning rate for that specific neuron.

The Universal Approximation Theorem suggests that:

• Any neural network can solve any problem.

• A single-layer perceptron can approximate any linear function.

• A feed-forward network with one hidden layer and a non-linear activation can approximate any continuous function to arbitrary precision.

• Deeper networks are always better than wider networks.

A model is exhibiting high variance and low bias. This is a classic case of:

• Underfitting, where the model is too simple for the data.

• Overfitting, where the model has learned the training data too well, including its noise.

• A well-generalized model.

• A model trained with an incorrect loss function.

If you observe that both your training loss and validation loss are high and have plateaued, your model is likely:

• Overfitting.

• Converging perfectly.

• Underfitting.

• In need of a higher dropout rate.

What is the fundamental mechanism of Backpropagation?

• It uses the chain rule of calculus to efficiently compute the gradient of the loss function with respect to every parameter in the network.

• It randomly perturbs weights to see if the loss decreases.

• It is a specific type of optimizer, similar to Adam.

• It calculates the loss by propagating a signal forward through the network.

In a multi-class classification problem with 10 classes, what is the most appropriate combination for the final layer and loss function?

• 10 neurons with Sigmoid activation and Binary Cross-Entropy loss.

• 1 neuron with a linear activation and Mean Squared Error loss.

• 10 neurons with Softmax activation and Categorical Cross-Entropy loss.

• 10 neurons with ReLU activation and Hinge loss.