ML B2 CH6

What is a major limitation of traditional RNNs?

• They can't process images
• They only work with fixed-length sequences
• They suffer from vanishing or exploding gradients
• They use too much memory

Answer explanation

RNNs struggle with long-term dependencies due to vanishing or exploding gradients during backpropagation.

What technique helps to solve the exploding gradient problem?

• Weight decay
• Gradient Clipping
• Dropout
• Softmax normalization

Answer explanation

Gradient clipping limits the maximum gradient value to prevent it from becoming too large during training.

Which component is unique to LSTM compared to traditional RNNs?

• Weight matrix
• ReLU activation
• Memory cell (Ct)
• One-hot encoding

Answer explanation

LSTM introduces a memory cell (Ct) to store long-term information and address gradient issues.

What is the purpose of gates in LSTM?

• To sort word embeddings
• To open and close paths for data flow
• To control dropout rate
• To manage word frequencies

Answer explanation

Gates in LSTM (input, forget, output) regulate how much data is passed or discarded at each time step.

Which function is used in LSTM gates?

• ReLU
• Tanh
• Sigmoid
• Softmax

Answer explanation

The sigmoid function outputs values between 0 and 1, ideal for controlling gate flow in LSTM.

What does the forget gate in LSTM control?

• The activation of the output layer
• Which information to delete from memory
• How to encode input words
• The size of the embedding layer

Answer explanation

The forget gate controls which parts of the memory cell should be erased during updates.

How does TimeLSTM improve over standard LSTM?

• It uses softmax instead of sigmoid
• It processes multiple time steps at once
• It ignores memory cells
• It uses different gates

Answer explanation

TimeLSTM handles full sequences of time steps in parallel for better training efficiency.

What does weight tying achieve in RNNLM?

• Larger models
• More training time
• Parameter reduction and improved performance
• Overfitting prevention

Answer explanation

Weight tying reduces model parameters by sharing weights between the embedding and affine layers.

Why is dropout applied in depth direction in RNNs?

• To avoid loss of information across time steps
• To randomly reset hidden states
• To delete rare words
• To generate sentence vectors

Answer explanation

Dropout in time direction may lose sequential data, so depth-wise dropout preserves temporal consistency.

What is one benefit of stacking multiple LSTM layers?

• Faster training
• Reduces perplexity and improves accuracy
• Smaller model size
• Eliminates need for backpropagation

Answer explanation

Stacking multiple LSTM layers enhances model capacity and typically leads to better language modeling results.