ML B2 CH8

What is the main limitation of the traditional seq2seq model?

• It cannot translate short sentences
• It uses too many output layers
• It encodes input into a fixed-length vector regardless of sentence length
• It requires bidirectional input only

Answer explanation

Traditional seq2seq models compress all input into a fixed-length vector, which limits performance on longer inputs.

What does the attention mechanism allow the decoder to use?

• Only the last hidden state
• Only encoder output embeddings
• All encoder hidden states with different importance
• Static word vectors

Answer explanation

Attention allows the decoder to focus on all encoder hidden states, assigning different weights to each.

How is the context vector in attention calculated?

• Average of all vectors
• Maximum value among vectors
• Weighted sum of encoder outputs
• Concatenation of last and first vectors

Answer explanation

The context vector is a weighted sum of all encoder output vectors based on attention weights.

Which mathematical operation is used to compute attention scores?

• Matrix inverse
• Subtraction
• Dot product
• Hadamard product

Answer explanation

Attention scores are calculated using the dot product between decoder state and encoder outputs.

What is the role of the softmax function in attention?

• To generate output probabilities
• To normalize attention scores into weights
• To embed characters
• To reduce vocabulary size

Answer explanation

Softmax normalizes raw attention scores into a probability distribution over encoder outputs.

What does a Time Attention layer do?

• Calculates self-attention only
• Aggregates attention across all time steps
• Forgets context information
• Adds noise to encoder outputs

Answer explanation

Time Attention layers apply attention mechanisms at each time step of the sequence.

What is bidirectional RNN used for in attention-based models?

• To skip decoding steps
• To allow input data to flow only backward
• To consider context from both directions of the input
• To reduce computation time

Answer explanation

Bidirectional RNNs allow the model to gather information from both past and future inputs.

What is a benefit of using skip connections in deep seq2seq models?

• Reduces sequence length
• Simplifies vocabulary mapping
• Prevents vanishing gradients
• Eliminates the need for attention

Answer explanation

Skip connections help gradients flow directly, which mitigates vanishing gradient problems in deep networks.

What is the difference between attention and self-attention?

• Self-attention uses only decoder data
• Attention maps between encoder and decoder; self-attention maps within the same sequence
• They are identical concepts
• Self-attention only uses the first word

Answer explanation

Self-attention relates parts of the same sequence to each other, while attention connects different sequences.

What is the core idea behind Neural Turing Machines (NTMs)?

• Replacing LSTM with Transformers
• Using external memory with attention mechanisms
• Training without backpropagation
• Writing gradients to disk

Answer explanation

NTMs enhance neural networks by adding external memory, using attention to read/write from it.