Transformer Quiz

What is the primary advantage of using self-attention in Transformers?

• It reduces the model size

• It eliminates the need for labeled data

• It allows for parallel processing of tokens

• It restricts the context to local information only

In the Transformer model, what does the “multi-head” part of multi-head attention refer to?

• Multiple outputs per token

• Multiple attention layers stacked together

• Multiple parallel attention computations with different projections

• Attention heads used only during inference

What type of complexity does the self-attention mechanism have with respect to sequence length (n)?

• O(n)

• O(log n)

• O(n²)

• O(1)

In a standard Transformer, how many decoder blocks are used in the original architecture for machine translation?

• 4

• 6

• 8

• 12

Which of the following best describes how the Transformer decoder generates output during inference?

• It attends to all positions in the input and output sequences

• It attends to input tokens and already-generated output tokens

• It uses only the encoder’s final output

• It processes the entire sequence at once

How is the final representation of each token computed in multi-head self-attention?

• By summing the outputs of all attention heads

• By averaging token embeddings

• By concatenating attention head outputs and projecting them

• By selecting the maximum attention value

What is the key difference between encoder self-attention and encoder-decoder attention?

• Encoder self-attention is masked, encoder-decoder attention is not

• Encoder-decoder attention uses keys and values from the encoder, queries from the decoder

• Encoder-decoder attention is only used during pre-training

• Encoder-decoder attention uses keys and values from the decoder, queries from the encoder

What special token is added at the beginning of every BERT input sequence?

• <START>

• <EOS>

• [SEP]

• [CLS]

Compared to BERT, GPT's unidirectional training makes it

• Better for classification tasks

• More memory efficient

• More suitable for generative tasks

• Less prone to hallucination

Which of the following is the correct formula for computing the attention scores in the scaled dot-product attention mechanism?

• 
• 
• 
•