DSBC - Attention Is All You Need

For what RNN is used and achieve the best results?

• Speech and image recognition

• Handwriting and image recognition

• Financial predictions

• Handwriting and speech recognition

What is the basic concept of Recurrent Neural Network?

• Use recurrent features from dataset to find the best answers.

• Use a loop between inputs and outputs in order to achieve the better prediction.

• Use loops between the most important features to predict next output.

• Use previous inputs to find the next output according to the training set.

What architecture represents many-to-many RNNs ?

• 
• 
• 
• 

When RNN sequences are too long, what is most likely to happen ?

• the model is prone to ignoring the hidden states with the highest gradients.

• the model is prone to mixing the content of distant positions it with following positions’ content.

• the model is prone to overfitting easily as exploding gradients hinder efficient back-propagation.

• the model is prone to forgetting the content of distant positions in sequence.

The multi-head attention block is fed three matrices named the Values (V), the Keys (K) and the Query (Q). Which of the following statements is correct ?

• V and K are outputted from the input embedding and Q from the output embedding.

• Q and K are outputted from the input embedding and V from the output embedding.

• V is outputted from the input embedding and K and Q from the output embedding.

• Q is outputted from the input embedding and K and V from the output embedding.

The formula summarizing the multi-head attention operations is:

The self-attention mechanism is permutation invariant.

• True

• False

Select the false statement

• Every position should have the same identifier irrespectively of the sequence length.

• Positional encodings have the same dimension as the embeddings.

• There are many choices of positional encodings.

• Using sinusoidal functions for positional embeddings allow large displacements of positional similarity/dissimilarity.

We can give the self attention greater power of discrimination, by combining several self attention heads.

• True

• False

The attention mechanism contains considerably more weights than a classic RNN.

• True

• False