ACM AI Projects Week 3 Kahoot

I have a piece of text and two tokenizers: tokenizer A tokenizes per character, and tokenizer B tokenizes per subword. Which tokenizer produces a shorter sequence on the same text?

• Tokenizer A

• Tokenizer B

Vector A: [1, 2, 5]

Vector B: [0, 2, 1]

What's the dot product between vector A and vector B?

Reminder: A dot product is the sum of the products of each position: a1b1 + a2b2 + a3b3

• 7

• 4

• 5

• 9

In standard sinusoidal positional embeddings, how do we integrate the positional embeddings with the token (semantic) embeddings?

• Addition

• Dot Product

• Concatenation

• Subtraction

Tensor X has shape [32, 16, 8, 4].

What's the shape of X.permute(1, 2, 0, 3)?

• [32, 4, 8, 16]

• [16, 32, 4, 8]

• [16, 8, 32, 4]

• [4, 8, 16, 32]

In the following code:
X = torch.ones((6, 4, 1))

X = X.squeeze()

X = X.unsqueeze(0)

What's the final shape of X?

Recall that unsqueeze(dim) adds a new dimension of size 1 at shape index dim. tensor.squeeze() removes dimensions of size 1.

• (1, 6, 4)

• (1, 4, 6)

• (6, 4, 1)

• (4, 6, 1)

Bonus Question:

What's the standard shape we pass into a transformer?

• [batch_size, model_dim, seq_len]

• [model_dim, seq_len, batch_size]

• [batch_size, seq_len, model_dim]

What's the result of this operation?

torch.zeros((3)).dot(torch.ones((3)))?

• [1, 1, 1]

• [0, 0, 0]

• 0

• [1, 2, 3]

• 1

In a training loop, where should we call

data.to(device)? Assume your model is on the device already. Write the line it should be immediately before.

• Before output = model(data)

• Before loss.backward().

• Before optimizer.step().

• Before you define your model.

What does the ADAM optimizer stand for?

Remember what it implements:

1. Tracks momentum using a sliding window

2. Tracks the second moment for adaptive gradient scaling.

• Automatic Descent Adjustment Method

• Absolutely Determined to Achieve Minimization

• Adaptive Moment Estimation

• Astonishingly Decent At Minimizing

Bonus Question: I feed a tensor of shape [batch_size, seq_len, model_dim] (after the embedding layer) into a transformer's attention and MLP stack. What is the shape returned by the first attention layer?

• [batch_size, model_dim, model_dim]

• [batch_size, seq_len, model_dim]

• [batch_size, model_dim]