Transformer Models and Tokenization Concepts

Front

What is a Transformer model?

Back

A Transformer model is a type of neural network architecture that uses self-attention mechanisms to process input data in parallel, enabling efficient handling of sequential data such as text.

Front

What is tokenization in NLP?

Back

Tokenization is the process of converting a sequence of text into smaller units called tokens, which can be words, subwords, or characters, to facilitate processing by machine learning models.

Front

What is the main advantage of using Transformer models over RNNs?

Back

The main advantage is that Transformer models allow for parallelization of computations, eliminating the sequential bottlenecks present in RNNs, leading to faster training and inference.

Front

What does 'self-attention' mean in the context of Transformers?

Back

Self-attention is a mechanism that allows the model to weigh the importance of different words in a sentence relative to each other, enabling it to capture contextual relationships.

Front

What is greedy search in decoding?

Back

Greedy search is a decoding strategy that selects the token with the highest probability at each step, leading to a sequence that may lack diversity.

Front

What is top-K sampling?

Back

Top-K sampling is a decoding method where the model samples from the top K most probable tokens, allowing for more diverse outputs compared to greedy search.

Front

What is the role of the encoder in a Transformer model?

Back

The encoder processes the input data and generates a set of continuous representations that capture the contextual information of the input.

Front

What is the role of the decoder in a Transformer model?

Back

The decoder takes the encoder's output and generates the final output sequence, often used in tasks like translation or text generation.

Front

What is BERT and how is it different from GPT-2?

Back

BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only model designed for understanding context in text, while GPT-2 (Generative Pre-trained Transformer 2) is a decoder-only model focused on text generation.

Front

What does the pass@k metric measure?

Back

The pass@k metric measures the probability that the correct output is included within the top-k generated outputs by the model.