WorksheetsTokenization: Hướng Dẫn Chuyên Sâu
Total questions: 121
Worksheet time: 3hrs 26mins
What is a Token?
Why is Tokenization an essential step?
What are the inherent challenges of Tokenization?
What is Word-level Tokenization?
Simple but limited
Comprehensive but inefficient
Balanced and effective
None of the above
What is Character-level Tokenization?
Simple but limited
Comprehensive but inefficient
Balanced and effective
None of the above
What is Subword Tokenization?
Simple but limited
Comprehensive but inefficient
Balanced and effective
None of the above
What are the main techniques of Subword Tokenization?
What is Byte Pair Encoding (BPE)?
What is the training phase mechanism of BPE?
What is the inference phase process of BPE?
What is WordPiece?
What are the characteristics and differences with BPE?
What is the training phase mechanism of WordPiece?
What is the inference phase process of WordPiece?
What is SentencePiece?
What is the philosophy of 'Language-Agnostic' and 100% recovery capability?
What is the mechanism for handling spaces and underscores in SentencePiece?
What is SentencePiece BPE?
What is SentencePiece Unigram?
What is the comparison between SentencePiece BPE and SentencePiece Unigram?
What is Vocabulary and its role?
What is the size of vocabulary (vocab_size) and its impact?
What are Special Tokens and their common types?
What is OOV (Out-Of-Vocabulary) Handling?
What is the difference between cased and uncased Tokenization?
What is Max Sequence Length and its implications?
What is Reversibility in Tokenization?
What is Tokenization in modern PLM architectures?
What is BERT and its relation to WordPiece?
What is RoBERTa / GPT / Llama and its relation to Byte-level BPE?
What is T5 / XLNet and its relation to SentencePiece?
What is PhoBERT and its relation to RDRsegmenter + SentencePiece Unigram?
What is the process of Tokenization in NLP Pipeline?
What is the process of converting to Token IDs?
What is the process of creating Embeddings?
What is the calculation and learning of Embeddings?
What is the process of inputting into the Transformer Model?
What is the training process?
What is a token in the context of NLP?
A unit of meaning that a text string is divided into
A type of punctuation
A character in a string
A special token used in models
Why is tokenization an essential step?
It helps convert unstructured text into discrete units
It reduces data dimensionality
It manages large vocabularies
All of the above
What are the inherent challenges of tokenization?
Word boundaries
Punctuation and special characters
Capitalization
All of the above
Fill in the blank: Tokens are a bridge to convert text into ______.
How should emojis be processed?
What are the advantages of word-level tokenization?
Easy to implement and understand
Tokens have clear semantic meaning
No OOV issues
Small vocabulary size
What are the disadvantages of character-level tokenization?
Never OOV
Long token sequences
Small vocabulary size
Loss of semantic information
What is Subword Tokenization?
What is Subword Tokenization?
A method that balances between word-level and character-level
A technique that only uses complete words
A process that ignores subwords
A way to tokenize only characters
What is one of the main benefits of Subword Tokenization?
It reduces OOV effectively
It increases vocabulary size
It only uses characters
It ignores semantic information
What are the three main Subword Tokenization techniques?
Byte Pair Encoding, WordPiece, SentencePiece
Character Encoding, WordPiece, SentencePiece
Byte Pair Encoding, Word Encoding, SentencePiece
Byte Pair Encoding, WordPiece, Character Encoding
What is the core idea of Byte Pair Encoding (BPE)?
Start with characters or bytes in the corpus
Only use complete words
Ignore the frequency of tokens
Use only one character at a time
What is the first step in the training phase of BPE?
Pre-tokenization
Token merging
Corpus initialization
Character counting
What is the first step in the BPE process?
Split words into characters
Merge tokens
Update vocabulary
Count token pairs
What happens in the merging step of BPE?
Tokens are combined into a new token
Tokens are split into characters
Vocabulary is updated
Corpus is replaced
What is the final vocabulary of BPE after multiple iterations?
{l, o, w, _, e, r, n, s, t, i, d, es, est, lo, low, er, low_, new, new_, newes, newest_, wid, widest_}
{}
Only single characters
Only merged tokens
What is the main characteristic of the WordPiece method?
It relies on pre-tokenization
It does not require a vocabulary
It is faster than BPE
It uses a different merging strategy
What is the output for the input 'newest' using the greedy segmentation?
['new', 'est_']
['newest']
['newest_']
['new', 'est']
What is the output for the OOV input 'lowerer'?
['low', 'er', 'er_']
['lowerer']
['low', 'er']
['[UNK]']
What is the main characteristic of WordPiece?
It relies on pre-tokenization.
It loses whitespace information.
It uses a likelihood score for merging.
It does not require traditional word segmentation.
What does the prefix ## indicate in WordPiece?
It marks the beginning of a word.
It indicates a subword that is not the start of a word.
It shows a special character.
It is used for punctuation.
What is the first step in the training phase of WordPiece?
Preprocess the text into words.
Merge pairs based on score.
Update the vocabulary.
Segment words into characters.
What is the goal of the pair merging process?
To find pairs that occur together more often than randomly.
To create new tokens.
To update the corpus.
To preprocess the text.
What happens in step B of the pair merging process?
Calculate the frequency of the pair.
Select the pair with the highest score.
Update the vocabulary.
Replace occurrences in the corpus.
What is the process of supervised learning?
What is the output for the input 'unsupervised'?
unsuper
##vised
What is the advantage of using SentencePiece?
What does SentencePiece do with spaces?
What is the special Unicode character for U+0020?
U+2581 LOWER HALF BLOCK
U+005F LOW LINE
U+0020 SPACE
U+2580 FULL BLOCK
What does the underscore represent in the normalization process?
Space
Low line
Token
Subword
What is the first step in the SentencePiece BPE training?
Preprocess words
Normalize
Tokenize
Pair merging
What is the output of the greedy longest match?
_my_variable
_my
_variable
_my_variable_
What is the purpose of the pruning mechanism in Unigram model?
To reduce vocabulary size
To increase likelihood
To enhance training speed
To improve accuracy
What is the process of splitting each sentence in the corpus into the optimal subword strings?
What is the new calculation based on the frequency of subwords in the optimally segmented corpus?
How do you calculate the 'Loss' for each subword?
What is the process of trimming the least useful subwords?
What is the normalization process for input?
What algorithm is used for segmentation inference?
What are the characteristics of SentencePiece BPE and SentencePiece Unigram?
What is the definition of vocabulary in the context of tokenization?
What is the importance of vocabulary size in model performance?
What are special tokens in tokenization?
What is the purpose of the [CLS] token in BERT?
It is used for classification tasks.
It separates sentences.
It pads shorter sequences.
It represents unknown tokens.
What does the [SEP] token do?
It separates sentences or segments.
It is used for padding.
It represents unknown tokens.
It is the first token in a sequence.
What is the function of the [PAD] token?
To ensure all sequences in a batch have the same length.
To represent unknown tokens.
To separate sentences.
To classify input.
What does the [UNK] token represent?
Tokens not found in the vocabulary.
The beginning of a sentence.
The end of a sentence.
Padding for shorter sequences.
What is the purpose of the [MASK] token?
To predict masked words during training.
To separate sentences.
To represent unknown tokens.
To pad shorter sequences.
What does OOV stand for in tokenization?
Out-Of-Vocabulary
Over-Optimized Vocabulary
Open-Ended Vocabulary
Ordered Vocabulary
What is Subword Tokenization?
Breaking new words into known subwords.
A method to pad sequences.
A way to classify tokens.
A technique for sentence separation.
What does Character Fallback ensure?
100% coverage by retaining all single characters.
Separation of sentences.
Padding of sequences.
Classification of tokens.
What is the difference between cased and uncased tokenization?
Cased retains case information, uncased does not.
Uncased retains case information, cased does not.
Both are the same.
Cased is used for padding.
What is the process of converting all text to lowercase before tokenization?
Lowercasing
Casing
Tokenization
Padding
What are the advantages of lowercasing text?
Reduces vocabulary size
Increases computational efficiency
Preserves grammatical information
Improves generalization
What is the disadvantage of lowercasing text?
Loss of important grammatical information
Increased vocabulary size
Better generalization
Improved resource efficiency
What does 'cased' mean in tokenization?
Preserving case information
Lowercasing all text
Removing punctuation
Tokenizing without spaces
What is the maximum sequence length for BERT-base?
What is truncation in the context of tokenization?
Cutting off tokens that exceed max length
Adding padding tokens
Reversing the tokenization process
Preserving original text
What is padding in tokenization?
Adding tokens to meet max length
Removing excess tokens
Reversing tokenization
Preserving case information
What is reversibility in tokenization?
Ability to convert tokens back to original text
Ability to tokenize text
Ability to preserve case
Ability to add padding
What is the tokenizer used in BERT?
WordPiece
Byte-level BPE
SentencePiece
RDRsegmenter
What special tokens are used in BERT?
[CLS], [SEP], [PAD], [UNK], [MASK]
, ,
, ,
What is the characteristic of RoBERTa's tokenizer?
Always cased by default
Requires pre-tokenization
Uses WordPiece
None of the above
What tokenizer does T5 use?
WordPiece
Byte-level BPE
SentencePiece
RDRsegmenter
What is the output of RDRsegmenter in PhoBERT?
Subword tokens
Word segments
Token IDs
None of the above
What is the first step in the NLP pipeline?
Tokenization
Model Input
Text Processing
None of the above
What is the output of the tokenizer when applied to raw text?
List of subword tokens
Token IDs
Raw text
None of the above
What is the output of converting subword tokens to Token IDs?
List of subword tokens
Input tensor
Unique numerical ID
None of the above
What is the process of converting subword tokens to Token IDs?
How are Token, Positional, and Segment Embeddings combined?
What is the purpose of Layer Normalization and Dropout in the embedding process?
How are embeddings computed and learned during pre-training?
When do you need to self-train a Tokenizer?
Unsupported language: When there is no pre-trained model or effective tokenizer for your language.
Domain-specific: Your data has vocabulary, slang, and structure that is very different from existing datasets.
Resource optimization: To create a tokenizer with an optimal vocabulary size for your specific needs.
Research: Experimenting with new tokenizer variants.
What are the detailed steps to self-train a Tokenizer?
What is the importance of data quality in training a Tokenizer?
What should be done to remove noise in the data?
Remove HTML/XML, non-printable characters.
Process social media features.
Decide how to handle emojis.
Ensure each line is a reasonable sentence/paragraph.
What are the two algorithms mentioned for Tokenization?
SentencePiece (Unigram or BPE)
BytePairEncoding from the tokenizers library
What is the purpose of BytePairEncoding from the tokenizers library?
To have more control over the tokenizer
To ensure 100% recovery of SentencePiece
To simplify the training process
To increase the vocabulary size
What is the model_prefix used for in tokenizer configuration?
To specify the name for the tokenizer model
To define the vocabulary size
To set the model type
To control special tokens
What is the purpose of character_coverage in tokenizer training?
To ensure most Unicode characters are covered
To define the model type
To specify the vocabulary size
To add special tokens
What does the special_tokens array contain?
All necessary special tokens for your Transformer model
Only the vocabulary size
The model prefix
The character coverage value
What is the command to train SentencePiece Unigram?
spm.SentencePieceTrainer.train()
spm.Train()
spm.TrainUnigram()
spm.TrainBPE()
What does the tokenizer.bos_token represent?
Beginning of sequence token
End of sequence token
Unknown token
Padding token
