FNLP W3

Who is this person?

• Joel Than

• Lisa Blackpink

• Kelvin Yong

• Colin Tan

Which sentence best describes tokenization?

• Coins used for generating responses

• Segmentation of a document into unique words that will be called tokens

• Splitting words into characters

• Segmentation of a corpus to unique documents

Which word below best fits this description: "A group of documents"

• Vector Space

• Tokenization

• Document

• Corpus

Which of the below does not reduce number of tokens?

• Lemmatization

• Stemming

• Normalization

• Vectorization

True or False? "The Bag of Words model focuses on the occurrence, structure and order of words"

• True

• False

Which of the below describes normalised term frequency?

• Counts of a word in a document divided by the document length

• Equal length of vectors for comparison

• value of each element to represent same word in each vector

• Capitalisation consistent over all words

Cosine similarity has a value between?

• 0 to 1

• 0 to 10

• -1 to +1

• -1 to 0

Zipf's law can be described as?

• frequency of a token in a text is directly proportional to its rank or position in the sorted list

• frequency of a token in a text is inveserly proportional to its rank or position in the sorted list

• Ratio of the total number of documents to the number of documents the term appears in.

Which of the below is not an example of stopwords?

• a

• the

• of

• ate