sem6_ai_week_8_formula_code

In TF‑IDF code, the weight for term t is often computed as

• tf[t] / (idf[t] + 1e‑9)

• np.exp(tf[t] * idf[t])

• tf[t] ** idf[t]

• tf[t] * idf[t]

Keras Tokenizer converts text to indices through the call

• tokenizer.join_texts(texts)

• tokenizer.pad_on_texts(texts)

• tokenizer.mask_texts(texts)

• tokenizer.texts_to_sequences(texts)

The cosine similarity formula in numpy can be written as

• np.dot(u,v)

• np.sum(u-v)

• np.linalg.norm(u*v)

• np.dot(u,v)/(np.linalg.norm(u)*np.linalg.norm(v))

To pad sequences post‑style you would use

• pad_sequences(seq, padding='pre')

• pad_sequences(seq, padding='mid')

• pad_sequences(seq, truncating='post')

• pad_sequences(seq, padding='post')

In Word2Vec Skip‑gram negative sampling, training loss per pair is

• log(sigmoid(pos))

• sigmoid(neg) - pos

• mean_squared_error(pos,neg)

• ‑log(sigmoid(pos)) ‑ Σ log(sigmoid(‑neg_i))

FastText builds subword keys with the code

• for w in words: make_wordpiece(w)

• for c in chars: merge_pair(c)

• ngrams = None

• for i in range(len(word)): ngrams.add(word[i:j])

One‑hot encoding size in code is derived from

• np.unique(labels).shape[1]

• len(texts)

• embedding_dim

• len(tokenizer.word_index) + 1

The IDF component in scikit‑learn is computed internally as

• df / N

• log(df)

• 1 + tf

• log(N / (df + 1)) + 1

Setting embeddings non‑trainable in Keras uses

• layer.freeze=True

• layer.finetune=False

• model.trainable=False

• Embedding(..., trainable=False)

BPE merges are selected by choosing the pair with

• Lowest alphabetical order rank

• Largest token index value

• Random uniform probability

• Highest frequency count in corpus