Font size
WorksheetsLarge Scale Machine Learning using PySpark
Total questions: 35
Worksheet time: 3mins
What does inferSchema=True do in spark.read.csv()?
Reads all columns as strings
Automatically detects data types
Removes nulls
Formats columns
If header=False, Spark assigns column names as:
Based on first row
Based on data types
_c0, _c1, ...
Blank
Which function trims whitespaces in PySpark?
clean()
strip()
trim()
format()
df.dropDuplicates() returns:
Sorted rows
Only numeric rows
Unique rows
Rows without nulls
Which expression counts nulls in a column?
sum(col(c).isNull())
_sum(when(col(c).isNull(),1))
countNull(c)
dropNulls()
Which function renames a column in PySpark?
aliasColumn()
renameColumn()
withColumnRenamed()
setName()
What does StringIndexer do?
Standardizes numbers
Converts text → numeric index
Drops missing values
Creates vectors
OneHotEncoder produces:
Dense vectors
Probabilities
Sparse binary vectors
Label indices
Which merges all features into features column?
FeatureMerger
VectorAssembler
FeatureAssembler
VectorCreator
Best scaler for Logistic Regression is:
Normalizer
StandardScaler
VectorBinner
MinMaxEncoder
Indexed target variable is stored as:
target
output
label
y2
With handleInvalid='keep' unseen labels are:
Dropped
Randomly encoded
Assigned to last index
Converted to null
Output of VectorAssembler is:
Scalar
Multiple columns
A single vector column
List of numbers
K-Means optimizes:
Logistic loss
Between-cluster variance
Within-cluster SSE
Accuracy
Centroids represent:
Outliers
Cluster centers
PCA components
Maximum distance points
K-Means stops when:
Accuracy = 100%
Clusters equal size
Centroids stop changing
Loss = 0
Silhouette Score measures:
Model accuracy
Cluster cohesion & separation
Computation time
Outliers
Silhouette score near 1 means:
Poor clustering
Perfect clustering
Overfitted clusters
Random labels
PCA reduces dimensions by:
Dropping rows
Dropping columns
Creating new orthogonal axes
One-hot encoding
Principal Components are:
Raw features
Random variables
Linear combinations of original features
Cluster labels
PCA maximizes:
Runtime
Accuracy
Variance captured
Outliers removed
Logistic Regression uses which function?
tanh
step
sigmoid
softmax
Logistic Regression preferred because:
Higher accuracy
Output between 0 and 1
Handles text
Faster training
If probability = 0.69 and threshold=0.5, prediction is:
0
1
Unknown
Depends
Which evaluator computes ROC-AUC?
RegressionEvaluator
ModelEvaluator
MulticlassEvaluator
BinaryClassificationEvaluator
Accuracy formula:
(a)
Which of the following is the correct formula for accuracy?
TP/(TP+FN)
TP/(TP+FP)
(TP+TN)/Total
FP/Total
Precision measures:
Correct predictions out of actual positives
Correct predictions out of predicted positives
Total negatives
Recall
Recall measures:
Predicted positives correct
Actual positives predicted positive
Total observations predicted positive
Outliers detected
F1 score is the:
Arithmetic mean
Geometric mean
Harmonic mean
Maximum value
ROC curve plots:
Precision vs Recall
TP vs FP
TPR vs FPR
Accuracy vs Recall
What does VectorAssembler do in PySpark?
Combines multiple columns into one feature vector
Standardizes data
Encodes strings
Evaluates model
The command
best_k = sorted(results, key=lambda x: x[1], reverse=True)[0][0]
returns:
Cluster centers
Number of clusters with highest silhouette
Distance threshold
PCA components
In anomaly detection, what does approxQuantile("dist_to_centroid", [0.99], 0.01) compute?
Mean distance
99th percentile distance
Median
Cluster center
Why do we scale features before K-Means?
To avoid negative values
To ensure equal contribution of features
To normalize categorical data
To improve PCA speed
