Apache Spark 3 for Data Engineering and Analytics with Python - Aggregations - Count and Count Distinct

The video tutorial introduces aggregation functions, focusing on count and count distinct functions. It explains how to import these functions using PySpark and demonstrates their application on dataframes, highlighting how count excludes null values. The tutorial also covers using count distinct to identify unique values in datasets. The session concludes with a preview of upcoming lessons on other aggregation functions like min, max, sum, and average.

Which of the following is NOT an aggregation function mentioned in the introduction?

• Count

• Min

• Median

• Kurtosis

What is the primary purpose of the Count function in data analysis?

• To calculate the average of a dataset

• To find the maximum value in a dataset

• To count the number of non-null records

• To determine the minimum value in a dataset

In PySpark, which module is used to import the Count and Count Distinct functions?

• pyspark.sql.functions

• pyspark.sql.context

• pyspark.sql.types

• pyspark.sql.dataframe

How does the Count function handle null values in a DataFrame?

• It replaces null values with zero before counting

• It excludes null values from the count

• It includes null values in the count

• It throws an error if null values are present

What is the result of using the Count Distinct function on a dataset?

• It counts records and replaces nulls with a default value

• It counts records excluding nulls

• It counts only the unique records

• It counts all records including duplicates

What was the number of unique destination airports found using Count Distinct?

• 322

• 5000

• 4693

• 1000

Which function would you use to find the number of unique entries in a column?

• Count

• Average

• Sum

• Count Distinct