Apache Spark 3 for Data Engineering and Analytics with Python - Rows and Union

This tutorial teaches how to create individual row items in PySpark and package them into a DataFrame. It covers creating a list of rows, accessing row items, and using the Union transformation to combine dataframes. The lesson includes practical steps and code examples to guide learners through the process.

What is the primary purpose of using the Union transformation in PySpark?

• To delete duplicate rows from a DataFrame

• To filter rows based on a condition

• To combine two DataFrames into one

• To sort a DataFrame in ascending order

Which library is essential for creating rows in PySpark SQL?

• pyspark.sql

• pandas

• matplotlib

• numpy

What attribute is NOT set when creating a row in the tutorial?

• Date of Birth

• Favorite Movies

• Email

• ID

How can you access the second item in a row using PySpark?

• By using a for loop

• By using the index position 1

• By using the attribute name

• By using the index position 2

What is required to infer a schema when creating a DataFrame from a list of rows?

• A CSV file

• A JSON configuration

• A predefined schema file

• A list of headings

Which function is used to create a DataFrame from a list of rows in PySpark?

• spark.sql

• spark.createDataFrame

• spark.read.csv

• spark.write

What is the final step after creating and combining DataFrames in the tutorial?

• Grouping the DataFrame by last name

• Sorting the DataFrame by ID in descending order

• Filtering the DataFrame by active status

• Saving the DataFrame to a file