Spark Programming in Python for Beginners with Apache Spark 3 - Spark DataFrameWriter API

What is the primary purpose of the Dataframe Writer API in Spark?

• To visualize data in Spark

• To read data from various sources

• To write data to different internal and external sources

• To manage Spark cluster resources

Which file format does Spark assume by default if none is specified?

• Avro

• Parquet

• JSON

• CSV

What happens in the 'overwrite' save mode in Spark?

• New files are created without affecting existing data

• Existing data files are removed and new files are created

• An error is thrown if data already exists

• Data is written only if the target directory is empty

Which method is used to repartition data based on a key column?

• Repartition

• Partition by

• Bucket by

• Sort by

What is the purpose of the 'max records per file' option?

• To limit the number of partitions

• To control the file size based on the number of records

• To sort data within each file

• To specify the output format

Which method is commonly used with 'bucket by' to create sorted buckets?

• Partition by

• Max records per file

• Repartition

• Sort by

What is a key benefit of key-based partitioning?

• It reduces the number of partitions

• It increases the file size

• It simplifies the data writing process

• It improves Spark SQL performance using partition pruning