Spark Programming in Python for Beginners with Apache Spark 3 - Spark Transformations and Actions

The video tutorial explains the basics of Spark data processing, focusing on dataframes, their immutability, and how transformations and actions work. It covers the concept of narrow and wide dependency transformations, the importance of lazy evaluation, and how actions trigger execution. The tutorial also highlights the creation of a DAG of operations and the role of the Spark driver in optimizing execution plans.

What is a key characteristic of Spark Dataframes?

• They require manual partitioning.

• They are immutable.

• They are stored in a single location.

• They are mutable.

In Spark, what is the purpose of using intermediate variables?

• To store data permanently.

• To increase execution time.

• To break down transformations into smaller steps.

• To avoid using Spark sessions.

What is a Directed Acyclic Graph (DAG) in Spark?

• A single node operation.

• A linear sequence of tasks.

• A cycle of operations.

• A graph of operations representing transformations and actions.

Which type of transformation can be performed independently on each partition?

• Shuffle transformation.

• Narrow dependency transformation.

• Sort transformation.

• Wide dependency transformation.

What is a characteristic of wide dependency transformations?

• They do not require data from other partitions.

• They can be executed independently on each partition.

• They are always faster than narrow dependencies.

• They require data from other partitions to produce correct results.

What operation is necessary to fix issues caused by wide dependency transformations?

• Repartitioning.

• Sorting.

• Filtering.

• Joining.

What is lazy evaluation in Spark?

• Execution of transformations in parallel.

• Execution of actions before transformations.

• Execution of transformations only when an action is called.

• Immediate execution of transformations.

Which of the following is considered an action in Spark?

• Group By

• Show

• Select

• Filter

How does Spark optimize the execution of transformations?

• By executing them in reverse order.

• By rearranging them to create an optimized execution plan.

• By ignoring intermediate steps.

• By executing them immediately.

What triggers the execution of a transformation DAG in Spark?

• A transformation.

• A partition.

• An action.

• A session.