PySpark Training

In Spark, what is the primary function of a Driver?

• To distribute data across the cluster and perform computations on individual partitions.

• To manage the execution of Spark applications, including tasks and coordinating with executors.

• To store and process data in a distributed manner, providing fault tolerance and scalability.

• To define transformations and actions in RDDs, Dataframes, and Datasets.

What is the key difference between Transformations and Actions?

• Transformations create new RDDs, while actions trigger computations and return results.

• Transformations are lazy, while actions are eager.

• Transformations operate on individual elements of an RDD, while actions operate on the entire RDD.

An RDD can be modified after it is created.

• TRUE

• FALSE

What is the primary responsibility of a Worker Node?

• To execute tasks assigned by the driver program and store data in memory or on disk.

• To manage the overall execution of the Spark application, including tasks and coordinating with executors.

• To store the Spark application code and dependencies, making them accessible to executors.

• To monitor the health of the cluster and handle failures of executors or driver programs.

In a Spark application, if a task running on an executor node fails, what is the immediate course of action taken by the driver?

• The driver restarts the entire Spark application

• The driver marks the failed task and waits for the executor to automatically retry the task

• The driver reschedules the failed task on a different available executor.

• The driver terminates the executor that ran the failed task and launches a new executor.