Pyspark day 1

What is Pyspark?

• A new species of snake

• A type of firework

• Python API for Apache Spark

• A type of computer virus

Answer explanation

Pyspark is a Python API for Apache Spark, a powerful distributed computing system for big data processing.

What are the advantages of using Pyspark?

• Pyspark has no advantages compared to other big data tools

• Pyspark has limited APIs in Python

• Pyspark offers easy integration with other big data tools, high-level APIs in Python, and a powerful processing engine.

• Pyspark has a slow processing engine

Answer explanation

Pyspark offers easy integration, high-level APIs, and a powerful processing engine, making it advantageous compared to other big data tools.

Explain the concept of Resilient Distributed Datasets (RDDs) in Pyspark.

• RDDs cannot be rebuilt if a partition is lost

• RDDs are only stored in a single node in a cluster

• RDDs are a fundamental data structure in Pyspark that represents a collection of items distributed across multiple nodes in a cluster, and they are resilient in the sense that they can be rebuilt if a partition is lost.

• RDDs are a type of database in Pyspark

Answer explanation

RDDs are a fundamental data structure in Pyspark that represents a collection of items distributed across multiple nodes in a cluster, and they are resilient in the sense that they can be rebuilt if a partition is lost.

How can you create an RDD in Pyspark?

• sc.makeRDD(data)

• spark.createRDD(data)

• sc.parallelize(data)

Answer explanation

To create an RDD in Pyspark, use the 'sc.parallelize(data)' method. It is the correct choice for creating RDDs.

What are the different transformations in Pyspark?

• transform

• There are various transformations in Pyspark such as map, filter, reduce, flatMap, groupByKey, reduceByKey, sortByKey, join, and many more.

• aggregate

• sort

Answer explanation

The correct choice is 'There are various transformations in Pyspark such as map, filter, reduce, flatMap, groupByKey, reduceByKey, sortByKey, join, and many more.'

Explain the map transformation in Pyspark.

• Map transformation only works on numeric data in Pyspark.

• Map transformation applies a function to the entire RDD at once.

• Map transformation applies a function to each element in the RDD and returns a new RDD.

• Map transformation returns the original RDD without any changes.

Answer explanation

Map transformation applies a function to each element in the RDD and returns a new RDD.

What is the difference between map and flatMap transformations in Pyspark?

• The map transformation applies a function that returns an iterator and then flattens the result.

• The flatMap transformation applies a function to each element of the RDD independently.

• Map and flatMap transformations are the same and can be used interchangeably.

• The map transformation applies a function to each element of the RDD independently, while the flatMap transformation applies a function that returns an iterator and then flattens the result.

Answer explanation

The map transformation applies a function to each element of the RDD independently, while the flatMap transformation applies a function that returns an iterator and then flattens the result.

What are actions in Pyspark?

• Statements that control the flow of the program

• Variables used to store data in Pyspark

• Functions that define the structure of the data

• Operations that trigger the execution of the Spark job

Answer explanation

Actions in Pyspark are operations that trigger the execution of the Spark job.

What is the purpose of the collect action in Pyspark?

• To retrieve all the elements of a distributed dataset and bring them back to the driver program.

• To split the elements of a distributed dataset into multiple partitions

• To delete elements from a distributed dataset

• To shuffle the elements of a distributed dataset

Answer explanation

The collect action is used to retrieve all elements of a distributed dataset and bring them back to the driver program. It is used to obtain the final result of a Spark job.

Explain the take action in Pyspark.

• Return a new RDD containing only the elements that satisfy a predicate function.

• Return a new RDD containing all elements

• Return a new DataFrame

• Return a new RDD containing elements that do not satisfy the predicate function

Answer explanation

The take action in Pyspark returns a new RDD containing only the elements that satisfy a predicate function. It filters the RDD based on the given condition.