PySpark Day2

What is PySpark and how is it different from Apache Spark?

• PySpark is used for data visualization, while Apache Spark is used for data processing

• PySpark is the Python API for Apache Spark, allowing developers to write Spark applications using Python. It is different from Apache Spark as it provides a Python interface to the Spark framework.

• PySpark is a standalone tool not related to Apache Spark

• PySpark is the Java API for Apache Spark

Explain the concept of Resilient Distributed Datasets (RDDs) in PySpark.

• RDDs are a fundamental data structure in PySpark that represents a collection of items distributed across multiple nodes in a cluster, and they are resilient in the sense that they can recover from failures.

• RDDs are a type of database in PySpark

• RDDs are not fault-tolerant in PySpark

• RDDs are only used for single-node processing in PySpark

What are some common transformations that can be applied to RDDs in PySpark?

• read, write, update, delete

• sort, reverse, shuffle, groupBy

• map, filter, flatMap, reduceByKey, sortByKey, join

• add, subtract, multiply, divide

What are some common actions that can be performed on RDDs in PySpark?

• add, subtract, multiply

• insert, update, delete

• collect, count, take, first, and reduce

• search, filter, sort

How can you create a DataFrame in PySpark?

• By using the createDataFrame method in PySpark

• By using the createTable method in PySpark

• By using the readDataFrame method in PySpark

• By converting a list to a DataFrame in PySpark

What are some common operations for manipulating DataFrames in PySpark?

• Sorting and merging data

• Creating and deleting columns

• Selecting, filtering, grouping, joining, and aggregating data

• Looping and iterating through rows

Explain the concept of caching in PySpark DataFrames.

• Caching reduces performance by increasing the need for recomputation.

• Caching improves performance by storing DataFrames in memory to avoid recomputation.

• Caching only works for small DataFrames and has no effect on large ones.

• Caching has no impact on performance in PySpark DataFrames.

How can you perform joins between DataFrames in PySpark?

• Using the 'merge' method

• Using the 'join' method or the 'join' function

• Using the 'concat' function

• Using the 'combine' method

Which of the following language is not supported by Spark?

• Python
• Scala
• Java
• Pascal