Pertemuan 6 - Azure Data Engineer

Which definition best describes Apache Spark?

• A highly scalable relational database management system

• A virtual server with a Python runtime

• A distributed platform for parallel data processing using multiple languages

You need to use Spark to analyze data in a parquet file. What should you do?

• Load the parquet file into a dataframe

• Import the data into a table in a serverless SQL pool

• Convert the data to CSV format

Which method of the Dataframe object is used to save a dataframe as a file?

• toFile()

• write()

• save()

Which method is used to split the data across folders when saving a dataframe?

• splitBy()

• distributeBy()

• partitionBy()

What happens if you drop an external table that is based on existing files?

• An error – you must delete the files first

• The table is dropped from the metastore but the files remain unaffected

• The table is dropped from the metastore and the files are deleted

You want to write code in a notebook cell that uses a SQL query to retrieve data from a view in the Spark catalog. Which magic should you use?

• %%spark

• %%pyspark

• %%sql

You've loaded a Spark dataframe with data, that you now want to use in a Delta Lake table. What format should you use to write the dataframe to storage?

• CSV

• PARQUET

• DELTA