T3

With Streaming

• We can use delta

• we can use event hub

• we can use kafka

• all the above

dbutils.fs.rm

• allow mount rm (resource manager)

• must run every time a folder is created

• allow remove a file/directory

• allow the use of parquet

wich one is not a set of dbutils commands

• fs

• run

• play

• help

HDInsight

• is spark as a service on azure

• to use this technology an onPrem VM must be present

• is part of Databricks

• is hadoop as a service on azure

dbutils.notebook.run("LOCATION_OF_CALLEE_NOTEBOOK", 60)

• this notebook is executed 60x

• only return 60 records

• have a 60h timeout

• none of the above

By default a Databricks cluster teminate after the last execution

• 120h

• 120m

• 120s

• 1d

Cluster workers

• are presisted

• must be grerater than 2

• can have different VM types

• can have a fixed or variable number

Databricks have a small storage area

• to save our data

• to enable autoscale

• to build datalakes

• to enable things like mount

Notebooks are

• computation power

• a place to write spark code

• a place to write pyspark, r, java

• all the above

to write to a parquet file the 'df' dataframe, one possible command is

• spark.write.parquet('path')

• spark.write.parquet(df)

• df.write.parquet('path')

• df.parquet('path').display*(