T2

In New Big Data thinking

• all data has value

• all data has same schema

• all data must be saved as parquet

• all data is partitioned

Data Lake Store

• scales from KB to MB

• scales from B to KB

• scales seamlessly from KB to PB

• must be replicated manualy

How many drivers have a regular Databircks cluster

• 4

• 3

• 2

• 1

Cluster workers

• can have a fixed number
• must be grerater than 2
• can have different VM types
• are presisted

Databricks Workspace

• Must always be connected to a cluster
• can not be shared
• can be shared at folder and notebook level
• can be shared all or nothing

Cluster autoscaling

• can be a problem with vertical data
• is configured per data lake
• is automatic
• is configured per notebook

wich is not a databricks notebook supported language

• Spark SQL
• Python
• Scala
• Java

to comment a cell with markdown

• use %fs
• use #
• use %md
• use #md

Databricks have a small storage area

• to save our data
• to enable things like mount
• to build datalakes
• to enable autoscale

to write to a parquet file the 'df' dataframe the correct command is

• spark.write.parquet('path')

• spark.write.parquet(df)

• df.write.parquet('path')

• df.saveAs.parquet('path')