Spark

External

• https://spark.apache.org
• https://spark.apache.org/docs/latest/index.html
• https://www.macrometa.com/event-stream-processing/spark-vs-flink

Internal

• Distributed Systems
• Stream Processing
• Flink
• Beam
• Iceberg
• Alluxio
• Spark Operator
• Genie
• Livy
• dbt

Overview

Spark is a third generation unified analytics engine for large-scale data processing. It natively supports batch processing and stream processing. Stream processing is implemented as micro-batching. It uses HDFS as state backend.

Subjects

• Concepts

Organizatorium

• Spark SQL
• PySpark/Spark SQL in interactive mode on JupyterHub.
• Spark batch and streaming.
• Spark job.
• Spark UI
• Spark history server
• Spark remote shuffle service
• Spark Operator