Airflow Concepts

External

• https://airflow.apache.org/docs/apache-airflow/stable/concepts/index.html

Internal

• Airflow

DAG

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html

Graph Concepts | Directed Acyclic Graph

Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html

Tasks have dependencies on each other.

Operator

https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html

Sensor

https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html

A Sensor is a subclass of Operator.

TaskFlow-decorated Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

Passing Data between Tasks

Tasks pass data among each other using:

• XComs, when the amount of metadata to be exchanged is small.
• Uploading and downloading large files from a storage service.

XComs

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

"Cross-communications".

Scheduler

https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html

Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html