Airflow Concepts

External

• https://airflow.apache.org/docs/apache-airflow/stable/concepts/index.html

Internal

• Airflow

DAG

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html

Graph Concepts | Directed Acyclic Graph

SubDAG

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags

A DAG is made of tasks among which there are relations of dependency. The DAG is not concerned about what happens inside the tasks, it is only concerned about how to run them: order, retries, timeouts. etc.

Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html

Tasks have dependencies on each other.

Task Dependencies

If a task B has a dependency on task A (A → B), it is said that A is upstream of B and B is downstream of A. The dependencies are the directed edges of the directed acyclic graph.

Task Types

Operator

https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html

Sensor

https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html

A Sensor is a subclass of Operator.

TaskFlow-decorated Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

Passing Data between Tasks

Tasks pass data among each other using:

• XComs, when the amount of metadata to be exchanged is small.
• Uploading and downloading large files from a storage service.

TaskGroup

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-taskgroups

XComs

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

"Cross-communications".

Scheduler

https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html

Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html

Connections & Hooks

https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html

Pool

https://airflow.apache.org/docs/apache-airflow/stable/concepts/pools.html