Airflow Concepts: Difference between revisions
(→DAG) |
|||
| Line 8: | Line 8: | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html}} | ||
{{Internal|Graph_Concepts#Directed_Acyclic_Graph_.28DAG.29|Graph Concepts | Directed Acyclic Graph}} | {{Internal|Graph_Concepts#Directed_Acyclic_Graph_.28DAG.29|Graph Concepts | Directed Acyclic Graph}} | ||
The edges can be [https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#edge-labels labeled] in the UI. | |||
==SubDAG== | ==SubDAG== | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags}} | ||
Revision as of 02:03, 11 July 2022
External
Internal
Workflow
DAG
The edges can be labeled in the UI.
SubDAG
A DAG is made of tasks among which there are relations of dependency. The DAG is not concerned about what happens inside the tasks, it is only concerned about how to run them: order, retries, timeouts. etc.
Declaring a DAG
- Via a context manager.
- With the
DAG()constructor. - With the
@dagdecorator. TO PARSE: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#the-dag-decorator.
Control Flow
Dynamic DAG
The DAGs can be purely declarative, or they can be declared in Python code, by adding tasks dynamically.
Task
Every task must be assigned to a DAG to run. Tasks have dependencies on each other.
Task Dependencies
If a task B has a dependency on task A (A → B), it is said that A is upstream of B and B is downstream of A. The dependencies are the directed edges of the directed acyclic graph.
Task Types
Airflow has three types of tasks: Operator, Sensor, which is a subclass of Operator, and TaskFlow-decorated Task.
Operator
Sensor
A Sensor is a subclass of Operator.
TaskFlow-decorated Task
Task Assignment to DAG
Passing Data between Tasks
Tasks pass data among each other using:
- XComs, when the amount of metadata to be exchanged is small.
- Uploading and downloading large files from a storage service.
TaskGroup
This is a pure UI concept.
XComs
"Cross-communications".