Airflow Concepts: Difference between revisions
Jump to navigation
Jump to search
(→DAG) |
|||
| Line 10: | Line 10: | ||
==SubDAG== | ==SubDAG== | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags}} | ||
A DAG is made of [[#Task|tasks]] among which there are relations of dependency. The DAG is not concerned about what happens inside the tasks, it is only concerned about how to run them: order, retries, timeouts. etc. | |||
=Task= | =Task= | ||
Revision as of 01:31, 11 July 2022
External
Internal
Workflow
DAG
SubDAG
A DAG is made of tasks among which there are relations of dependency. The DAG is not concerned about what happens inside the tasks, it is only concerned about how to run them: order, retries, timeouts. etc.
Task
Tasks have dependencies on each other.
Task Types
Operator
Sensor
A Sensor is a subclass of Operator.
TaskFlow-decorated Task
Passing Data between Tasks
Tasks pass data among each other using:
- XComs, when the amount of metadata to be exchanged is small.
- Uploading and downloading large files from a storage service.
TaskGroup
XComs
"Cross-communications".