Airflow Concepts: Difference between revisions
Jump to navigation
Jump to search
(→DAG) |
(→Task) |
||
| Line 14: | Line 14: | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html}} | ||
Tasks have dependencies on each other. | Tasks have dependencies on each other. | ||
==Operator== | ==Task Types== | ||
===Operator=== | |||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html}} | ||
===Sensor=== | ====Sensor==== | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html}} | ||
A Sensor is a subclass of [[#Operator|Operator]]. | A Sensor is a subclass of [[#Operator|Operator]]. | ||
==TaskFlow-decorated Task== | ===TaskFlow-decorated Task=== | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | ||
==Passing Data between Tasks== | ==Passing Data between Tasks== | ||
| Line 25: | Line 26: | ||
* [[#XComs|XComs]], when the amount of metadata to be exchanged is small. | * [[#XComs|XComs]], when the amount of metadata to be exchanged is small. | ||
* Uploading and downloading large files from a storage service. | * Uploading and downloading large files from a storage service. | ||
==TaskGroup== | |||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-taskgroups}} | |||
=XComs= | =XComs= | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | ||
Revision as of 00:57, 11 July 2022
External
Internal
Workflow
DAG
SubDAG
Task
Tasks have dependencies on each other.
Task Types
Operator
Sensor
A Sensor is a subclass of Operator.
TaskFlow-decorated Task
Passing Data between Tasks
Tasks pass data among each other using:
- XComs, when the amount of metadata to be exchanged is small.
- Uploading and downloading large files from a storage service.
TaskGroup
XComs
"Cross-communications".