Airflow Concepts: Difference between revisions
Jump to navigation
Jump to search
(→Task) |
(→Task) |
||
| Line 18: | Line 18: | ||
==TaskFlow-decorated Task== | ==TaskFlow-decorated Task== | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | ||
==Passing Data between Tasks== | |||
Tasks pass data among each other using: | |||
* [[#XComs|XComs]], when the amount of metadata to be exchanged is small. | |||
* Uploading and downloading large files from a storage service. | |||
=XComs= | |||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}} | |||
"Cross-communications". | |||
=Workload= | =Workload= | ||
Revision as of 00:55, 11 July 2022
External
Internal
Workflow
DAG
Task
Tasks have dependencies on each other.
Operator
Sensor
A Sensor is a subclass of Operator.
TaskFlow-decorated Task
Passing Data between Tasks
Tasks pass data among each other using:
- XComs, when the amount of metadata to be exchanged is small.
- Uploading and downloading large files from a storage service.
XComs
"Cross-communications".