Airflow Concepts: Difference between revisions
Jump to navigation
Jump to search
(→Task) |
|||
| Line 41: | Line 41: | ||
=Worker= | =Worker= | ||
=Metadata Database= | =Metadata Database= | ||
=Connections & Hooks= | |||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html}} | |||
=Pool= | |||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/pools.html}} | |||
Revision as of 00:58, 11 July 2022
External
Internal
Workflow
DAG
SubDAG
Task
Tasks have dependencies on each other.
Task Types
Operator
Sensor
A Sensor is a subclass of Operator.
TaskFlow-decorated Task
Passing Data between Tasks
Tasks pass data among each other using:
- XComs, when the amount of metadata to be exchanged is small.
- Uploading and downloading large files from a storage service.
TaskGroup
XComs
"Cross-communications".