Airflow Concepts: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
Line 14: Line 14:
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html}}
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html}}
Tasks have dependencies on each other.
Tasks have dependencies on each other.
==Operator==
==Task Types==
===Operator===
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html}}
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html}}
===Sensor===
====Sensor====
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html}}
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html}}
A Sensor is a subclass of [[#Operator|Operator]].
A Sensor is a subclass of [[#Operator|Operator]].
==TaskFlow-decorated Task==
===TaskFlow-decorated Task===
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}}
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}}
==Passing Data between Tasks==
==Passing Data between Tasks==
Line 25: Line 26:
* [[#XComs|XComs]], when the amount of metadata to be exchanged is small.
* [[#XComs|XComs]], when the amount of metadata to be exchanged is small.
* Uploading and downloading large files from a storage service.
* Uploading and downloading large files from a storage service.
==TaskGroup==
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-taskgroups}}
=XComs=
=XComs=
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}}
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html}}

Revision as of 00:57, 11 July 2022

External

Internal

Workflow

DAG

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html
Graph Concepts | Directed Acyclic Graph

SubDAG

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-subdags

Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html

Tasks have dependencies on each other.

Task Types

Operator

https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html

Sensor

https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html

A Sensor is a subclass of Operator.

TaskFlow-decorated Task

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

Passing Data between Tasks

Tasks pass data among each other using:

  • XComs, when the amount of metadata to be exchanged is small.
  • Uploading and downloading large files from a storage service.

TaskGroup

https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-taskgroups

XComs

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html

"Cross-communications".

Workload

Scheduler

https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html

Executor

https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html

Worker

Metadata Database