Airflow Dynamic Task Mapping: Difference between revisions
Jump to navigation
Jump to search
| Line 8: | Line 8: | ||
* [[Airflow_Concepts#Dynamic_DAG|Airflow Concepts]] | * [[Airflow_Concepts#Dynamic_DAG|Airflow Concepts]] | ||
=Overview= | =Overview= | ||
Many workflows are static, in that all the component task instances and their relationships are statically declared in the workflow definition code: | |||
<syntaxhighlight lang='py'> | |||
# noinspection PyPep8Naming | |||
@task | |||
def A(): | |||
... | |||
@task | |||
def B(): | |||
... | |||
@task | |||
def C(): | |||
... | |||
@dag(...) | |||
def some_dag(): | |||
A() >> B() >> C() | |||
</syntaxhighlight> | |||
However, since the workflow definition code is Python, it allows for a certain degree of flexibility when declaring tasks in a loop. This is equivalent with declaring the same task, individually, multiple times: | |||
<syntaxhighlight lang='py'> | |||
</syntaxhighlight> | |||
A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. <font color=darkkhaki>Tasks cannot create dynamic tasks.</font> | A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. <font color=darkkhaki>Tasks cannot create dynamic tasks.</font> | ||
Revision as of 21:25, 18 July 2022
External
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html
- https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#dynamic-dags
- https://www.astronomer.io/guides/dynamic-tasks/
Internal
Overview
Many workflows are static, in that all the component task instances and their relationships are statically declared in the workflow definition code:
# noinspection PyPep8Naming
@task
def A():
...
@task
def B():
...
@task
def C():
...
@dag(...)
def some_dag():
A() >> B() >> C()
However, since the workflow definition code is Python, it allows for a certain degree of flexibility when declaring tasks in a loop. This is equivalent with declaring the same task, individually, multiple times:
A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. Tasks cannot create dynamic tasks.
Questions
- Can dependencies be declared on dynamic tasks?