View source for Airflow Dynamic Task Mapping

=External=
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html
* https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#dynamic-dags
* https://www.astronomer.io/guides/dynamic-tasks/

=Internal=
* [[Airflow_Concepts#Dynamic_DAG|Airflow Concepts]]
=Overview=
Many workflows are static, in that all the component task instances and their relationships are statically declared in the workflow definition code:
<syntaxhighlight lang='py'>
# noinspection PyPep8Naming
@task
def A():
    ...

@task
def B():
    ...

@task
def C():
    ...

@dag(...)
def some_dag():
    A() >> B() >> C()
</syntaxhighlight>

However, since the workflow definition code is Python, it allows for a certain degree of flexibility when declaring tasks in a loop:
<syntaxhighlight lang='py'>
@dag(...)
def some_dag():
  a = A()
  c = C()
  for i in [1, 2, 3]:
      a >> B() >> c
</syntaxhighlight>
The resulting DAG looks similar to:

::::[[Image:Airflow_Parallel_1.png]]
This is not really dynamic, in the sense that the structure of the DAG changes at runtime, but equivalent with declaring the same task, individually, multiple times:
<syntaxhighlight lang='py'>
@dag(...)
def some_dag():
  a = A()
  c = C()
  a >> B() >> c
  a >> B() >> c
  a >> B() >> c
</syntaxhighlight>

However, Airflow 2.3 and newer allow declaring truly dynamic DAGs, where the structure of the DAG is determined at runtime by data dynamically generated by previous tasks.







A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. <font color=darkkhaki>Tasks cannot create dynamic tasks.</font>

=Questions=
<font color=darkkhaki>
* Can dependencies be declared on dynamic tasks?
</font>