Airflow Dynamic Task Mapping: Difference between revisions
Jump to navigation
Jump to search
| Line 7: | Line 7: | ||
* [[Airflow_Concepts#Dynamic_DAG|Airflow Concepts]] | * [[Airflow_Concepts#Dynamic_DAG|Airflow Concepts]] | ||
=Overview= | =Overview= | ||
A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. <font color=darkkhaki>Tasks cannot create dynamic tasks.</font> | A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. <font color=darkkhaki>Tasks cannot create dynamic tasks.</font> | ||
=Organizatorium= | =Organizatorium= | ||
Revision as of 20:51, 18 July 2022
External
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#dynamic-dags
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html
- https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html
Internal
Overview
A workflow can create a number of tasks at runtime, based upon current workflow instance state, rather than the DAG author having to know in advance how many tasks would be needed. However, the tasks can be created in the DAG definition itself. Tasks cannot create dynamic tasks.
Organizatorium
TO PROCESS:
- https://www.astronomer.io/guides/dynamic-tasks/
- https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html