Airflow TaskFlow: Difference between revisions
Jump to navigation
Jump to search
(Created page with "=External= * https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html * https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html =I...") |
|||
| (8 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html | * https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html | ||
* https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html | * https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html | ||
* https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#python-task-decorator | |||
=Internal= | =Internal= | ||
* [[Airflow_Concepts#TaskFlow|Airflow Concepts]] | * [[Airflow_Concepts#TaskFlow|Airflow Concepts]] | ||
=Overview= | =Overview= | ||
Tasks can be declared as Python functions annotated with <code>@task</code>. | |||
=Programming Model= | |||
Functions annotated with <code>@task</code> are executed when the corresponding [[Airflow_Concepts#Task_Instance|task instance]] is executed. | |||
<syntaxhighlight lang='py'> | |||
from airflow.decorators import dag, task | |||
from datetime import datetime | |||
@task | |||
def task_x(ti=None): | |||
print(f"executing task X, task instance {ti}") | |||
@dag( | |||
[...] | |||
) | |||
def some_dag(): | |||
@task | |||
def task_a(ti=None): | |||
print(f"executing task A, task instance {ti}") | |||
@task | |||
def task_b(ti=None): | |||
print(f"executing task B, task instance {ti}") | |||
task_a() >> task_b() >> task_x() | |||
</syntaxhighlight> | |||
If the function's first argument is <code>ti</code>, a reference to the corresponding [[Airflow_Concepts#Task_Instance|task instance]] it will be passed with it. An equivalent key is <code>task_instance</code>. | |||
<font color=darkkhaki>TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables</font> | |||
=Context= | =Context= | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context}} | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables}} | |||
=Logging= | =Logging= | ||
{{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging}} | {{External|https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging}} | ||
Latest revision as of 02:07, 18 July 2022
External
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
- https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
- https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#python-task-decorator
Internal
Overview
Tasks can be declared as Python functions annotated with @task.
Programming Model
Functions annotated with @task are executed when the corresponding task instance is executed.
from airflow.decorators import dag, task
from datetime import datetime
@task
def task_x(ti=None):
print(f"executing task X, task instance {ti}")
@dag(
[...]
)
def some_dag():
@task
def task_a(ti=None):
print(f"executing task A, task instance {ti}")
@task
def task_b(ti=None):
print(f"executing task B, task instance {ti}")
task_a() >> task_b() >> task_x()
If the function's first argument is ti, a reference to the corresponding task instance it will be passed with it. An equivalent key is task_instance.
TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables