Airflow TaskFlow: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 2: Line 2:
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
* https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
* https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
* https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#python-task-decorator
=Internal=
=Internal=
* [[Airflow_Concepts#TaskFlow|Airflow Concepts]]
* [[Airflow_Concepts#TaskFlow|Airflow Concepts]]
=Overview=
=Overview=
Tasks can be declared as Python functions annotated with <code>@task</code>.
=Programming Model=
=Programming Model=
Functions annotated with <code>@task</code> are executed when the corresponding [[Airflow_Concepts#Task_Instance|task instance]] is executed.  
Functions annotated with <code>@task</code> are executed when the corresponding [[Airflow_Concepts#Task_Instance|task instance]] is executed.  
Line 11: Line 15:
from datetime import datetime
from datetime import datetime


@task
def task_x(ti=None):
  print(f"executing task X, task instance {ti}")


@dag(
@dag(
Line 24: Line 32:
         print(f"executing task B, task instance {ti}")
         print(f"executing task B, task instance {ti}")


     task_a() >> task_b()
     task_a() >> task_b() >> task_x()
</syntaxhighlight>
</syntaxhighlight>


If the function's first argument is <code>ti</code>, a reference to the corresponding  [[Airflow_Concepts#Task_Instance|task instance]] it will be passed with it.
If the function's first argument is <code>ti</code>, a reference to the corresponding  [[Airflow_Concepts#Task_Instance|task instance]] it will be passed with it. An equivalent key is <code>task_instance</code>.
 
<font color=darkkhaki>TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables</font>


=Context=
=Context=

Latest revision as of 02:07, 18 July 2022

External

Internal

Overview

Tasks can be declared as Python functions annotated with @task.

Programming Model

Functions annotated with @task are executed when the corresponding task instance is executed.

from airflow.decorators import dag, task
from datetime import datetime


@task
def task_x(ti=None):
   print(f"executing task X, task instance {ti}")

@dag(
    [...]
)
def some_dag():
    @task
    def task_a(ti=None):
        print(f"executing task A, task instance {ti}")

    @task
    def task_b(ti=None):
        print(f"executing task B, task instance {ti}")

    task_a() >> task_b() >> task_x()

If the function's first argument is ti, a reference to the corresponding task instance it will be passed with it. An equivalent key is task_instance.

TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Context

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context
https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Logging

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging