Airflow TaskFlow: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
* https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
* https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
* https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
* https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#python-task-decorator
=Internal=
=Internal=
* [[Airflow_Concepts#TaskFlow|Airflow Concepts]]
* [[Airflow_Concepts#TaskFlow|Airflow Concepts]]
Line 13: Line 15:
from datetime import datetime
from datetime import datetime


@task
def task_x(ti=None):
  print(f"executing task X, task instance {ti}")


@dag(
@dag(
Line 26: Line 32:
         print(f"executing task B, task instance {ti}")
         print(f"executing task B, task instance {ti}")


     task_a() >> task_b()
     task_a() >> task_b() >> task_x()
</syntaxhighlight>
</syntaxhighlight>



Latest revision as of 02:07, 18 July 2022

External

Internal

Overview

Tasks can be declared as Python functions annotated with @task.

Programming Model

Functions annotated with @task are executed when the corresponding task instance is executed.

from airflow.decorators import dag, task
from datetime import datetime


@task
def task_x(ti=None):
   print(f"executing task X, task instance {ti}")

@dag(
    [...]
)
def some_dag():
    @task
    def task_a(ti=None):
        print(f"executing task A, task instance {ti}")

    @task
    def task_b(ti=None):
        print(f"executing task B, task instance {ti}")

    task_a() >> task_b() >> task_x()

If the function's first argument is ti, a reference to the corresponding task instance it will be passed with it. An equivalent key is task_instance.

TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Context

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context
https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Logging

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging