Airflow TaskFlow

From NovaOrdis Knowledge Base
Revision as of 02:07, 18 July 2022 by Ovidiu (talk | contribs) (→‎Programming Model)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

External

Internal

Overview

Tasks can be declared as Python functions annotated with @task.

Programming Model

Functions annotated with @task are executed when the corresponding task instance is executed.

from airflow.decorators import dag, task
from datetime import datetime


@task
def task_x(ti=None):
   print(f"executing task X, task instance {ti}")

@dag(
    [...]
)
def some_dag():
    @task
    def task_a(ti=None):
        print(f"executing task A, task instance {ti}")

    @task
    def task_b(ti=None):
        print(f"executing task B, task instance {ti}")

    task_a() >> task_b() >> task_x()

If the function's first argument is ti, a reference to the corresponding task instance it will be passed with it. An equivalent key is task_instance.

TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Context

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context
https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Logging

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging