Airflow TaskFlow

External

• https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
• https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
• https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#python-task-decorator

Internal

• Airflow Concepts

Overview

Tasks can be declared as Python functions annotated with @task.

Programming Model

Functions annotated with @task are executed when the corresponding task instance is executed.

from airflow.decorators import dag, task
from datetime import datetime


@dag(
    [...]
)
def some_dag():
    @task
    def task_a(ti=None):
        print(f"executing task A, task instance {ti}")

    @task
    def task_b(ti=None):
        print(f"executing task B, task instance {ti}")

    task_a() >> task_b()

If the function's first argument is ti, a reference to the corresponding task instance it will be passed with it. An equivalent key is task_instance.

TO PROCESS: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Context

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#context

https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#templates-variables

Logging

https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html#logging