Airflow Programming Model: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
Line 9: Line 9:
=Declaring a DAG=
=Declaring a DAG=
==With <tt>@dag</tt> Decorator==
==With <tt>@dag</tt> Decorator==
The <code>@dag</code> decorator turns a Python function into a DAG. This is only available in Airflow 2 and newer. Also see: {{Internal|Airflow_Concepts#Declaring_a_DAG|Airflow Concepts &#124; Declaring a DAG}}
The <code>@dag</code> decorator turns a Python function into a '''DAG generator function'''. This is only available in Airflow 2 and newer. Also see: {{Internal|Airflow_Concepts#Declaring_a_DAG|Airflow Concepts &#124; Declaring a DAG}}


<font color=darkkhaki>TO PARSE:  
<font color=darkkhaki>TO PARSE:  

Revision as of 01:25, 16 July 2022

Internal

Overview

requirements.txt

apache-airflow == 2.3.3

Declaring a DAG

With @dag Decorator

The @dag decorator turns a Python function into a DAG generator function. This is only available in Airflow 2 and newer. Also see:

Airflow Concepts | Declaring a DAG

TO PARSE:

With DAG() Constructor

Also see:

Airflow Concepts | Declaring a DAG
import time
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import PythonOperator

default_args = {
    'owner': 'somebody',
    "depends_on_past": False,
    "email": ["somebody@example.com"],
    "email_on_failure": True,
    "email_on_retry": False
}

dag = DAG('some_dag',
          max_active_runs=1,
          catchup=True,
          start_date=datetime(2022, 7, 13, 0),
          schedule_interval="* * * * *",
          dagrun_timeout=timedelta(minutes=2),
          default_args=default_args
          )


def some_function():
    print("something")


with dag:
    start = EmptyOperator(task_id='start', dag=dag)
    end = EmptyOperator(task_id='end', dag=dag)
    some_function = PythonOperator(
        task_id='some_function',
        python_callable=some_function,
        dag=dag
    )

    start >> some_function >> end

Declaring a Task

Using the PythonOperator

Using the @task Decorator

Module Management

TO PROCESS: https://airflow.apache.org/docs/apache-airflow/2.3.2/modules_management.html?highlight=import