Airflow: Difference between revisions
Jump to navigation
Jump to search
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=External= | =External= | ||
* https://airflow.apache.org | * https://airflow.apache.org/docs/apache-airflow/stable/start/index.html | ||
* https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow | |||
* Release Notes https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html | |||
=Internal= | =Internal= | ||
* [[Workflow Orchestration Tools#Overview|Workflow Orchestration Tools]] | |||
=Overview= | =Overview= | ||
Airflow is a platform to programmatically author, schedule and monitor workflows. The workflows are expressed as [[Graph_Concepts#Directed_Acyclic_Graph_.28DAG.29|DAGs]] of tasks. | Airflow is a platform to programmatically author, schedule and monitor workflows. The workflows are expressed as [[Graph_Concepts#Directed_Acyclic_Graph_.28DAG.29|DAGs]] of tasks. | ||
=Subjects= | =Subjects= | ||
* [[Airflow Concepts|Concepts]] | * [[Airflow Concepts|Concepts]] | ||
* [[Airflow Operations|Operations]] | |||
* [[Airflow Programming Model|Programming Model]] | |||
=Organizatorium= | =Organizatorium= | ||
Latest revision as of 00:17, 14 July 2022
External
- https://airflow.apache.org/docs/apache-airflow/stable/start/index.html
- https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow
- Release Notes https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html
Internal
Overview
Airflow is a platform to programmatically author, schedule and monitor workflows. The workflows are expressed as DAGs of tasks.
Subjects
Organizatorium
- Users schedule batch jobs in Airflow.
- Celery Executor: https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html
- Kubernetes Executor introduced in Airflow 1.10.0
- Dag Sync