Airflow XComs: Difference between revisions
| Line 8: | Line 8: | ||
=Overview= | =Overview= | ||
Tasks communicate using inputs and outputs, and the XComs ("cross-communications") mechanism is an implementation of this pattern. Note that XComs must be relatively small. If there is need to exchange large amounts of data, this is usually done uploading and downloading large files from a storage service. | Tasks communicate using inputs and outputs, and the XComs ("cross-communications") mechanism is an implementation of this pattern. Note that XComs must be relatively small. If there is need to exchange large amounts of data, this is usually done uploading and downloading large files from a storage service. | ||
The XComs are stored in the xcom table and they need to be explicitly deleted after use, otherwise they'll leak in the table. | |||
<font color=darkkhaki>When you call a [[Airflow Concepts#TaskFlow|TaskFlow]] function in the DAG file, rather than executing it, you will get an object representing the XCom for the result (an <code>XComArg</code>, that you can use as inputs to downstream tasks and operators.</font> | <font color=darkkhaki>When you call a [[Airflow Concepts#TaskFlow|TaskFlow]] function in the DAG file, rather than executing it, you will get an object representing the XCom for the result (an <code>XComArg</code>, that you can use as inputs to downstream tasks and operators.</font> | ||
Revision as of 23:43, 11 July 2022
External
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/taskflow.html
- https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#xcoms
Internal
Overview
Tasks communicate using inputs and outputs, and the XComs ("cross-communications") mechanism is an implementation of this pattern. Note that XComs must be relatively small. If there is need to exchange large amounts of data, this is usually done uploading and downloading large files from a storage service.
The XComs are stored in the xcom table and they need to be explicitly deleted after use, otherwise they'll leak in the table.
When you call a TaskFlow function in the DAG file, rather than executing it, you will get an object representing the XCom for the result (an XComArg, that you can use as inputs to downstream tasks and operators.
Variables are an alternative mechanism for tasks to share data. However, variables are global and should be used for overall configuration that covers the entire installation. To pass data to and from tasks, XComs should be preferred instead.