Airflow Sensor
External
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html
- https://airflow.apache.org/docs/apache-airflow/stable/concepts/smart-sensors.html
- https://airflow.apache.org/docs/apache-airflow/2.0.0/concepts.html#sensors
Internal
Overview
A Sensor is a subclass of Operator. Sensors poll (wait and then periodically check) for an external event to happen. When the event they are waiting for occurs, the tasks succeeds, so their downstream tasks can run. The sensors are primarily idle, and because of that, they have primarily three modes of running, that allows executing them with various degrees of efficiency: poke, reschedule and smart sensors.
Also see Deferrable Operators and Triggers.
Sensor Types
Poke
poke is the default run mode for a sensor. The Sensor takes up a worker slot for its entire runtime and it sleeps between "pokes". Something that is checking every second should be in poke mode.
Reschedule
The reschedule mode can be configured when the sensor is instantiated. The sensor takes up a worker slot only when it's checking, and sleeps for a set duration between checks. reschedule trades of latency for resources. Something that is checking every minute should be in reschedule mode.
Smart Sensor
There is a single centralized version of this sensor that batches all executions of it.