Prometheus Concepts
Internal
Overview
Prometheus is a time series database plus tools to collect metrics to be stored in the database as aggregated metrics. Prometheus is a solution for system monitoring. It is not designed to catch individual events in time (such as a service outage) but it is designed to gather aggregated metrics about services. Prometheus also provides alerting capabilities with Alertmanager. Prometheus has some built-in visualization capabilities, but Grafana can also be used.
Target
Prometheus will monitor targets. A target can be a server, database, virtual machine, etc. Different targets can have different pull rates.
Pull vs. Push
Prometheus will actively pull (scrap) metrics, by connecting to targets. The retrieval is done by invoking HTTP into endpoints, which are defined in the configuration.
Advantages of pull vs. push:
- Centralized control - the configuration is done on Prometheus and not on individual targets.
- Adjustable pull rate - the system has control over pull rate, and avoids being overloaded.
Pull Rate
Sources of Metrics
Instrumented Applications
Prometheus pulls metrics from instrumented applications. An instrumented application exposes Prometheus metrics on a given URL. Such an application will be identified by Prometheus as a target and scrapped at regular periods.
Exporters
An alternative are prebuilt exporters. Examples: node exporter, SQL exporter, HAProxy exporter.
Push Gateway
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.
Storage
Alertmanager
Prometheus pushes alerts to Alertmanager via custom rules defined in its configuration files.