Prometheus Concepts: Difference between revisions
| (52 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=External= | |||
* https://devconnected.com/the-definitive-guide-to-prometheus-in-2019/ | |||
=Internal= | =Internal= | ||
* [[Prometheus#Subjects|Prometheus]] | * [[Prometheus#Subjects|Prometheus]] | ||
=Overview= | =Overview= | ||
Prometheus is a [[Time Series#Overview|time series]] database plus tools to collect [[Monitoring_Concepts#Metric|metrics]] to be stored in the database as [[Monitoring_Concepts#Aggregated_Metric|aggregated metrics]]. Prometheus is a solution for system monitoring. It is not designed to catch individual events in time (such as a service outage) but it is designed to gather aggregated metrics about services. | Prometheus is a [[Time Series#Overview|time series]] database plus tools to collect [[Monitoring_Concepts#Metric|metrics]] to be stored in the database as [[Monitoring_Concepts#Aggregated_Metric|aggregated metrics]]. Prometheus is a solution for system monitoring. It is not designed to catch individual events in time (such as a service outage) but it is designed to gather aggregated metrics about services. Prometheus is not designed to store raw information, such as text. Prometheus provides alerting capabilities with [[#Alertmanager|Alertmanager]]. Prometheus has some built-in visualization capabilities, but [[Grafana]] can also be used. | ||
=Target= | =Target= | ||
Prometheus will monitor targets. A target can be a server, database, virtual machine, etc. Different targets can have different [[#Pull_Rate|pull rates]]. | Prometheus will monitor targets. They are also sometimes referred to as "resources". A target can be a server, database, virtual machine, etc. Different targets can have different [[#Pull_Rate|pull rates]]. Targets can be discovered dynamically with [[#Service_Discovery|service discovery]]. | ||
=Pull vs. Push= | =Pull vs. Push= | ||
| Line 19: | Line 22: | ||
Prometheus pulls metrics from instrumented applications. An instrumented application exposes Prometheus metrics on a given URL. Such an application will be identified by Prometheus as a [[#Target|target]] and scrapped at regular periods. | Prometheus pulls metrics from instrumented applications. An instrumented application exposes Prometheus metrics on a given URL. Such an application will be identified by Prometheus as a [[#Target|target]] and scrapped at regular periods. | ||
==Exporters== | ==Exporters== | ||
An alternative are prebuilt exporters. Examples: node exporter, SQL exporter, HAProxy exporter. | An alternative are prebuilt exporters. Examples: node exporter, SQL exporter, HAProxy exporter. The exporters are usually exposed as Docker images, and can be configured to monitor existing [[#Target|targets]]. Available exporters: | ||
== | ===Node Exporter=== | ||
{{Internal|Prometheus Node Exporter|Node Exporter}} | |||
==Pushgateway== | |||
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly. | Push gateways are used in case of applications or short-lived jobs that do not export metrics directly. | ||
{{Internal|Prometheus Pushgateway|Pushgateway}} | |||
=Storage= | =Storage= | ||
{{External|https://prometheus.io/docs/prometheus/latest/storage/}} | |||
{{External|https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md}} | |||
By default, the prometheus container stores metrics on an [[Kubernetes_Storage_Concepts#emptyDir|emtpyDir]] volume named "prometheus-prometheus-kube-prometheus-prometheus-db" mounted in /prometheus in the container. | |||
=Alertmanager= | |||
Prometheus pushes alerts to Alertmanager via custom rules defined in its configuration files. | |||
=Service Discovery= | |||
Prometheus can discover [[#Target|targets]] dynamically. | |||
=Metrics= | |||
A Prometheus metric is a key/value pair plus a set of labels, commonly referred to as metadata. The key is the [[#Name|name]] of the metric. The value is a float numerical value. | |||
<syntaxhighlight lang='text'> | |||
cpu_usage{core="1",ip="127.0.01"} 14.04 | |||
</syntaxhighlight> | |||
Also see: {{Internal|Monitoring_Concepts#Metric|Monitoring Concepts | Metrics}} | |||
<font color=darkgray> | |||
TODO: | |||
* https://prometheus.io/docs/concepts/data_model/ | |||
* https://prometheus.io/docs/instrumenting/exposition_formats/ | |||
</font> | |||
==<span id='Name'></span>Metric Name== | |||
==Sample== | |||
Also see [[#Instant_Vector|instant vector]] | |||
==Series== | |||
Series can be queried at the /api/v1/series endpoint. | |||
===Data Types used in Queries=== | |||
====Instant Vector==== | |||
A set of time series containing a simple [[#Sample|sample]] for each time series, all sharing the same timestamp. More: {{Internal|PromQL#Instant_Vector|PromQL | Instant Vector}} | |||
====Range Vector==== | |||
A set of time series containing a range of data points over time for each series. More: {{Internal|PromQL#Range_Vector|PromQL | Range Vector}} | |||
====Scalar==== | |||
A simple numeric floating point value. More: {{Internal|PromQL#Scalar|PromQL | Scalar}} | |||
====String==== | |||
A simple string value. More: {{Internal|PromQL#String|PromQL | String}} | |||
==Labels== | |||
===Job=== | |||
Prometheus server will attach "job" and "instance" labels to each metric: | |||
<syntaxhighlight lang='text'> | |||
go_threads{job="coredns",instance="10.1.0.9:9153",namespace="kube-system",pod="coredns-66bff467f8-pwz6r",container="coredns",...} | |||
</syntaxhighlight> | |||
===Instance=== | |||
Prometheus server will attach "job" and "instance" labels to each metric: | |||
<syntaxhighlight lang='text'> | |||
go_threads{job="coredns",instance="10.1.0.9:9153",namespace="kube-system",pod="coredns-66bff467f8-pwz6r",container="coredns",...} | |||
</syntaxhighlight> | |||
==Metric Types== | |||
===Counter=== | |||
A counter counts elements over time. A counter is a monotonically increasing positive value. A counter can only go up or can be reset. It must never decrease. | |||
===Gauge=== | |||
A gauge measures values that can go up and down. | |||
===Histogram=== | |||
===Summary=== | |||
==Prometheus Format== | |||
{{External|https://prometheus.io/docs/instrumenting/exposition_formats/}} | |||
The Prometheus format is structured plain text. | |||
=PromQL= | |||
{{Internal|PromQL|PromQL}} | |||
=Configuration= | |||
{{Internal|Prometheus Configuration|Configuration}} | |||
=Prometheus Console= | |||
Prometheus comes with a built-in console that allows performing metric queries and inspecting configuration. When Prometheus is deployed in Kubernetes and on some Kubernetes instances, such as Docker Desktop Kubernetes, access to console must be enabled explicitly by [[Prometheus_Operations#Expose_Prometheus_Console_in_Kubernetes|deploying a LoadBalancer service]]. | |||
Server configuration can be viewed with Status → Configuration. | |||
=Prometheus Operator= | |||
{{Internal|Prometheus Operator Concepts|Prometheus Operator Concepts}} | |||
=Prometheus Adapter for Kubernetes Metrics APIs= | |||
{{Internal|Prometheus Adapter for Kubernetes Metrics APIs|Prometheus Adapter for Kubernetes Metrics APIs}} | |||
Latest revision as of 02:09, 17 October 2020
External
Internal
Overview
Prometheus is a time series database plus tools to collect metrics to be stored in the database as aggregated metrics. Prometheus is a solution for system monitoring. It is not designed to catch individual events in time (such as a service outage) but it is designed to gather aggregated metrics about services. Prometheus is not designed to store raw information, such as text. Prometheus provides alerting capabilities with Alertmanager. Prometheus has some built-in visualization capabilities, but Grafana can also be used.
Target
Prometheus will monitor targets. They are also sometimes referred to as "resources". A target can be a server, database, virtual machine, etc. Different targets can have different pull rates. Targets can be discovered dynamically with service discovery.
Pull vs. Push
Prometheus will actively pull (scrap) metrics, by connecting to targets. The retrieval is done by invoking HTTP into endpoints, which are defined in the configuration.
Advantages of pull vs. push:
- Centralized control - the configuration is done on Prometheus and not on individual targets.
- Adjustable pull rate - the system has control over pull rate, and avoids being overloaded.
Pull Rate
Sources of Metrics
Instrumented Applications
Prometheus pulls metrics from instrumented applications. An instrumented application exposes Prometheus metrics on a given URL. Such an application will be identified by Prometheus as a target and scrapped at regular periods.
Exporters
An alternative are prebuilt exporters. Examples: node exporter, SQL exporter, HAProxy exporter. The exporters are usually exposed as Docker images, and can be configured to monitor existing targets. Available exporters:
Node Exporter
Pushgateway
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.
Storage
By default, the prometheus container stores metrics on an emtpyDir volume named "prometheus-prometheus-kube-prometheus-prometheus-db" mounted in /prometheus in the container.
Alertmanager
Prometheus pushes alerts to Alertmanager via custom rules defined in its configuration files.
Service Discovery
Prometheus can discover targets dynamically.
Metrics
A Prometheus metric is a key/value pair plus a set of labels, commonly referred to as metadata. The key is the name of the metric. The value is a float numerical value.
cpu_usage{core="1",ip="127.0.01"} 14.04
Also see:
TODO:
- https://prometheus.io/docs/concepts/data_model/
- https://prometheus.io/docs/instrumenting/exposition_formats/
Metric Name
Sample
Also see instant vector
Series
Series can be queried at the /api/v1/series endpoint.
Data Types used in Queries
Instant Vector
A set of time series containing a simple sample for each time series, all sharing the same timestamp. More:
Range Vector
A set of time series containing a range of data points over time for each series. More:
Scalar
A simple numeric floating point value. More:
String
A simple string value. More:
Labels
Job
Prometheus server will attach "job" and "instance" labels to each metric:
go_threads{job="coredns",instance="10.1.0.9:9153",namespace="kube-system",pod="coredns-66bff467f8-pwz6r",container="coredns",...}
Instance
Prometheus server will attach "job" and "instance" labels to each metric:
go_threads{job="coredns",instance="10.1.0.9:9153",namespace="kube-system",pod="coredns-66bff467f8-pwz6r",container="coredns",...}
Metric Types
Counter
A counter counts elements over time. A counter is a monotonically increasing positive value. A counter can only go up or can be reset. It must never decrease.
Gauge
A gauge measures values that can go up and down.
Histogram
Summary
Prometheus Format
The Prometheus format is structured plain text.
PromQL
Configuration
Prometheus Console
Prometheus comes with a built-in console that allows performing metric queries and inspecting configuration. When Prometheus is deployed in Kubernetes and on some Kubernetes instances, such as Docker Desktop Kubernetes, access to console must be enabled explicitly by deploying a LoadBalancer service.
Server configuration can be viewed with Status → Configuration.