Prometheus Pushgateway: Difference between revisions
Jump to navigation
Jump to search
| (18 intermediate revisions by the same user not shown) | |||
| Line 8: | Line 8: | ||
=Overview= | =Overview= | ||
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly. | Push gateways are used in case of applications or short-lived jobs that do not export metrics directly. | ||
=Installation= | |||
{{Internal|Prometheus Pushgateway Installation|Installation}} | |||
=Sending Metrics= | |||
{{External|https://github.com/prometheus/pushgateway#command-line}} | |||
Metrics are pushed into the gateway by sending a POST request to: | |||
<syntaxhighlight lang='text'> | |||
http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name> | |||
<syntaxhighlight lang=' | </syntaxhighlight> | ||
The job name and instance name are attached as metadata to the sample, as values for the labels "[[Prometheus_Concepts#Job|job]]" and "[[Prometheus_Concepts#Instance|instance]]". At least the job name must be specified. If the instance name is not specified, it will default to the empty string | |||
The simplest possible body of the POST request could be: | |||
metadata: | <syntaxhighlight lang='text'> | ||
<metric_name> <metric_value> | |||
</syntaxhighlight> | |||
Example: | |||
<syntaxhighlight lang='text'> | |||
some_metric 12.5 | |||
</syntaxhighlight> | |||
As such, | |||
<syntaxhighlight lang='text'> | |||
echo "some_metric 1.2" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance | |||
</syntaxhighlight> | |||
will produce: | |||
<syntaxhighlight lang='text'> | |||
some_metric{instance="test-instance",job="test-job"} 1.2 | |||
</syntaxhighlight> | |||
and | |||
<syntaxhighlight lang='text'> | |||
echo "some_other_metric 1.3" | curl --data-binary @- http://localhost:9091/metrics/job/other-test-job | |||
</syntaxhighlight> | |||
will produce: | |||
<syntaxhighlight lang='text'> | |||
some_other_metric{instance="",job="other-test-job"} 1.3 | |||
</syntaxhighlight> | |||
Multiple metrics with additional labels can be sent as such: | |||
<syntaxhighlight lang='bash'> | |||
--- | cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance | ||
some_metric{color="blue",size="large"} 12.1 | |||
some_other_metric{color="green"} 12.2 | |||
EOF | |||
</syntaxhighlight> | |||
Alternatively, additional label name and values can be specified in the URL: | |||
<syntaxhighlight lang='bash'> | |||
echo "some_metric 12.3" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance/color/blue/size/large | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=Storing Metrics= | |||
<syntaxhighlight lang=' | Upon receipt, Prometheus add extra labels, converting the job and instance in "exported_job" and "exported_instance": | ||
<syntaxhighlight lang='bash'> | |||
action_queue_depth{exported_job="test-job",exported_instance="test-instance",job="pushgateway",instance="10.1.0.33:9091",endpoint="pushgateway",namespace="prometheus",pod="pushgateway",service="pushgateway"} | |||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 91: | Line 75: | ||
curl http://localhost:9091/metrics | curl http://localhost:9091/metrics | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The result is the last sample for each timeseries. | |||
==Query Health== | ==Query Health== | ||
Latest revision as of 03:23, 16 October 2020
External
- https://prometheus.io/docs/instrumenting/pushing/
- https://github.com/prometheus/pushgateway/blob/master/README.md
- https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
Internal
Overview
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.
Installation
Sending Metrics
Metrics are pushed into the gateway by sending a POST request to:
http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name>
The job name and instance name are attached as metadata to the sample, as values for the labels "job" and "instance". At least the job name must be specified. If the instance name is not specified, it will default to the empty string The simplest possible body of the POST request could be:
<metric_name> <metric_value>
Example:
some_metric 12.5
As such,
echo "some_metric 1.2" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
will produce:
some_metric{instance="test-instance",job="test-job"} 1.2
and
echo "some_other_metric 1.3" | curl --data-binary @- http://localhost:9091/metrics/job/other-test-job
will produce:
some_other_metric{instance="",job="other-test-job"} 1.3
Multiple metrics with additional labels can be sent as such:
cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
some_metric{color="blue",size="large"} 12.1
some_other_metric{color="green"} 12.2
EOF
Alternatively, additional label name and values can be specified in the URL:
echo "some_metric 12.3" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance/color/blue/size/large
Storing Metrics
Upon receipt, Prometheus add extra labels, converting the job and instance in "exported_job" and "exported_instance":
action_queue_depth{exported_job="test-job",exported_instance="test-instance",job="pushgateway",instance="10.1.0.33:9091",endpoint="pushgateway",namespace="prometheus",pod="pushgateway",service="pushgateway"}
Playground
Operations
Push a Metric Sample
echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job
Console
Query Metrics
Do not use trailing slash otherwise you'll get "Moved Permanently":
curl http://localhost:9091/metrics
The result is the last sample for each timeseries.
Query Health
curl http://localhost:9091/-/healthy
Query Readiness
curl http://localhost:9091/-/ready