Prometheus Pushgateway: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(26 intermediate revisions by the same user not shown)
Line 2: Line 2:
* https://prometheus.io/docs/instrumenting/pushing/
* https://prometheus.io/docs/instrumenting/pushing/
* https://github.com/prometheus/pushgateway/blob/master/README.md
* https://github.com/prometheus/pushgateway/blob/master/README.md
* https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
=Internal=
=Internal=
* [[Prometheus Concepts#Pushgateway|Prometheus Concepts]]
* [[Prometheus Concepts#Pushgateway|Prometheus Concepts]]
=Overview=
=Overview=
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.
Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.
=Installation=
{{Internal|Prometheus Pushgateway Installation|Installation}}
=Sending Metrics=
{{External|https://github.com/prometheus/pushgateway#command-line}}


=Installation=
Metrics are pushed into the gateway by sending a POST request to:
==Kubernetes==
<syntaxhighlight lang='text'>
Deploy the pod and associated services:
http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name>
<syntaxhighlight lang='yaml'>
</syntaxhighlight>
apiVersion: v1
The job name and instance name are attached as metadata to the sample, as values for the labels "[[Prometheus_Concepts#Job|job]]" and "[[Prometheus_Concepts#Instance|instance]]". At least the job name must be specified. If the instance name is not specified, it will default to the empty string
kind: Pod
The simplest possible body of the POST request could be:
metadata:
<syntaxhighlight lang='text'>
  name: 'pushgateway'
<metric_name> <metric_value>
  labels:
</syntaxhighlight>
    function: 'pushgateway'
Example:
spec:
<syntaxhighlight lang='text'>
  containers:
some_metric 12.5
  - name: 'pushgateway'
</syntaxhighlight>
    image: prom/pushgateway
As such,
---
<syntaxhighlight lang='text'>
apiVersion: v1
echo "some_metric 1.2" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
kind: Service
</syntaxhighlight>
metadata:
will produce:
  name: 'pushgateway'
<syntaxhighlight lang='text'>
  labels:
some_metric{instance="test-instance",job="test-job"} 1.2
    function: 'prometheus-target'
</syntaxhighlight>
spec:
and
  type: 'ClusterIP'
<syntaxhighlight lang='text'>
  selector:
echo "some_other_metric 1.3" | curl --data-binary @- http://localhost:9091/metrics/job/other-test-job
    function: 'pushgateway'
</syntaxhighlight>
  ports:
will produce:
    - name: 'pushgateway'
<syntaxhighlight lang='text'>
      protocol: 'TCP'
some_other_metric{instance="",job="other-test-job"} 1.3
      port: 9091
</syntaxhighlight>
      targetPort: 9091
Multiple metrics with additional labels can be sent as such:
---
<syntaxhighlight lang='bash'>
apiVersion: v1
cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
kind: Service
some_metric{color="blue",size="large"} 12.1
metadata:
some_other_metric{color="green"} 12.2
  name: 'pushgateway-lb'
EOF
spec:
</syntaxhighlight>
  type: 'LoadBalancer'
Alternatively, additional label name and values can be specified in the URL:
  selector:
<syntaxhighlight lang='bash'>
    function: 'pushgateway'
echo "some_metric 12.3" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance/color/blue/size/large
  ports:
    - name: 'pushgateway'
      protocol: 'TCP'
      port: 9091
      targetPort: 9091
</syntaxhighlight>
</syntaxhighlight>
Expose it then as a target with a [[Prometheus_Operator_Concepts#ServiceMonitor_Resource|ServiceMonitor]]:
=Storing Metrics=
<syntaxhighlight lang='yaml'>
Upon receipt, Prometheus add extra labels, converting the job and instance in "exported_job" and "exported_instance":
<syntaxhighlight lang='bash'>
action_queue_depth{exported_job="test-job",exported_instance="test-instance",job="pushgateway",instance="10.1.0.33:9091",endpoint="pushgateway",namespace="prometheus",pod="pushgateway",service="pushgateway"}
</syntaxhighlight>
</syntaxhighlight>


Line 64: Line 68:
echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job
echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job
</syntaxhighlight>
</syntaxhighlight>
==Console==
http://localhost:9091/
==Query Metrics==
==Query Metrics==
Do not use trailing slash otherwise you'll get "Moved Permanently":
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
curl http://localhost:9091/metrics/
curl http://localhost:9091/metrics
</syntaxhighlight>
</syntaxhighlight>
The result is the last sample for each timeseries.
==Query Health==
==Query Health==
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
Line 76: Line 85:
curl http://localhost:9091/-/ready
curl http://localhost:9091/-/ready
</syntaxhighlight>
</syntaxhighlight>
==Configure Pushgateway as a Prometheus Target==

Latest revision as of 03:23, 16 October 2020

External

Internal

Overview

Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.

Installation

Installation

Sending Metrics

https://github.com/prometheus/pushgateway#command-line

Metrics are pushed into the gateway by sending a POST request to: 

http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name>

The job name and instance name are attached as metadata to the sample, as values for the labels "job" and "instance". At least the job name must be specified. If the instance name is not specified, it will default to the empty string The simplest possible body of the POST request could be: 

<metric_name> <metric_value>

Example: 

some_metric 12.5

As such, 

echo "some_metric 1.2" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance

will produce: 

some_metric{instance="test-instance",job="test-job"} 1.2

and 

echo "some_other_metric 1.3" | curl --data-binary @- http://localhost:9091/metrics/job/other-test-job

will produce: 

some_other_metric{instance="",job="other-test-job"} 1.3

Multiple metrics with additional labels can be sent as such: 

cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
some_metric{color="blue",size="large"} 12.1
some_other_metric{color="green"} 12.2
EOF

Alternatively, additional label name and values can be specified in the URL: 

echo "some_metric 12.3" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance/color/blue/size/large

Storing Metrics

Upon receipt, Prometheus add extra labels, converting the job and instance in "exported_job" and "exported_instance": 

action_queue_depth{exported_job="test-job",exported_instance="test-instance",job="pushgateway",instance="10.1.0.33:9091",endpoint="pushgateway",namespace="prometheus",pod="pushgateway",service="pushgateway"}

Playground

https://github.com/ovidiuf/playground/tree/master/prometheus/pushgateway

Operations

Push a Metric Sample

echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job

Console

http://localhost:9091/

Query Metrics

Do not use trailing slash otherwise you'll get "Moved Permanently": 

curl http://localhost:9091/metrics

The result is the last sample for each timeseries.

Query Health

curl http://localhost:9091/-/healthy

Query Readiness

curl http://localhost:9091/-/ready
Retrieved from "https://kb.novaordis.com/index.php?title=Prometheus_Pushgateway&oldid=72094"