Prometheus Pushgateway

External

• https://prometheus.io/docs/instrumenting/pushing/
• https://github.com/prometheus/pushgateway/blob/master/README.md
• https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/

Internal

• Prometheus Concepts

Overview

Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.

Sending Metrics

Metrics are pushed into the gateway by sending a POST request to:

http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name>

The job name and instance name are attached as metadata to the sample, as values for the labels "job" and "instance". At least the job name must be specified. If the instance name is not specified, it will default to the empty string The simplest possible body of the POST request could be:

<metric_name> <metric_value>

Example:

some_metric 12.5

As such,

echo "some_metric 1.2" | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance

will produce:

some_metric{instance="test-instance",job="test-job"} 1.2

and

echo "some_other_metric 1.3" | curl --data-binary @- http://localhost:9091/metrics/job/other-test-job

will produce:

some_other_metric{instance="",job="other-test-job"} 1.3

Multiple metrics with additional labels can be sent as such:

cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test-job/instance/test-instance
some_metric{label="blue",label="large"} 12.1
some_other_metric{label="green"} 12.2

Installation

Kubernetes

Deploy the pod and associated services:

apiVersion: v1
kind: Pod
metadata:
  name: 'pushgateway'
  labels:
    function: 'pushgateway'
spec:
  containers:
  - name: 'pushgateway'
    image: prom/pushgateway
---
apiVersion: v1
kind: Service
metadata:
  name: 'pushgateway'
  labels:
    function: 'pushgateway-target'
spec:
  type: 'ClusterIP'
  selector:
    function: 'pushgateway'
  ports:
    # 'name' is important, it will be referred by ServiceMonitor configuration
    - name: 'pushgateway'
      protocol: 'TCP'
      port: 9091
      targetPort: 9091
---
apiVersion: v1
kind: Service
metadata:
  name: 'pushgateway-lb'
spec:
  type: 'LoadBalancer'
  selector:
    function: 'pushgateway'
  ports:
    - name: 'pushgateway'
      protocol: 'TCP'
      port: 9091
      targetPort: 9091

Expose it then as a Prometheus scraping target with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pushgateway
  namespace: prometheus
  labels:
    release: prometheus
spec:
  endpoints:
  - port: 'pushgateway' # must match the label from Service declaration
    path: /metrics
    interval: 15s
  namespaceSelector:
    matchNames:
    - prometheus
  selector:
    matchLabels:
      function: 'pushgateway-target'

Also see:

Configuring Prometheus with Prometheus Operator | Add a Monitoring Target

Playground

https://github.com/ovidiuf/playground/tree/master/prometheus/pushgateway

Operations

Push a Metric Sample

echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job

Console

http://localhost:9091/

Query Metrics

Do not use trailing slash otherwise you'll get "Moved Permanently":

curl http://localhost:9091/metrics

The result is the last sample for each timeseries.

Query Health

curl http://localhost:9091/-/healthy

Query Readiness

curl http://localhost:9091/-/ready