Prometheus Pushgateway

External

• https://prometheus.io/docs/instrumenting/pushing/
• https://github.com/prometheus/pushgateway/blob/master/README.md
• https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/

Internal

• Prometheus Concepts

Overview

Push gateways are used in case of applications or short-lived jobs that do not export metrics directly.

Sending Metrics

Metrics are pushed into the gateway by sending a POST request to http://<gateway-host>:<gateway-port>/metrics/job/<job-name>/instance/<instance-name>.

The simplest possible body of the POST request could be:

<metric_name> <metric_value>

Example:

some_metric 12.5

Installation

Kubernetes

Deploy the pod and associated services:

apiVersion: v1
kind: Pod
metadata:
  name: 'pushgateway'
  labels:
    function: 'pushgateway'
spec:
  containers:
  - name: 'pushgateway'
    image: prom/pushgateway
---
apiVersion: v1
kind: Service
metadata:
  name: 'pushgateway'
  labels:
    function: 'pushgateway-target'
spec:
  type: 'ClusterIP'
  selector:
    function: 'pushgateway'
  ports:
    # 'name' is important, it will be referred by ServiceMonitor configuration
    - name: 'pushgateway'
      protocol: 'TCP'
      port: 9091
      targetPort: 9091
---
apiVersion: v1
kind: Service
metadata:
  name: 'pushgateway-lb'
spec:
  type: 'LoadBalancer'
  selector:
    function: 'pushgateway'
  ports:
    - name: 'pushgateway'
      protocol: 'TCP'
      port: 9091
      targetPort: 9091

Expose it then as a Prometheus scraping target with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pushgateway
  namespace: prometheus
  labels:
    release: prometheus
spec:
  endpoints:
  - port: 'pushgateway' # must match the label from Service declaration
    path: /metrics
    interval: 15s
  namespaceSelector:
    matchNames:
    - prometheus
  selector:
    matchLabels:
      function: 'pushgateway-target'

Also see:

Configuring Prometheus with Prometheus Operator | Add a Monitoring Target

Playground

https://github.com/ovidiuf/playground/tree/master/prometheus/pushgateway

Operations

Push a Metric Sample

echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job

Console

http://localhost:9091/

Query Metrics

Do not use trailing slash otherwise you'll get "Moved Permanently":

curl http://localhost:9091/metrics

Query Health

curl http://localhost:9091/-/healthy

Query Readiness

curl http://localhost:9091/-/ready