Configuring Prometheus with Prometheus Operator

From NovaOrdis Knowledge Base
Revision as of 00:45, 15 October 2020 by Ovidiu (talk | contribs) (→‎Add a Target)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

External

Internal

Overview

Add a Monitoring Target

To add a monitoring target, deploy a ServiceMonitor custom resource:

namespace="prometheus"
cat <<EOF | kubectl -n ${namespace} apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pushgateway
  namespace: ${namespace}
  labels:
    release: prometheus
spec:
  endpoints:
  - port: 'pushgateway'
    path: /metrics
    # default interval 30s
    interval: 15s
  namespaceSelector:
    matchNames:
    - ${namespace}
  selector:
    matchLabels:
      function: pushgateway-target
EOF

⚠️ The endpoint "port" element must be a string, and it must be the name of the port exposed by the corresponding Service:

apiVersion: v1
kind: Service
...
spec:
  ports:
  - name: 'pushgateway'
    port: 9091
    targetPort: 9091
    ...

The ServiceMonitor resource must carry a label that will identify it to the Prometheus custom resource by matching the Prometheus resource "serviceMonitorSelector":

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  serviceMonitorSelector:
    matchLabels:
      release: prometheus

Also, ServiceMonitor selector must match monitored Services' labels.

If the ServiceMonitor was configured correctly, the new monitoring target surfaces in the Prometheus server configuration after a while. The configuration can be inspected from the server's console (Status → Configuration). Also the console has a "Targets" section where the target should show up: SuccessfulPrometheusTarget.png