Prometheus Operator Concepts

External

• https://github.com/prometheus-operator/prometheus-operator
• https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md

Internal

• Prometheus Concepts

Overview

The Operator decouples the deployment of Prometheus instances from the configuration of which targets they are monitoring. The Operator defines two custom resources: Prometheus and ServiceMonitor.

The Operator ensures that for each Prometheus resource there is a set of Prometheus servers that are running with the correct configuration. The operator manages the servers' configuration, including persistent volume claims, number of replicas, the server version, the Alertmanager instances, the monitoring targets and their associated parameters.

The monitoring targets and their parameters can be set manually, but the preferred option is to let the Operator generate the configuration based on declared ServiceMonitor instances.

Custom Resources

Prometheus Resource

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheus

The Prometheus resource declares a "serviceMonitorSelector":

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  serviceMonitorSelector:
    matchLabels:
      release: prometheus

ServiceMonitor Resource

A ServiceMonitor is a custom resource that can be used to specify a monitoring target to the Prometheus server. The ServiceMonitor describes how metrics can be retrieved from a set of Services exposing the monitoring targets' pods. The Prometheus instance is associated with the ServiceMonitors by labels: the ServiceMonitor instance must match Prometheus serviceMonitorSelector. The Operator configures the Prometheus server instances to monitor all Services that match the included ServiceMonitors and keeps this configuration synchronized with any changes happening in the cluster. The Services are associated with their ServiceMonitors with labels

ServiceMonitor Manifest

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example
  labels:
    # ServiceMonitor's labels must match Prometheus instance  serviceMonitorSelector
    release: prometheus
spec:
  endpoints:
  - path: /metrics
    port: "9091"
  namespaceSelector:
    matchNames:
    - prometheus
  # ServiceMonitor's selector identifies matching Services 
  selector:
    matchLabels:
      function: monitoring-target-1

ServiceMonitor Operations

The current service monitor list can be obtained with:

kubectl -n <namespace> get servicemonitors

Configuring Prometheus with Prometheus Operator

Configuring Prometheus with Prometheus Operator