Prometheus Operator Concepts
External
- https://github.com/prometheus-operator/prometheus-operator
- https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md
Internal
Overview
The Operator decouples the deployment of Prometheus instances from the configuration of which targets they are monitoring. The Operator defines two custom resources: Prometheus and ServiceMonitor.
The Operator ensures that for each Prometheus resource there is a set of Prometheus servers that are running with the correct configuration. The operator manages the servers' configuration, including persistent volume claims, number of replicas, the server version, the Alertmanager instances, the monitoring targets and their associated parameters.
The monitoring targets and their parameters can be set manually, but the preferred option is to let the Operator generate the configuration based on declared ServiceMonitor instances.
Custom Resources
Prometheus Resource
The Prometheus resource declares a "serviceMonitorSelector":
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
serviceMonitorSelector:
matchLabels:
release: prometheus
ServiceMonitor Resource
A ServiceMonitor is a custom resource that can be used to specify a monitoring target to the Prometheus server. The ServiceMonitor describes how metrics can be retrieved from a set of Services exposing the monitoring targets' pods. The Prometheus instance is associated with the ServiceMonitors by labels: the ServiceMonitor instance must match Prometheus serviceMonitorSelector. The Operator configures the Prometheus server instances to monitor all Services that match the included ServiceMonitors and keeps this configuration synchronized with any changes happening in the cluster. The Services are associated with their ServiceMonitors with labels
ServiceMonitor Manifest
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example
labels:
# ServiceMonitor's labels must match Prometheus instance serviceMonitorSelector
release: prometheus
spec:
endpoints:
- path: /metrics
port: "9091"
namespaceSelector:
matchNames:
- prometheus
# ServiceMonitor's selector identifies matching Services
selector:
matchLabels:
function: monitoring-target-1
ServiceMonitor Operations
The current service monitor list can be obtained with:
kubectl -n <namespace> get servicemonitors