Configuring Prometheus with Prometheus Operator

External

• https://coreos.com/operators/prometheus/docs/0.17.0/index.html
• https://github.com/prometheus-operator/prometheus-operator
• https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md

Internal

• Prometheus Configuration
• Prometheus Operator Concepts

Add a Target

To add a monitoring target, deploy a ServiceMonitor custom resource:

namespace="prometheus"
cat <<EOF | kubectl -n ${namespace} apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pushgateway
  namespace: ${namespace}
  labels:
    release: prometheus
spec:
  endpoints:
  - port: 'pushgateway'
    path: /metrics
    # default interval 30s
    interval: 15s
  namespaceSelector:
    matchNames:
    - ${namespace}
  selector:
    matchLabels:
      function: pushgateway-target
EOF

⚠️ The endpoint "port" element must be a string, and it must be the name of the port exposed by the corresponding Service:

apiVersion: v1
kind: Service
...
spec:
  ports:
  - name: 'pushgateway'
    port: 9091
    targetPort: 9091
    ...

The ServiceMonitor resource must carry a label that will identify it to the Prometheus custom resource by matching the Prometheus resource "serviceMonitorSelector":

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  serviceMonitorSelector:
    matchLabels:
      release: prometheus

Also, ServiceMonitor selector must match monitored Services' labels.

If the ServiceMonitor was configured correctly, the new monitoring target surfaces in the Prometheus server configuration after a while. The configuration can be inspected from the server's console (Status → Configuration). Also the console has a "Targets" section where the target should show up: