Configuring Prometheus with Prometheus Operator

External

• https://coreos.com/operators/prometheus/docs/0.17.0/index.html
• https://github.com/prometheus-operator/prometheus-operator

Internal

• Prometheus Configuration

Add a Target

To add a monitoring target, deploy a ServiceMonitor custom resource:

namespace="prometheus"
cat <<EOF | kubectl -n ${namespace} apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pushgateway
  namespace: ${namespace}
  labels:
    release: prometheus
spec:
  endpoints:
  - port: pushgateway-standard-port
    targetPort: 9091
    path: /metrics
  namespaceSelector:
    matchNames:
    - ${namespace}
  selector:
    matchLabels:
      function: pushgateway-target
EOF

The ServiceMonitor resource must carry a label that will identify it to the Prometheus custom resource by matching the Prometheus resource "serviceMonitorSelector":

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  serviceMonitorSelector:
    matchLabels:
      release: prometheus

Also, ServiceMonitor selector must match monitored Services' labels.

If the ServiceMonitor was configured correctly, the new monitoring target surfaces in the Prometheus server configuration, which can be inspected from the server's console (Status → Configuration). Also the console has a "Targets" section where the target should show up.