Prometheus Adapter for Kubernetes Metrics APIs: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
Line 51: Line 51:
</syntaxhighlight>
</syntaxhighlight>
instead of using an overrides map. The list of resources, including the custom resources, is provided by [[Kubernetes_API_Resources_Concepts#API_Resources|kubectl api-resources]].
instead of using an overrides map. The list of resources, including the custom resources, is provided by [[Kubernetes_API_Resources_Concepts#API_Resources|kubectl api-resources]].
The name of the metric can be changed with a "name" translation rule:
<syntaxhighlight lang='yaml'>
rules:
  - ...
    name:
      matches: "^(.*)_total"
      as: "${1}_per_second"
</syntaxhighlight>


=Helm Installation=
=Helm Installation=

Revision as of 05:37, 15 October 2020

External

Internal

Overview

The Prometheus adapter is a Kubernetes Aggregation Layer extension and operates as an extension API server. It knows how to communicate with both Kubernetes and Prometheus, acting as a translator between the two. The API is registered as "custom.metrics.k8s.io/v1beta1".

Metrics and Autoscaling PrometheusAdapter.png

How it Works

The adapter processes the metrics coming from Prometheus as follows:

Discovery: it discovers available metrics.

Association: it determines which kubernetes resource each metric is associated with.

Naming: it determines how it should expose the metrics in the custom metric API.

Querying: Finally, it figures out how it should query Prometheus to get the actual numbers.

The adapter performs each of the steps for each metric. These steps are formally described for each metric with a rule:

rules:
  - {...}
  - {...}

The discovery step is defined by a "seriesQuery":

rules:
  - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'

The association step maps labels to known resources. It is introduced by the "resources" keyword, followed by an "overrides" map, where labels are mapped to known resources.

rules:
  - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}

This says that each label represents its corresponding resource. For resources in the "core" kubernetes API the group does not need to be specified. The adapter will handle pluralization, "pod" is equivalent with "pods". The resources can be any resource available in your kubernetes cluster, as long as you've got a corresponding label.

If labels follow a consistent pattern, like "kubernetes_<resource>", the mapping can be done with an expression similar to:

resources: {template: "kubernetes_<<.Resource>>"}

instead of using an overrides map. The list of resources, including the custom resources, is provided by kubectl api-resources.

The name of the metric can be changed with a "name" translation rule:

rules:
  - ...
    name:
      matches: "^(.*)_total"
      as: "${1}_per_second"

Helm Installation

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter

Install Prometheus first.

The adapter will be installed in the same namespace as Prometheus.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community
helm install -n prom prometheus-adapter prometheus-community/prometheus-adapter -f ./prometheus-adapter-overlay.yaml

See Configuration below for prometheus-adapter-overlay.yaml.

Test installation:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "jobs.batch/kubernetes_build_info",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "services/kube_deployment_spec_strategy_rollingupdate_max_surge",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
   ...
}

Configuration

Use the following prometheus-adapter-overlay.yaml:

helm show values prometheus-community/prometheus-adapter > ~/tmp/prometheus-adapter-overlay.yaml
prometheus:
  url: http://prometheus-kube-prometheus-prometheus.prom.svc

Increase log level when installing the chart.

Test:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests?selector=app%3Dsample-app"


Configuration Resources:

Troubleshooting

tail the logs, Prometheus connection refused is reported there.