OpenShift Metrics Concepts
External
Internal
Overview
OpenShift comes with built in support for metric collection and display. If enabled during the installation phase, Heapster retrieves a list of all nodes from the master and then contacts each node individually through the /stats endpoint, collecting CPU, memory and network metrics. The data is sent to Hawkular Metrics, which stores them in a Cassandra database. Metrics are then exposes through the OpenShift web console, and the URL for accessing the Hawkular Metrics service is configured in master-config.yml. The API can also be directly accessed. The metrics are also available to the horizontal pod autoscalers.
The metrics components must be deployed as part of the "openshift-infra" project. The name can be changed with 'openshift_metrics_project' during installation.
Metric data can be stored in persistent storage or temporary pod volumes. The data stored in temporary pod volumes will be discarded when the pod exists. In case of persistent volumes, is the administrator’s responsibility to ensure that the storage is large enough. The size of the persistent volume claim is specified during installation with 'openshift_metrics_cassandra_pvc_size'. Metrics data is stored by default for seven days. After seven days, Cassandra purges the oldest metrics data. Metrics data for deleted pods and projects is not automatically purged; it is only removed once the data is more than seven days old. If the Cassandra persisted volume runs out of sufficient space, then data loss will occur. For more details on capacity planning see Capacity Planning for Cluster Metrics and Scaling Cluster Metrics.
The metric pods should run on dedicated infrastructure nodes.
Resources
This is the memory consumption based on a test installation:
- hawkular pod: 1GB
- cassandra pod: 960 MB
- heapster pod: 100MB
Troubleshooting
Metrics can be troubleshot with:
oadm diagnostics MetricsApiProxy
To uninstall and repeat installation:
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml -e openshift_metrics_install_metrics=false