OpenShift Resource Management Concepts: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(69 intermediate revisions by the same user not shown)
Line 3: Line 3:
* https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-guide-compute-resources
* https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-guide-compute-resources
* https://docs.openshift.com/container-platform/latest/admin_guide/quota.html
* https://docs.openshift.com/container-platform/latest/admin_guide/quota.html
* https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/resource_management_guide/


=Internal=
=Internal=


* [[OpenShift Concepts#Resource_Management|OpenShift Concepts]]
* [[OpenShift Concepts#Resource_Management|OpenShift Concepts]]
* [[Linux Resource Management]]
* [[Kubernetes Resource Management Concepts]]


=Overview=
=Overview=


OpenShift provides API-level support for establishing and enforcing [[#Quota|resource quotas]]. The runtime monitors resource usage and [[#Resource_Consumption_Enforcement|intervenes]] when the quotas are reached or exceeded. [[#Quota|Resource quotas]] can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on [[#CPU_Usage|CPU]] and [[#Memory_Usage|memory]]) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. [[#Opaque_Integer_Resources|Opaque integer resources]] can also be set and monitored.
OpenShift provides API-level support for establishing and enforcing [[#Quota|resource quotas]]. The runtime monitors resource usage and [[#Quota_Enforcement|intervenes]] when the quotas are reached or exceeded. [[#Quota|Resource quotas]] can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on [[#CPU_Usage|CPU]] and [[#Memory_Usage|memory]]) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. [[#Opaque_Integer_Resources|Opaque integer resources]] can also be set and monitored.


Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.
Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.
Line 18: Line 21:
A ''resource quota'' specifies constraints that limit aggregate resource consumption ''per project'', and they are set by cluster administrators. The resource quota for a project is defined by a <tt>[[Resource Quota Definition#Overview|ResourceQuota]]</tt> object. Resource quotas per cluster can be managed with <tt>ClusterResourceQuota</tt>. The resource quota limits:
A ''resource quota'' specifies constraints that limit aggregate resource consumption ''per project'', and they are set by cluster administrators. The resource quota for a project is defined by a <tt>[[Resource Quota Definition#Overview|ResourceQuota]]</tt> object. Resource quotas per cluster can be managed with <tt>ClusterResourceQuota</tt>. The resource quota limits:
* the quantity of objects, per type, that can be created in a project:
* the quantity of objects, per type, that can be created in a project:
** ConfigMaps
** Pods ("pods") - the total number of pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** Persistent Volume Claims
** ConfigMaps ("configmaps")
** Replication Controllers
** Replication Controllers ("replicationcontrollers")
** Secrets
** Secrets ("secrets")
** Services
** Services ("services")
* the total amount of compute resources consumed by the project.
** Image Streams ("openshift.io/imagestreams")
* the total amount of storage consumed by the project.
** Resource Quotas ("resourcequotas")
* the total amount of compute resources consumed by the project. Note that if a quota has a value specified for "requests.cpu" or "requests.memory", then it requires that every incoming container make an explicit request for those resources. The same rule applies for "limits.cpu" and "limits.memory":
** [[#CPU_Request|CPU Requests]] ("cpu" and "requests.cpu" are equivalent) - the sum of CPU requests across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#CPU_Limit|CPU Limits]] ("limits.cpu") - the sum of CPU limits across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#Memory_Request|Memory Requests]] ("memory" and "requests.memory" are equivalent) - the sum of memory requests across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#Memory_Limit|Memory Limits]] ("limits.memory") - the sum of CPU limits across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
* the total amount of storage consumed by the project:
** Persistent Volume Claims ("persistentvolumeclaims")
** "requests.storage" - across all persistent volume claims in the project, the sum of storage requests cannot exceed this value.
** "gold.storageclass.storage.k8s.io/persistentvolumeclaims", "gold.storageclass.storage.k8s.io/requests.storage"
** "silver.storageclass.storage.k8s.io/persistentvolumeclaims", "silver.storageclass.storage.k8s.io/requests.storage"
** "bronze.storageclass.storage.k8s.io/persistentvolumeclaims", "bronze.storageclass.storage.k8s.io/requests.storage"


All quotas for a project can be obtained with [[OpenShift_Resource_Management_Operations#Get_Quotas_in_a_Project|oc get quota]], and information about individual quotas can be obtained with [[OpenShift_Resource_Management_Operations#Information_about_an_Individual_Quota|oc describe quota]]. Quotas can also be viewed from the web console, in the project's "Quota" page. Quotas can be created with [[OpenShift_Resource_Management_Operations#Creating_a_Resource_Quota|oc create]].
==Quota Scope==


=Limit Range=
The compute resource quotas can be restricted only to pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]] within a certain ''scope''. Each quota can have an associated set of scopes, and the quota will only measure usage for a resource if it matches the ''intersection'' of enumerated scopes. The scope can designate the pod's [[#Quality_of_Service|quality of service]] ("[[#BestEffort|BestEffort]]", "NotBestEffort"), or type ("[[OpenShift Concepts#NonTerminating|NonTerminating]]", "[[OpenShift Concepts#Terminating|Terminating]]").


=CPU Usage=
apiVersion: v1
kind: ResourceQuota
spec:
  ...
  scopes:
  - BestEffort 2


==CPU Request==
A "BestEffort" scope restricts quota to limiting the number of pods ("pods").


==CPU Limit==
A "Terminating", "NotTerminating" and "NotBestEffort" scope restricts a quota to tracking the following resources: "pods", "requests.memory"/"memory", "limits.memory" "requests.cpu"/"cpu", "limits.cpu".


=Memory Usage=
==Quota Enforcement==


==Memory Request==
After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. Once the usage statistics are updated, content can be created or modified, but only if by doing so quota is not exceeded. If the quota is exceeded, the action will be denied and an error message will be returned. The quota will be modified immediately after creation/update. When a resource is deleted, the quota is decremented during the next full recalculation of quota statistics per project.


==Memory Limit==
==Quota Operations==


=Opaque Integer Resources=
All quotas for a project can be obtained with [[OpenShift_Resource_Management_Operations#Get_Quotas_in_a_Project|oc get quota]], and information about individual quotas can be obtained with [[OpenShift_Resource_Management_Operations#Information_about_an_Individual_Quota|oc describe quota]]. Quotas can also be viewed from the web console, in the project's "Quota" page. Quotas can be created with [[OpenShift_Resource_Management_Operations#Creating_a_Resource_Quota|oc create]].


=Resource Consumption Enforcement=
=Limit Range=


After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. If the resource creation request exceeds the quota, the server will deny the action and will return an error message.
{{External|https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-limit-ranges}}


When a resource is created, the quota usage is updated immediately. When a resource is delete, the quota usage is updated during the next full per-project statistics update.
A limit range enumerates compute resource constraints in a project a the pod/container/image/image stream/persistent volume claim level, and specifies the amount of resources that a pod/container/image/image stream/persistent volume claim can consume. Limit ranges are defined by the <tt>LimitRange</tt> object. All resource creation and modification requests are evaluated against each limit range in the project and rejected if the resource request is outside limits. If the resource does not set an explicit value, and if the constraint supports a default value, the default value is applied to the resource. Limit ranges are set by the cluster administrators and are project-scoped.


=Organizatorium=
The limits are accessed with:


oc get limits [-n ''project-name'']


=Opaque Integer Resources=


==Enforcement==
{{External|https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#opaque-integer-resources-dev}}


<font color=red>With cgroups?</font>
=Compute Resources for System Daemons=


===Limits===
{{External|https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/}}


====Memory Limit====
=Handling Out of Resource Events=


Propagates as [[Docker_Container_Downward_API#memory.limit_in_bytes|/sys/fs/cgroup/memory/memory.limit_in_bytes]] in container.
{{External|https://access.redhat.com/documentation/en-us/openshift_container_platform/3.7/html/cluster_administration/admin-guide-handling-out-of-resource-errors}}


===Requests===
=Access to Resource Specification from Pods=


====Memory Request====
...
        - env:
          - name: FLUENTD_CPU_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: fluentd-elasticsearch
                divisor: "0"
                resource: limits.cpu
          - name: FLUENTD_MEMORY_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: fluentd-elasticsearch
                divisor: "0"
                resource: limits.memory
...

Latest revision as of 18:29, 25 August 2020

External

Internal

Overview

OpenShift provides API-level support for establishing and enforcing resource quotas. The runtime monitors resource usage and intervenes when the quotas are reached or exceeded. Resource quotas can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on CPU and memory) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. Opaque integer resources can also be set and monitored.

Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.

Quota

A resource quota specifies constraints that limit aggregate resource consumption per project, and they are set by cluster administrators. The resource quota for a project is defined by a ResourceQuota object. Resource quotas per cluster can be managed with ClusterResourceQuota. The resource quota limits:

  • the quantity of objects, per type, that can be created in a project:
    • Pods ("pods") - the total number of pods in a non-terminal state.
    • ConfigMaps ("configmaps")
    • Replication Controllers ("replicationcontrollers")
    • Secrets ("secrets")
    • Services ("services")
    • Image Streams ("openshift.io/imagestreams")
    • Resource Quotas ("resourcequotas")
  • the total amount of compute resources consumed by the project. Note that if a quota has a value specified for "requests.cpu" or "requests.memory", then it requires that every incoming container make an explicit request for those resources. The same rule applies for "limits.cpu" and "limits.memory":
  • the total amount of storage consumed by the project:
    • Persistent Volume Claims ("persistentvolumeclaims")
    • "requests.storage" - across all persistent volume claims in the project, the sum of storage requests cannot exceed this value.
    • "gold.storageclass.storage.k8s.io/persistentvolumeclaims", "gold.storageclass.storage.k8s.io/requests.storage"
    • "silver.storageclass.storage.k8s.io/persistentvolumeclaims", "silver.storageclass.storage.k8s.io/requests.storage"
    • "bronze.storageclass.storage.k8s.io/persistentvolumeclaims", "bronze.storageclass.storage.k8s.io/requests.storage"

Quota Scope

The compute resource quotas can be restricted only to pods in a non-terminal state within a certain scope. Each quota can have an associated set of scopes, and the quota will only measure usage for a resource if it matches the intersection of enumerated scopes. The scope can designate the pod's quality of service ("BestEffort", "NotBestEffort"), or type ("NonTerminating", "Terminating").

apiVersion: v1
kind: ResourceQuota
spec:
  ...
  scopes:
  - BestEffort 2

A "BestEffort" scope restricts quota to limiting the number of pods ("pods").

A "Terminating", "NotTerminating" and "NotBestEffort" scope restricts a quota to tracking the following resources: "pods", "requests.memory"/"memory", "limits.memory" "requests.cpu"/"cpu", "limits.cpu".

Quota Enforcement

After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. Once the usage statistics are updated, content can be created or modified, but only if by doing so quota is not exceeded. If the quota is exceeded, the action will be denied and an error message will be returned. The quota will be modified immediately after creation/update. When a resource is deleted, the quota is decremented during the next full recalculation of quota statistics per project.

Quota Operations

All quotas for a project can be obtained with oc get quota, and information about individual quotas can be obtained with oc describe quota. Quotas can also be viewed from the web console, in the project's "Quota" page. Quotas can be created with oc create.

Limit Range

https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-limit-ranges

A limit range enumerates compute resource constraints in a project a the pod/container/image/image stream/persistent volume claim level, and specifies the amount of resources that a pod/container/image/image stream/persistent volume claim can consume. Limit ranges are defined by the LimitRange object. All resource creation and modification requests are evaluated against each limit range in the project and rejected if the resource request is outside limits. If the resource does not set an explicit value, and if the constraint supports a default value, the default value is applied to the resource. Limit ranges are set by the cluster administrators and are project-scoped.

The limits are accessed with:

oc get limits [-n project-name]

Opaque Integer Resources

https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#opaque-integer-resources-dev

Compute Resources for System Daemons

https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/

Handling Out of Resource Events

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.7/html/cluster_administration/admin-guide-handling-out-of-resource-errors

Access to Resource Specification from Pods

...
       - env:
         - name: FLUENTD_CPU_LIMIT
           valueFrom:
             resourceFieldRef:
               containerName: fluentd-elasticsearch
               divisor: "0"
               resource: limits.cpu
         - name: FLUENTD_MEMORY_LIMIT
           valueFrom:
             resourceFieldRef:
               containerName: fluentd-elasticsearch
               divisor: "0"
               resource: limits.memory
...