OpenShift Resource Management Concepts: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(83 intermediate revisions by the same user not shown)
Line 3: Line 3:
* https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-guide-compute-resources
* https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-guide-compute-resources
* https://docs.openshift.com/container-platform/latest/admin_guide/quota.html
* https://docs.openshift.com/container-platform/latest/admin_guide/quota.html
* https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/resource_management_guide/


=Internal=
=Internal=


* [[OpenShift Concepts#Resource_Management|OpenShift Concepts]]
* [[OpenShift Concepts#Resource_Management|OpenShift Concepts]]
* [[Linux Resource Management]]
* [[Kubernetes Resource Management Concepts]]


=Overview=
=Overview=


OpenShift provides API-level support for establishing and enforcing [[#Quota|resource quotas]]. The runtime monitors resource usage and [[#Resource_Consumption_Enforcement|intervenes]] when the quotas are reached or exceeded. [[#Quota|Resource quotas]] can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on [[#CPU_Usage|CPU]] and [[#Memory_Usage|memory]]) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. [[#Opaque_Integer_Resources|Opaque integer resources]] can also be set and monitored.
OpenShift provides API-level support for establishing and enforcing [[#Quota|resource quotas]]. The runtime monitors resource usage and [[#Quota_Enforcement|intervenes]] when the quotas are reached or exceeded. [[#Quota|Resource quotas]] can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on [[#CPU_Usage|CPU]] and [[#Memory_Usage|memory]]) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. [[#Opaque_Integer_Resources|Opaque integer resources]] can also be set and monitored.


Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.
Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.
Line 16: Line 19:
=Quota=
=Quota=


A ''resource quota'' specifies constraints that limit aggregate resource consumption per project. The resource quota is defined by a <tt>[[Resource Quota Definition#Overview|ResourceQuota]]</tt> object.
A ''resource quota'' specifies constraints that limit aggregate resource consumption ''per project'', and they are set by cluster administrators. The resource quota for a project is defined by a <tt>[[Resource Quota Definition#Overview|ResourceQuota]]</tt> object. Resource quotas per cluster can be managed with <tt>ClusterResourceQuota</tt>. The resource quota limits:
* the quantity of objects, per type, that can be created in a project:
** Pods ("pods") - the total number of pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** ConfigMaps ("configmaps")
** Replication Controllers ("replicationcontrollers")
** Secrets ("secrets")
** Services ("services")
** Image Streams ("openshift.io/imagestreams")
** Resource Quotas ("resourcequotas")
* the total amount of compute resources consumed by the project. Note that if a quota has a value specified for "requests.cpu" or "requests.memory", then it requires that every incoming container make an explicit request for those resources. The same rule applies for "limits.cpu" and "limits.memory":
** [[#CPU_Request|CPU Requests]] ("cpu" and "requests.cpu" are equivalent) - the sum of CPU requests across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#CPU_Limit|CPU Limits]] ("limits.cpu") - the sum of CPU limits across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#Memory_Request|Memory Requests]] ("memory" and "requests.memory" are equivalent) - the sum of memory requests across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
** [[#Memory_Limit|Memory Limits]] ("limits.memory") - the sum of CPU limits across all pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]].
* the total amount of storage consumed by the project:
** Persistent Volume Claims ("persistentvolumeclaims")
** "requests.storage" - across all persistent volume claims in the project, the sum of storage requests cannot exceed this value.
** "gold.storageclass.storage.k8s.io/persistentvolumeclaims", "gold.storageclass.storage.k8s.io/requests.storage"
** "silver.storageclass.storage.k8s.io/persistentvolumeclaims", "silver.storageclass.storage.k8s.io/requests.storage"
** "bronze.storageclass.storage.k8s.io/persistentvolumeclaims", "bronze.storageclass.storage.k8s.io/requests.storage"


=Limit Range=
==Quota Scope==


=CPU Usage=
The compute resource quotas can be restricted only to pods in a [[OpenShift_Concepts#Terminal_State|non-terminal state]] within a certain ''scope''. Each quota can have an associated set of scopes, and the quota will only measure usage for a resource if it matches the ''intersection'' of enumerated scopes. The scope can designate the pod's [[#Quality_of_Service|quality of service]] ("[[#BestEffort|BestEffort]]", "NotBestEffort"), or type ("[[OpenShift Concepts#NonTerminating|NonTerminating]]", "[[OpenShift Concepts#Terminating|Terminating]]").


==CPU Request==
apiVersion: v1
kind: ResourceQuota
spec:
  ...
  scopes:
  - BestEffort 2


==CPU Limit==
A "BestEffort" scope restricts quota to limiting the number of pods ("pods").


=Memory Usage=
A "Terminating", "NotTerminating" and "NotBestEffort" scope restricts a quota to tracking the following resources: "pods", "requests.memory"/"memory", "limits.memory" "requests.cpu"/"cpu", "limits.cpu".


==Memory Request==
==Quota Enforcement==


==Memory Limit==
After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. Once the usage statistics are updated, content can be created or modified, but only if by doing so quota is not exceeded. If the quota is exceeded, the action will be denied and an error message will be returned. The quota will be modified immediately after creation/update. When a resource is deleted, the quota is decremented during the next full recalculation of quota statistics per project.


=Opaque Integer Resources=
==Quota Operations==


=Resource Consumption Enforcement=
All quotas for a project can be obtained with [[OpenShift_Resource_Management_Operations#Get_Quotas_in_a_Project|oc get quota]], and information about individual quotas can be obtained with [[OpenShift_Resource_Management_Operations#Information_about_an_Individual_Quota|oc describe quota]]. Quotas can also be viewed from the web console, in the project's "Quota" page. Quotas can be created with [[OpenShift_Resource_Management_Operations#Creating_a_Resource_Quota|oc create]].


=Organizatorium=
=Limit Range=


<tt>ResourceQuota</tt> object enumerates hard resource usage limits ''per project''.
{{External|https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-limit-ranges}}


<tt>ClusterResourceQuota</tt> object enumerates hard resource usage limits across the cluster.
A limit range enumerates compute resource constraints in a project a the pod/container/image/image stream/persistent volume claim level, and specifies the amount of resources that a pod/container/image/image stream/persistent volume claim can consume. Limit ranges are defined by the <tt>LimitRange</tt> object. All resource creation and modification requests are evaluated against each limit range in the project and rejected if the resource request is outside limits. If the resource does not set an explicit value, and if the constraint supports a default value, the default value is applied to the resource. Limit ranges are set by the cluster administrators and are project-scoped.


After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. If the resource creation request exceeds the quota, the server will deny the action and will return an error message.
The limits are accessed with:


When a resource is created, the quota usage is updated immediately. When a resource is delete, the quota usage is updated during the next full per-project statistics update.
oc get limits [-n ''project-name'']


{{Internal|Resource Quota Definition|Resource Quota Definition}}
=Opaque Integer Resources=
 
Operations:
* [[Oc_get#quota|oc get quota]]
* [[Oc_describe#quota|oc describe quota]]
* [[Oc_create#Creating_a_Resource_Quota|oc create]]


==Enforcement==
{{External|https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#opaque-integer-resources-dev}}


<font color=red>With cgroups?</font>
=Compute Resources for System Daemons=


===Limits===
{{External|https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/}}


====Memory Limit====
=Handling Out of Resource Events=


Propagates as [[Docker_Container_Downward_API#memory.limit_in_bytes|/sys/fs/cgroup/memory/memory.limit_in_bytes]] in container.
{{External|https://access.redhat.com/documentation/en-us/openshift_container_platform/3.7/html/cluster_administration/admin-guide-handling-out-of-resource-errors}}


===Requests===
=Access to Resource Specification from Pods=


====Memory Request====
...
        - env:
          - name: FLUENTD_CPU_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: fluentd-elasticsearch
                divisor: "0"
                resource: limits.cpu
          - name: FLUENTD_MEMORY_LIMIT
            valueFrom:
              resourceFieldRef:
                containerName: fluentd-elasticsearch
                divisor: "0"
                resource: limits.memory
...

Latest revision as of 18:29, 25 August 2020

External

Internal

Overview

OpenShift provides API-level support for establishing and enforcing resource quotas. The runtime monitors resource usage and intervenes when the quotas are reached or exceeded. Resource quotas can be set up and managed on the following type of resources: the quantity of objects that can be created per project, the amount of compute resources (requests and limits on CPU and memory) consumed by the project and individually by project entities such as pods and containers, the amount of storage consumed by project entities. Opaque integer resources can also be set and monitored.

Resource monitoring and consumption is important because it insures that no projects are using more resources that is appropriate for the cluster size. Primarily, resource constraints are set by the cluster administrators, but developers can also set request and limits on compute resources.

Quota

A resource quota specifies constraints that limit aggregate resource consumption per project, and they are set by cluster administrators. The resource quota for a project is defined by a ResourceQuota object. Resource quotas per cluster can be managed with ClusterResourceQuota. The resource quota limits:

  • the quantity of objects, per type, that can be created in a project:
    • Pods ("pods") - the total number of pods in a non-terminal state.
    • ConfigMaps ("configmaps")
    • Replication Controllers ("replicationcontrollers")
    • Secrets ("secrets")
    • Services ("services")
    • Image Streams ("openshift.io/imagestreams")
    • Resource Quotas ("resourcequotas")
  • the total amount of compute resources consumed by the project. Note that if a quota has a value specified for "requests.cpu" or "requests.memory", then it requires that every incoming container make an explicit request for those resources. The same rule applies for "limits.cpu" and "limits.memory":
  • the total amount of storage consumed by the project:
    • Persistent Volume Claims ("persistentvolumeclaims")
    • "requests.storage" - across all persistent volume claims in the project, the sum of storage requests cannot exceed this value.
    • "gold.storageclass.storage.k8s.io/persistentvolumeclaims", "gold.storageclass.storage.k8s.io/requests.storage"
    • "silver.storageclass.storage.k8s.io/persistentvolumeclaims", "silver.storageclass.storage.k8s.io/requests.storage"
    • "bronze.storageclass.storage.k8s.io/persistentvolumeclaims", "bronze.storageclass.storage.k8s.io/requests.storage"

Quota Scope

The compute resource quotas can be restricted only to pods in a non-terminal state within a certain scope. Each quota can have an associated set of scopes, and the quota will only measure usage for a resource if it matches the intersection of enumerated scopes. The scope can designate the pod's quality of service ("BestEffort", "NotBestEffort"), or type ("NonTerminating", "Terminating").

apiVersion: v1
kind: ResourceQuota
spec:
  ...
  scopes:
  - BestEffort 2

A "BestEffort" scope restricts quota to limiting the number of pods ("pods").

A "Terminating", "NotTerminating" and "NotBestEffort" scope restricts a quota to tracking the following resources: "pods", "requests.memory"/"memory", "limits.memory" "requests.cpu"/"cpu", "limits.cpu".

Quota Enforcement

After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. Once the usage statistics are updated, content can be created or modified, but only if by doing so quota is not exceeded. If the quota is exceeded, the action will be denied and an error message will be returned. The quota will be modified immediately after creation/update. When a resource is deleted, the quota is decremented during the next full recalculation of quota statistics per project.

Quota Operations

All quotas for a project can be obtained with oc get quota, and information about individual quotas can be obtained with oc describe quota. Quotas can also be viewed from the web console, in the project's "Quota" page. Quotas can be created with oc create.

Limit Range

https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-limit-ranges

A limit range enumerates compute resource constraints in a project a the pod/container/image/image stream/persistent volume claim level, and specifies the amount of resources that a pod/container/image/image stream/persistent volume claim can consume. Limit ranges are defined by the LimitRange object. All resource creation and modification requests are evaluated against each limit range in the project and rejected if the resource request is outside limits. If the resource does not set an explicit value, and if the constraint supports a default value, the default value is applied to the resource. Limit ranges are set by the cluster administrators and are project-scoped.

The limits are accessed with:

oc get limits [-n project-name]

Opaque Integer Resources

https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#opaque-integer-resources-dev

Compute Resources for System Daemons

https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/

Handling Out of Resource Events

https://access.redhat.com/documentation/en-us/openshift_container_platform/3.7/html/cluster_administration/admin-guide-handling-out-of-resource-errors

Access to Resource Specification from Pods

...
       - env:
         - name: FLUENTD_CPU_LIMIT
           valueFrom:
             resourceFieldRef:
               containerName: fluentd-elasticsearch
               divisor: "0"
               resource: limits.cpu
         - name: FLUENTD_MEMORY_LIMIT
           valueFrom:
             resourceFieldRef:
               containerName: fluentd-elasticsearch
               divisor: "0"
               resource: limits.memory
...