OpenShift Concepts TODEPLETE: Difference between revisions
| Line 809: | Line 809: | ||
{{External|https://docs.openshift.org/latest/dev_guide/templates.html}} | {{External|https://docs.openshift.org/latest/dev_guide/templates.html}} | ||
A template describes a set of objects that can be parameterized and processed, so they can be created at once. The template can be processed to create anything within a project, provided that permissions exist. The template may define a set of labels to apply to every object defined in the template. A template can use preset variables or randomized values, like passwords. | A template is a resource that describes a set of objects that can be parameterized and processed, so they can be created at once. The template can be processed to create anything within a project, provided that permissions exist. The template may define a set of labels to apply to every object defined in the template. A template can use preset variables or randomized values, like passwords. | ||
Templates can be stored in, and processed from files, or they can be exposed via the OpenShift API and stored as part of the project. Users can define their own templates within their projects. | Templates can be stored in, and processed from files, or they can be exposed via the OpenShift API and stored as part of the project. Users can define their own templates within their projects. | ||
Revision as of 07:16, 21 November 2017
External
Internal
Overview
The OpenShift is a Platform-as-a-Service (PaaS) based on industry standards. It provides development environments on demand. It is a polyglot offering, including a range of languages, frameworks, runtimes and databases. It automates management of the entire application life cycle: build, deploy and retire. It enables collaboration between developers on projects.
OpenShift is PaaS by RedHat: an enterprise-grade, secure, multi-language, auto-scaling, self-service, elastic could application platform, automated with CI/CD and built on Red Hat technologies.
OpenShift is supported anywhere RHEL is: bare metal, virtualized infrastructure (Red Hat Virtualization, vSphere, Hyper-V), OpenStack platform, public cloud providers (Amazon, Google, Azure). It runs on RHEL and Red Hat Atomic.
OpenShift extends Kubernetes to provide a more feature rich development lifecycle platform.
OpenShift Workflow
Users or automation make calls to the REST API via the command line, web console or programmatically, to change the state of the system. The master analyses the state changes and acts to bring the state of the system in sync with the desired state. The state is maintained in the store layer. Most OpenShift commands and API calls do not require the corresponding actions to be performed immediately. They usually create or modify a resource description in etcd. etcd then notifies Kubernetes or the OpenShift controllers, which notify the resource about the change. Eventually, the system state reflects the change.
OpenShift functionality is provided by the interaction of several layers:
- The store layer holds the state of the OpenShift environment, which includes configuration information, current state and the desired state. It is implemented by etcd.
- The authentication layer provides a framework for collaboration and quota management.
- Scheduling is OpenShift master's main function and it is implemented in the scheduling layer, which contains the scheduler.
- The service layer handles internal requests. It provides an abstraction of service endpoints and provides internal load balancing. The pods get IP addresses associated with the service endpoints. Note that the external requests are not handled by the service layer, but by the routing layer.
- The routing layer handles external requests, to and from the applications. For a description of the relationship between a service and a router, see relationship between a service and router.
- Application requests are routed to the business logic serving them according to the networking workflow, by the networking layer.
- The replication layer contains the replication controller, whose role is to ensure that the number of instances and pods defined in the store layer actually exists.
Component Types
OpenShift Hosts
Master
A master is a RHEL or Red Hat Atomic host that orchestrates and schedules resources.
The master maintains the state of the OpenShift environment.
The master provides the single API all tooling clients must interact with. All OpenShift tools (CLI, web console, IDE plugins, etc. speak directly with the master).
The access is protected via fine-grained role-based access control (RBAC).
The master monitors application health via user-defined pod probes, insuring that all containers that should be running are running. It handles restarting pods that failed probes automatically. Pods that fail too often are marked as "failing" and are temporarily not restarted. The OpenShift service layer sends traffic only to healthy pods.
Masters use the etcd cluster to store state, and also cache some of the metadata in their own memory space.
Master HA
Multiple masters can be present to insure HA. A typical HA configuration involves three masters and three etcd nodes. Such a topology is built by the OpenShift 3.5 ansible inventory file shown here.
An alternative HA configuration consists in a single master node and multiple (at least three) etcd nodes.
The native HA Method
Node
A node is a Linux container host. It is based on RHEL or Red Hat Atomic and provides a runtime environment where applications run inside containers, which run inside pods assigned by the master. Nodes are orchestrated by masters and are usually managed by administrators and not by end users. Nodes can be organized into different topologies. From a networking perspective, nodes gets allocated their own subnets of the cluster network, which then they use to distribute to the pods, and containers, running on them. More details about OpenShift networking is available here.
A node daemon runs on node each node.
Node operations:
TODO:
- What is the difference between the kubelet and the node daemon?
- kube proxy daemons.
Infrastructure Node
Also referred to as "infra node".
This is where infrastructure pods run. Metrics, logging, routers are considered infrastructure pods.
The infrastructure nodes, especially those running the the metrics pods and ... should be closely monitored to detect early CPU, memory and disk capacity shortages on the host system.
OpenShift Cluster
All nodes that share the SDN.
Container
A container is a kernel-provided mechanism to run one or more processes, in a portable manner, in a Linux environment. Containers are isolated from each other on a host and are initialized from images. All application instances - based on various languages/frameworks/runtimes - as well as databases, run inside containers on nodes. For more details, see
A pod can have application containers and init containers.
In OpenShift, containers are never restarted. Instead, new containers are spun up to replace old containers when needed. Because of this behavior, persistent storage volumes mounted on containers are critical for maintaining state such as for configuration files and data files.
Init Container
Docker Support in OpenShift
All containers that are running in pods are managed by Docker servers.
Docker Storage in OpenShift
Each Docker server requires Docker storage, which is allocated on each node in a space specially provisioned as Docker storage backend. The Docker storage is ephemeral and separate from OpenShift storage. The default loopback storage back end for Docker is a thin pool on loopback devices, which is not appropriate for production.
The node Daemon
Pod
A pod is a set of one or more containers, deployed together onto a node, as a single unit.
The defining characteristic of a pod is that all its containers share a virtual network device - an unique IP -, and a set of persistent volumes. Pods also define the security and runtime policy for each container.
The "pod" is a Kubernetes concept, for more details, see Kubernetes Pods. Each pod gets a pod IP address that is routable by default from any other pod in the environment. The default addresses are part of the 10.x.x.x set. The containers on a pod share the IP address and TCP ports, because they share the pod's virtual network device. They also share persistent storage volumes, and other resources allocated to the pod. The pod contains collocated applications that are relatively tightly coupled and run with a shared context. Within that context, an application may have individual cgroups isolation applied. A pod models an application-specific logical host, containing applications that in a pre-container world would have run on the same physical or virtual host, and in consequence, the pod cannot span hosts. The pod is the smallest unit that can be defined, deployed and managed by OpenShift. Complex applications can be made of any number of pods, and OpenShift helps with pod orchestration. Pods do not maintain state, they are expendable.
Pods must not created or managed directly, but by their controllers, which are specified in the pod description.
OpenShift treats pods as largely immutable - changes cannot be made to a pod definition while the pod is running - and expendable, they do not maintain state when they are destroyed and recreated. Therefore, they are managed by controllers, not directly by users.
The pods for a project are displayed by the following commands:
oc get all oc get pods
Pods for a project can also be viewed in the web console to the project -> Applications -> Pods.
Controller
A controller is the OpenShift component that creates and manages pods. The controller of a pod is reported by oc describe pod command, under the "Controllers" section:
... Controllers: ReplicationController/logging-kibana-1 ...
The most common controllers are:
Pod Configuration
Pods are treated as static, and cannot be changed while they are running. To change a pod, the current pod must be terminated, and a new one with a modified base image and/or configuration must be created.
Pod Lifecycle
- A pod is defined in a pod definition.
- A pod is instantiated and assigned to run on a node as a result of the scheduling process.
- The pod runs until its containers exit or the pod is removed.
- Depending on policy and exit code, may be removed or retained to enable access to their container's logs.
Pod Definition
The definition of an already existing pod can be obtained with oc describe pod.
Pod Name
Pod must have an unique name in their namespace (project). The pod definition can specify a base name and use "generateName" attribute to append random characters at the end of the base name, thus generating an unique name.
Pod Definition File
Pod Placement
Pods can be configured to execute on a specific node, defined by the node name, or on nodes that match a specific node selector.
To assign a pod to a specific node, TODO https://docs.openshift.com/container-platform/3.5/admin_guide/scheduler.html#constraining-pod-placement-labels
To assign a pod to nodes that match a node selector, add the "nodeSelector" element in the pod configuration, with a value consisting in key/value pairs, as described here:
After a successful placement, either by a replication controller or by a DaemonSet, the pod records the successful node selector expression as part of its definition, which can be rendered with oc get pod -o yaml:
spec: ... nodeSelector: logging: "true" ...
Consolidate with OpenShift_Concepts#Node_Selector
Pod Probe
Users can configure pod probes for liveness or readiness.
Label
Labels are simple key/value pairs that can be assigned to any resource in the system and are used to group and select arbitrarily related objects. Most Kubernetes objects can include labels in their metadata. Labels provide the default way of manage objects as groups, instead of having to handle each object individually. Labels are a Kubernetes concept.
Labels associated with a node can be obtained with oc describe node.
Selector
A selector is a set of labels. It is also referred to as label selector. Selectors are a Kubernetes concept.
Node Selector
A node selector is an expression applied to a node, which, depending on whether the node has or does not have the labels expected by the node selector, allows or prevents pod placement on the node in question during the pod scheduling operation. It is the scheduler that evaluates the node selector expression and decides on which node to place the pod. The DaemonSets also use node selectors when placing the associated pods on nodes.
Node selectors can be associated with an entire cluster, with a project, or with a specific pod. The node selectors can be modified as part of a node selector operation.
Cluster-Wide Default Node Selector
The cluster-wide default node selector is configured during OpenShift cluster installation to restrict pod placement on specific nodes. It is specified in the projectConfig.defaultNodeSelector section of the master configuration file master-config.yml. It can also be modified after installation with the following procedure:
Per-Project Node Selector
The per-project node selector is used by the scheduler to schedule pods associated with the project. The per-project node selector and takes precedence over cluster-wide default node selector, when both exist. It is available as "openshift.io/node-selector" project metadata (see below). If "openshift.io/node-selector" is set to an empty string, the project will not have an adminstrator-set node selector, even if the cluster-wide default has been set. This means that a cluster administrator can set a default to restrict developer projects to a subset of nodes and still enable infrastructure or other projects to schedule the entire cluster.
The per-project node selector value can be queried with:
oc get project -o yaml
It is listed as:
...
kind: Project
metadata:
annotations:
...
openshift.io/node-selector: ""
...
The per-project node selector is usually set up when the project is created, as described in this procedure:
It can also be changed after project creation with oc edit as described in this procedure:
Per-Pod Node Selector
The declaration of a per-pod node selector can be obtained running:
oc get pod <pod-name> -o yaml
and it is rendered in the "spec:" section of the pod definition:
...
kind: Pod
spec:
...
nodeSelector:
key-A: value-A
How is the node selector of a pod generated?
TODO: merge with OpenShift_Concepts#Pod_Placement
Once the pod has been created, the node selector value becomes immutable and an attempt to change it will fail. For more details on pod state see Pods.
Precedence Rules when Multiple Node Selectors Apply
TODO
External Resources
Scheduler
Scheduling is the master's main function: when a pod is created, the master determines on what node(s) to execute the pod. This is called scheduling. The layer that handles this responsibility is called the scheduling layer.
The scheduler is a component that runs on master and determines the best fit for running pods across the environment. The scheduler also spreads pod replicas across nodes, for application HA. The scheduler reads data from the pod definition and tries to find a node that is a good fit based on configured policies. The scheduler does not modify the pod, it creates a binding that ties the pod to the selected node, via the master API.
The OpenShift scheduler is based on Kubernetes scheduler.
Most OpenShift pods are scheduled by the scheduler, unless they are managed by a DaemonSet. In this case, the DaemonSet selects the node to run the pod, and the scheduler ignores the pod.
The scheduler is completely independent and exists as a standalone, pluggable solution. The scheduler is deployed as a container, referred to as an infrastructure container. The functionality of the scheduler can be extended in two ways:
- Via enhancements, by adding predicates to the priority functions.
- Via replacement with a different implementation.
The pod placement process is described here:
Default Scheduler Implementation
The default scheduler is a scheduling engine that selects the node to host the pod in three steps:
- Filter all available nodes by running through a list of filter functions called predicates, discarding the nodes that do not meet the criteria.
- Prioritize the remaining nodes by passing through a series of priority functions that assigns each node a score between 0 - 10. 10 signifies the best possible fit to run the pod. By default all priority function are considered equivalent, but they can be weighted differently via configuration.
- Sorts the node by score and selects the node with the highest score. If multiple nodes come with the same score, one is chosen at random.
Note that an insight into how the predicates are evaluated and what scheduling decisions are taken can be achieved by increasing the logging verbosity of the master controllers processes.
Predicates
Static Predicates
- PodFitsPorts - a node is fit fi there are no port conflicts.
- PodFitsResources - a node is fit based on resource availability. Nodes declare resource capacities, pods specify what resources they require.
- NoDiskConflict - evaluates if a pod fits based on volumes requested and those already mounted.
- MatchNodeSelector - a node is fit based on the node selector query.
- HostName - a node is fit based on the presence of host parameter and string match with host name.
Configurable Predicates
- ServiceAffinity - filters out nodes that do not belong to the topological level defined by the provided labels.
- LabelsPresence - checks whether the node has certain labels defined, regardless of value.
Priority Functions
Existing Priority Functions
- LeastRequestedPriority - favors nodes with fewer requested resources, calculates percentage of memory and CPU requested by pods scheduled on node, and prioritizes nodes with highest available capacity.
- BalancedResourceAllocation - favors nodes with balanced resource usage rate, calculates difference between consumed CPU and memory as fraction of capacity and prioritizes nodes with the smallest difference. It should always be used with LeastRequestedPriority.
- ServiceSpreadingPriority - spreads pods by minimizing the number of pods that belong to the same service, onto the same node
- EqualPriority
Configurable Priority Functions
- ServiceAntiAffinity
- LabelsPreference
Scheduler Policy
The selection of the predicates and the priority functions defines the scheduler policy. The default policy is configured in the master configuration file master-config.yml as kubernetesMasterConfig.schedulerConfigFile. By default, it points to /etc/origin/master/scheduler.json. The current scheduling policy in force can be obtained with ?. A custom policy can replace it, if necessary, by following this procedure: Modify the Scheduler Policy.
Scheduler Policy File
The default scheduler policy is configured in the master configuration file master-config.yml as kubernetesMasterConfig.schedulerConfigFile, which by default points to /etc/origin/master/scheduler.json. Another example of scheduler policy file:
Storage
Volume
Volumes are mounted filesystems available to pods and their containers. Volumes may be backed by a number of host-local or network attached storage endpoints. The simplest volume type is EmptyDir, which is a temporary directory on a single machine. Administrators may also allow to request and attach Persistent Volumes.
Persistent Volume
The persistent volumes and persistent volume claims can be listed with oc get pv and oc get pvc.
Persistent Volume Definition File
Persistent Volume Claim
A persistent volume claim is a request for a persistence resource with specific attributes, such as storage size. Persistent volume claims are matched to available volumes and binds them together. This process allows a claim to be used as a volume in a pod: OpenShift finds the volume backing the claim and mounts it into the pod. Persistent volume claims are project-specific objects. All persistent volume claims for the current project can be listed with:
oc get pvc
Persistent Volume Claim Definition File
Temporary Pod Volume
When temporary pod volumes are used, the data is written to /var/lib/origin/openshift.local.volumes/pods on the node where the pod is running.
etcd
Implements OpenShift's store layer, which holds the current state, desired state and configuration information of the environment.
OpenShift stores image, build, and deployment metadata in etcd. Old resources must be periodically pruned. If a large number of images/build/deployments are planned, etcd must be placed on machines with large amounts of memory and fast SSD drives.
etcd and Master Caching
Masters cache deserialized resource metadata to reduce the CPU load and keep the metadata in memory. For small clusters, the cache can use a lot of memory for negligible CPU reduction. The default cache size is 50,000 entries, which, depending on the size of resources, can grow to occupy 1 to 2 GB of memory. The cache size can be configured in master-config.yml.
Image Registries
Integrated Docker Registry
OpenShift contains an integrated Docker image registry, providing authentication and access control to images. The integrated registry should not be confused with the OpenShift Container Registry (OCR), which is a standalone solution, which can be used outside of an OpenShift environment. The integrated registry is installed as part of the cluster installation process. The integrated registry is the default location where users can push images they build. Users push images into registry and whenever a new image is stored in the registry, the registry notifies OpenShift about it and passes along image information such as the namespace, the name and the image metadata. Various OpenShift components reacts by creating new builds and deployments. Third party image registry providers (http://dockerhub.com, or RedHat's own http://registry.access.redhat.com) can also be integrated. In general, any server implementing Docker registry API can be integrated.
The integrated registry is an application deployed within a project; it is deployed as a privileged container (referred to as an infrastructure container). The default integrated registry is deployed as part of the "default" project. It consists of a a "docker-registry" service, a "docker-registry" deployment configuration, a "registry" service account and a role binding that associates the service account to the "system:registry" cluster role. This is the registry's definition generated with oadm registry -o yaml.
Questions':
- How can I "instruct" an application to use a specific registry. a “lab” application to use registry-lab
- How do pods that need images go to that registry and not other? The answer to this question lays in the way an application uses the registry.
The default registry comes with a registry console.
The Registry Console
The registry comes with a "registry console" that allows web-based access to the registry. An URL example is https://registry-console-default.apps.openshift.novaordis.io. The registry console is deployed as a registry-console pod part of the "default" project.
More details:
Stand-alone Registry
OpenShift comes with an integrated container registry. However, a stand-alone container registry can be installed and used with the OpenShift cluster.
Registry Operations
Service
A service represents a group of pods, which may individually come and go, and its primary function is to provide the permanent IP, hostname and port for other applications to use. Each service has a service IP address and a port, which is allocated from the services subnet. The service IP address is sometimes referred to as the "cluster IP", which is consistent with the abstraction provided by the service: a cluster of pods providing the service. The services constitute the service layer.
The service IP is available internally in the cluster. It is not routable (in the IP sense) externally. The service can be exposed externally via a route.
A service resource is an abstraction that defines a logical set of pods and a policy that is used to access the pods. The service layer is how applications communicate with one another. The service is a Kubernetes concept. The service serves as an internal load balancer: it identifies a set of replicated pods and then proxies the connections it receives to those pods (routers provide external load balancing). The service is not a "thing", but an entry in the configuration. Backing pods can be added or removed to or from the service arbitrarily. This way, anything that depends on the service can refer to it as a consistent IP:port pair. The services uses a label selector to find all the running containers associated with it.
Services can be consumed from other pods using the values of the <SERVICE>_HOST environment variables that are injected by the cluster.
Services of a project can be displayed with:
oc get all oc get svc
Service Endpoint
An <pod IP address>:<port> pair that corresponds to a process running inside of a container, on a pod, on a node. The service endpoint coordinate can be obtained by executing:
oc describe service <service-name>
Name: logging-kibana ... Endpoints: 10.129.2.17:3000 ...
Relationship between Services and the Pods providing the Service
API
Networking
Required Network Ports
Default Network Interface on a Host
Every time OpenShift refers to the "default interface" on a host, it means the network interface associated with the default route. This is the logic that figures it out is here:
SDN, Overlay Network
All hosts in the OpenShift environment are clustered and are also members of the overlay network based on a Software Defined Network (SDN). Each pod gets its own IP address, by default from the 10.128.0.0/14 subnet, that is routable from any member of the SDN network. Giving each pod its own IP address means that pods can be treated like physical hosts or virtual machines in terms of port allocation, networking, naming, service discovery, load balancing, application configuration and migration.
However, it is not recommended that a pod talks to another directly by using the IP address. Instead, they should use services as an indirection layer, and interact with the service, that may be deployed on different pods at different times. Each service also gets its own own IP address from the 172.30.0.0/16 subnet.
The Cluster Network
The cluster network is the network from which pod IPs are assigned. These network blocks should be private and must not conflict with existing network blocks in your infrastructure that pods, nodes or the master may require access to. The default subnet value is 10.128.0.0/14 (10.128.0.0 - 10.131.255.255) and it cannot be arbitrarily reconfigured after deployment. The size and address range of the cluster network, as well as the host subnet size are configurable during installation. Configured with 'osm_cluster_network_cid' at installation.
The master maintains a registry of nodes in etcd, and each node gets allocated upon creation an unused subnet from the cluster network. Each node gets a /23 subnet, which means the cluster can allocate 512 subnets to nodes, and each node has 510 addresses to assign to containers running on it. Once the node is registered with the master, and gets its cluster network subnet, SDN creates on the node three devices:
- br0 - the OVS bridge device that pod containers will be attached to. The bridge is configured with a set of non-subnet specific flow rules.
- tun0 - an OVS internal port (port 2 on br0). This gets assigned the cluster subnet gateway address, and it is used for external access. The SDN configures net filter and routing rules to enable access from the cluster subnet to the external network via NAT.
- vxlan_sys_4789. This is the OVS VXLAN device (port 1 on br0) which provides access to containers on remote nodes. It is referred to as "vxlan0".
Each time a pod is started, the SDN assigns the pod a free IP address from the node's cluster subnet, attaches the host side of the pod's veth interface pair to the OVS bridge br0, adds OpenFlow rules to the OVS database to route traffic addressed to the new pod to the correct OVS port. If a ovs-multitenant plug-in is active, it also adds OpenFlow rules to tag traffic coming from the pod with the pod's VIND, and to allow traffic into the pod if the traffic's VIND matches the pod's VIND, or it is the privileged VIND 0.
Nodes update their OpenFlow rules in response to master's notification in case new nodes are added or leave. When a new subnet is added, the node adds rules on br0 so that packets with a destination IP address the remote subnet go to vxlan0 (port 1 on br0) and thus out onto the network. The ovs-subnet plug-in sends all packets across the VXLAN with VNID 0, but the ovs-multitenant plug-in uses the appropriate VNID for the source container.
The SDN does not allow the master host (which is running the OVS processes) to access the cluster network, so a master does not have access to pods via the cluster network, unless it is also running a node.
When ovs-multitenant plug-in is active, the master also allocates VXLAN VNIDs to projects. VNIDs are used to isolate the traffic.
Packet Flow
Pod IP Address
A cluster network IP address, that gets assigned to a pod.
The Services Subnet
OpenShift uses a "services subnet", also known as "kubernetes services network", in which OpenShift Services will be created within the SDN. This network block should be private and must not conflict with any existing network blocks in the infrastructure to which pods, nodes, or the master may require access to. It defaults to 172.30.0.0/16. It cannot be re-configured after deployment. If changed from the default, it must not be 172.16.0.0/16, which the docker0 network bridge uses by default, unless the docker0 network is also modified. It is configured with 'openshift_master_portal_net', 'openshift_portal_net' at installation and populates the master configuration file servicesSubnet element. Note that Docker expects its insecure registry to available on this subnet.
The services subnet IP address of a specific service can be displayed with:
oc describe service <service-name>
or
oc get svc <service-name>
The Service IP Address
An IP address from the services subnet that is associated with a service. The service IP address is reported as "Cluster IP" by:
oc get svc <service-name>
This makes sense if we think about a service as a cluster of pods that provide the service.
Example:
oc get svc logging-es NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es 172.30.254.155 <none> 9200/TCP 31d
Docker Bridge Subnet
Default 172.17.0.0/16.
Network Plugin
Open vSwitch
DNS
External DNS Server
An external DNS server is required to resolve the public wildcard name of the environment - and as consequence, all public names of various application access points - to the public address of the default router, as per networking workflow. If more than one router is deployed, the external DNS server should resolve the public wildcard name of the environment to the public IP address of the load balancer. For more details about initial configuration details see External DNS Setup.
Optional Support DNS Server
An optional support DNS server can be setup to translate local hostnames (such as master1.ocp36.local) to internal IP addresses allocated to the hosts running OpenShift infrastructure. There is no good reason to make these addresses and names publicly accessible. The OpenShift advanced installation procedure factors it in automatically, as long as the base host images OpenShift is installed in top of is already configured to use it to resolve DNS names via /etc/resolv.conf or NetworkManager. No additional configuration via openshift_dns_ip is necessary.
Internal DNS Server
A DNS server used to resolve local resources. The most common use is to translate service names to service addresses. The internal DNS server listens on the service IP address 172.30.0.1:53. It is a SkyDNS instance built into OpenShift. It is deployed on the master and answers the queries for services. The naming queries issued from inside containers/pods are directed to the internal DNS service by Dnsmasq instances running on each node. The process is described below.
The internal DNS server will answer queries on the ".cluster.local" domain that have the following form:
- <namespace>.cluster.local
- <service>.<namespace>.svc.cluster.local - service queries
- <name>.<namespace>.endpoints.cluster.local - endpoint queries
Service DNS can still be used and responds with multiple A records, one for each pod of the service, allowing the client to round-robin between each pod.
Dnsmasq
Dnsmasq is deployed on all nodes, including masters, as part of the installation process, and it works as a DNS proxy. It binds on the default interface, which is not necessarily the interface servicing the physical OpenShift subnet, and resolves all DNS requests. All OpenShift "internal" names - service names, for example - and all unqualified names are assumed to be in the "namespace.svc.cluster.local", "svc.cluster.local", "cluster.local" and "local" domains and forwarded to the internal DNS server. This is done by Dnsmasq, as configured from /etc/dnsmasq.d/origin-dns.conf:
server=/cluster.local/172.30.0.1
This configuration tells Dnsmasq to forward all queries for names in the "cluster.local" domain and sub-domains to the internal DNS server listening on 172.30.0.1.
This is an example of such query (where 192.168.122.17 is the IP address of the local node on which Dnsmasq binds, and 172.30.254.155 is a service IP address) :
nslookup logging-es.logging.svc.cluster.local Server: 192.168.122.17 Address: 192.168.122.17#53 Name: logging-es.logging.svc.cluster.local Address: 172.30.254.155
The non-OpenShift name requests are forwarded to the real DNS server stored in /etc/dnsmasq.d/origin-upstream-dns.conf:
server=192.168.122.12
The configuration file /etc/dnsmasq.d/origin-dns.conf is deployed at installation, while /etc/dnsmasq.d/origin-upstream-dns.conf is created dynamically every time the external DNS changes, as it is the case for example when the interface receives a new DHCP IP address.
For more details, see
Container /etc/resolv.conf
The container /etc/resolv.conf is created while the container is being assembled, based on information available in node-config.yamll. Also see:
External OpenShift DNS Resources
Routing Layer
OpenShift networking provides a platform-wide routing layer that directs outside traffic to the correct pods IPs and ports. The routing layer cooperates with the service layer. Routing layer components run themselves in pods. They provide automated load balancing to pods, and routing around unhealthy pods. The routing layer is pluggable and extensible. For more details about the overall OpenShift workflow, see "OpenShift Workflow".
Router
The router service routes external requests to applications inside the OpenShift environment. The router service is deployed as one or more pods. The pods associated with the router service are deployed as infrastructure containers on infrastructure nodes. A router is usually created during the installation process. Additional routers can be created with oadm router.
The router is the ingress point for all traffic flowing to backing pods implementing OpenShift Services. Routers run in containers, which may be deployed on any node (or nodes) in an OpenShift environment, as part of the "default" project. A router works by resolving fully qualified DNS external names ("kibana.apps.openshift.novaordis.io") external requests are associated with into pod IP addresses and proxying the requests directly into the pods that back the corresponding service. The router gets pod IPs from the service, and proxies the requests to pods directly, not through the Service.
Routers directly attach to port 80 and 443 on all interfaces on a host, so deployment of the corresponding containers should be restricted to the infrastructure hosts where those points are available. Statistics are usually available on port 1936.
Relationship between Service and Router
When the default router is used, the service layer is bypassed. The Service is used only to find which pods the service represents. The default router does the load balancing by proxying directly to the pod endpoints.
Sticky Sessions
The router configuration determines the sticky session implementation. The default HAProxy template implements sticky session using balance source directive.
Router Implementations
HAProxy Default Router
By default, the router process is an instance of HAProxy, referred to as the "Default Router". It uses "openshift3/OpenShift Container Platform-haproxy-router" image, and it supports unsecured, edge-terminated, re-encryption terminated and passthrough terminated routes matching on HTTP host and request path.
F5 Router
The F5 Router integrates with existing F5 Big-IP systems.
Router Operations
Information about the existing routers can be obtained with:
oadm router -o yaml
More details on router operations are available here:
Route
Logically, a route is an external DNS entry, either in a top level domain or a dynamically allocated name, that is created to point to an OpenShift service so that the service can be accessed outside the cluster. The administrator may configure one or more routers to handle those routes, typically through an Apache or HAProxy load balancer/proxy.
The route object describes a service to expose by giving it an externally reachable DNS host name.
A route is a mapping of an FQDN and path to the endpoints of a service. Each route consist of a route name, a service selector and an optional security configuration:
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD logging-kibana kibana.apps.openshift.novaordis.io logging-kibana <all> reencrypt/Redirect None
A route can be unsecured or secured.
The secure routes specify the TLS termination of the route, which relies on SNI (Server Name Indication). More specifically, they can be:
- edge secured (or edge terminated). Occurs at the router, prior to proxying the traffic to the destination. The front end of the router serves the TLS certificates, so they must be configured in the route. If the certificates are not configured in the route, the default certificate of the router is used. Connection from the router into the internal network are not encrypted. This is an example of edge-termintated route.
- passthrough secured (or passthrough terminated). The encrypted traffic is send to to destination, the router does not provide TLS termination, hence no keys or certificates are required on the router. The destination will handle certificates. This method supports client certificates. This is an example of passthrough-termintated route.
- re-encryption terminated secured. This is an example of re-encyrption-termintated route.
Routes can be displayed with the following commands:
oc get all oc get routes
Routes can be created with API calls, an JSON or YAML object definition file or the oc expose service command.
Path-Based Route
A path-based route specifies a path component, allowing the same host to server multiple routes.
Route with Hostname
Routes let you associate a service with an externally reachable hostname. If the hostname is not provided, OpenShift will generate on based on the <routename>-.<namespace>].<suffix> pattern.
Default Routing Subdomain
The suffix and the default routing subdomain can be configured in the master configuration file.
Route Definition File
Route Operations
Networking Workflow
The networking workflow is implemented at the networking layer.
- A user requests a page by pointing the browser to http://myapp.mydomain.com
- The external DNS server resolves that request to the IP address of one of the hosts that host the default router. That usually requires a wildcard C name in the DNS server pointed to the node that hosts the router container.
- Default Router selects the pod from the list of pods listed by the service and acts as a proxy for that pod (this is where the port can be translated from the external 443 to the internal 8443, for example).
Resources
- cpu, requests.cpu
- memory, requests.memory
- limits.cpu
- memory.cpu
- pods
- replicationcontrollers
- resourcequotas
- services
- Secrets
- configmaps
- persistentvolumeclaims
- openshif.io/imagestreams
Most resources can be defined in JSON or YAML files, or via an API call.
Resource Quota
ResourceQuota object enumerates hard resource usage limits per project.
ClusterResourceQuota object enumerates hard resource usage limits across the cluster.
After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. If the resource creation request exceeds the quota, the server will deny the action and will return an error message.
When a resource is created, the quota usage is updated immediately. When a resource is delete, the quota usage is updated during the next full per-project statistics update.
Operations:
Project
Projects allows groups of users to work together, define ownership of resources and manage resources - they can be seen as the central vehicle for managing regular users' access to resources. The project restricts and tracks use of resources with quotas and limits. A project is a Kubernetes namespace with additional annotations. A project can contain any number of containers of any kind, and any grouping or structure can be enforced using labels. The project is the closest concept to an application, OpenShift does not know of applications, though it provides a new-app command. A project lets a community of users to organize and manage their content in isolation from other communities. Users must be given access to projects by administrators—unless they are given permission to create projects, in which case they automatically have access to their own projects.
Most objects in the system are scoped by a namespace, with the exception of:
Each project has its own set of objects: pods, services, replication controllers, etc. The names of each resource are unique within a project. Developers may request projects to be create, but administrators control the resources allocated to the projects.
What actions a user can or cannot perform on a project's objects are specified by policies.
Constraints are quotas for each kind of object that can be limited.
New projects are created with:
oc new-project
Current Project
The current project is a concept that applies to oc, and specifies the project oc commands apply to, without to explicitly having to use the -n <project-name> qualifier. The current project can be set with oc project and read with oc status. The current project is part of the CLI tool's current context, maintained in user's .kube/config.
Global Project
In the context of an ovs-multitenant SDN plugin, a project is global if if can receive cluster network traffic from any pods, belonging to any project, and it can send traffic to any pods and services. Is the default project a global project?. A project can be made global with:
oadm pod-network make-projects-global <project-1-name> <project-2-name>
Standard Projects
Default Project
The "default" project is also referred to as "default namespace". It contains the following pods:
- the integrated docker registry pod (memory consumption based on a test installation: 280 MB)
- the registry console pod (memory consumption based on a test installation: 34 MB)
- the router pod (memory consumption based on a test installation: 140 MB)
In case the ovs-multitenant SDN plug-in is installed, the "default" project has VNID 0 and all its pods can send and receive traffic from any other pods.
The "default" project can be used to store a new project template, if the default one needs to be modified. See Template Operations - Modify the Template for New Projects.
"logging" Project
If logging support is deployed at installation or later, the participating pods (kibana, ElasticSearch, fluentd, curator) are members of the "logging" project.
This is the memory consumption based on a test installation:
- kibana pod: 95 MB
- elasticsearch pod: 1.4GB
- curator pod: 10 MB
- fluentd pods max 130 MB
"openshift-infra" Project
Contains the metrics components:
- the Hawkular Cassandra pod
- Hawkular pod
- Heapster pod
"openshift" Project
Contains templates.
Other Standard Projects
- "kube-system"
- "management-infra"
Project Operations
Template
A template is a resource that describes a set of objects that can be parameterized and processed, so they can be created at once. The template can be processed to create anything within a project, provided that permissions exist. The template may define a set of labels to apply to every object defined in the template. A template can use preset variables or randomized values, like passwords.
Templates can be stored in, and processed from files, or they can be exposed via the OpenShift API and stored as part of the project. Users can define their own templates within their projects.
The objects define in a template collectively define a desired state. OpenShift's responsibility is to make sure that the current state matches the desired state.
Specifying parameters in a template allows all objects instantiated by the template to see consistent values for these parameters when the template is processed. The parameters can be specified explicitly, or generated automatically.
The configuration can be generated from template with oc process command.
New Project Template
The master provisions projects based on the template that is identified by the "projectRequestTemplate" in master-config.yaml file. If nothing is specified there, new projects will be created based on a built-in new project template that can be obtained with:
oadm create-bootstrap-project-template -o yaml
Template Libraries
- Global template library - is this the "openshift" project?
- Project template library. A project template can be based on a JSON file and uploaded with oc create to the project library. A template stored in the library can be instantiate with oc new-app command.
- https://github.com/openshift/library
- https://github.com/jboss-openshift/application-templates
Template Operations
CICD Support
OpenShift enables DevOps. It has built-in support for the Jenkins CI server, providing a native way of doing CICD.
OpenShift can integrate with Git-based source code repositories. When a developer pushes code to the GIt repository, Jenkins pulls the code and performs a project build. After the build is complete, Jenkins invokes the OpenShift master node API to initiate a "source-to-image" (S2I) build. As part of the SI process, the latest project artifact is downloaded from Jenkins and included in the image that runs in the OpenShift node.
More details:
Build
A build is the process of transforming input parameters into a resulting object. In most cases, that means transforming source code, other images, Dockerfiles, etc. into a runnable image. A build is run inside of a container and has the same restrictions normal pods have. A build usually results in an image pushed to a Docker registry, subject to post-build tests.
Docker can be used to build images directly, but OpenShift supplies additional builders to assist with creating an image, by adding code or configuration to existing images. The build process is defined by its build strategy.
The build operations are executed by OpenShift in privileged containers.
Builds for a project can be reviewed by navigating with the web console to the project -> Builds -> Builds.
Build operations:
Build Configuration
A BuildConfig object is the definition of an entire build. It contains a description of how to build source code and a base image into a new image, the specification of triggers (web-hooks), which specify events for invoking builds, for example when a newer base image is available, or new deployments.
Builds can be:
- Source based and use builder images for common languages like Java, PHP, Ruby, or Python
- Docker based and create builds from a Dockerfile.
OpenShift creates Docker containers from build images and pushes the to an integrated registry.
oc new-app creates a BuildConfig object.
The BuildConfig consists of:
- Triggers (GitHub webhook, generic webhook, image changes)
- Parameters (source, strategy, output)
Build Strategy
Docker Build
The Docker build expects repository with Dockerfile and all artifacts required to produce runnable image. It invokes the docker build command, and the result is a runnable image.
Source-to-Image (S2I) Build
It is a process for building reproducible Docker images, which produces runnable images by injecting application source into a Docker image and assembling a new Docker image. The new image incorporates the base image (also known as the builder image) and the built source code and it is ready to use with the docker run command. S2I supports incremental builds, which reuse previously downloaded dependencies, previously built artifacts, etc.
A Git repository URL, which contains code written in any supported framework, and a builder image, which contains the O/S and the framework to support the code, must be specified to start a S2I build.
S2I scripts can be written to inject application code into almost any existing Docker image. The assemble process can perform a large number of complex operations without creating a new layer at each step, resulting in a fast process. S2I allows to build the application consistently if an underlying image needs a patch due to a security issue.
S2I provide a developer-centric flow that focuses on turning the developer source code into a running application as simple as possible.
Builder Image
The builder image contain both the build and the runtime environment for that source code.
Custom Build
The custom build strategy allows to define a specific builder image responsible for the entire build process, which allows you to customize the build process.
The custom builder image is a plain Docker image embedded with build process logic.
Pipeline Build
The pipeline build strategy allows developers to define a Jenkins pipeline for execution by Jenkins pipeline plugin.
Image
An image is a layered Linux filesystem that contains all dependencies a container needs to run (application code, application dependencies and any supporting operating system libraries), and the metadata describing its needs and capabilities. An image is identified by a name that can be local to the current cluster - meaning that it is maintained in the internal Docker registry - or point to a remote Docker registry.
For more details about Docker images see:
Images in OpenShift are immutable.
Image Stream
An image stream is similar to a Docker image repository, in that it contains a group of related Docker images identified by image stream tags. Logically it is analogous to a branch in a source code repository.
The image stream presents a single virtual view of related images. The stream may contain images from the OpenShift integrated Docker registry, other image streams, other image repositories.
OpenShift stores complete metadata about each image, including example command, entry point and environment variables.
Builds and deployments can watch an image stream to receive notifications when new images are added and react by performing a build or a deployment.
Operations:
Image Stream Tag
The default tag is called "latest".
A tag may point to an external Docker registry, at other tags in the same stream, or be controlled to directly point to known images.
Images can be pushed to an image stream tag directly via the integrated Docker registry.
Image Pull Policy
When the container is created, the runtime uses the "imagePullPolicy" to determine whether to pull the image prior to starting the container. More details available here:
Deployment
A deployment is a replication controller based on a user-defined template called a deployment configuration.
A deployment adds extended support for software development and deployment lifecycle. Deployments are created manually or in response to triggered events.
The deployment system provides a deployment configuration, which is a template for deployments, triggers that drive automated deployments in response to events, user-customizable strategies to transition from the previous deployment to the new deployment, rollback procedure, and manual replication scaling.Deployment configuration version increments each time new deployment is created from configuration. Deployments allow defining hooks to be run before and after the replication controller is created.
Deployments allow rollbacks.
Deployments allow manual replication scaling or autoscaling.
The deployments are triggered with oc deploy.
DeploymentConfig
A deployment configuration is a user-defined template for performing deployments, which result in running applications. The deployment configuration defines the template for a pod. It manages deploying new images or configuration changes whenever those change. A single deployment configuration is usually analogous to a single micro service.
Each deployment is represented as a replication controller. The OpenShift environment creates a replication controller to run the application in response to a deployment configuration. The deployment configuration contains a version number that is incremented each time a replication controller is created from the configuration.
Deployment configurations can support many different deployment patterns, including full restart, customizable rolling updates, and fully custom behaviors, as well as pre- and post-hooks. It supports automatic rolling back to the last successful revision of configuration, in case the current template fails to deploy.
The DeploymentConfig contains:
- Replication controller definition.
- Default replica count fo the deployment.
- Triggers for creating new deployments automatically, in response to events. If no triggers are defined, deployments must be started manually.
- Strategy for transitioning before deployments.
- Life-cycle hooks. Every hook has a failure policy (Abort, Retry, Ignore).
DeploymentConfiguation triggers:
- ConfigChange - results in a new deployment whenever changes are detected to replication controller template of deployment configuration.
- ImageChange - results in a new deployment whenever value of image stream tag changes.
The DeploymentConfig for a project can be listed with:
oc get all oc get dc
External
Replication Controller
A replication controller is one of the pod controller types available in OpenShift.
It resides on master, and insures that the specified number of pod replicas defined in the environment state are running at all times.
Logically, the replication controllers constitute the replication layer.
The definition of a replication controller includes the number of replicas to be maintained, the pod definition for creating the replicated pod, and a selector for identifying managed pods. If pods exit or are deleted, either explicitly or because the node they run on is taken out of service, the replication controller instantiates more pods up to desired number. If there are more pods running than desired, the replication controller deletes as many as necessary. However, it is NOT the replication controller's job to perform autoscaling based on load or traffic. Replication controllers do not exist as physical processes, meaning they do not run in pods, they only exist as entries in etcd, and the master executes the logic.
A replication controller is most commonly used to represent a single deployment of part of an application based on a built image.
The replication controllers of a project can be listed with:
oc get all oc get rc
Rollback
Deployments allow rollbacks to previous versions of an application: when one deployment is superseded by another, the previous replication controller is retained, with its number of replicas set to 0. When triggered - the template fails to deploy -, the rollback reverts an application to the last successful deployment. It is done with oc rollback, API or web console.
Deployment Strategy
The deployment configuration defines a deployment strategy. The deployment strategy determines the deployment process. It uses readiness checks to determine if a new host is ready for use. If the readiness check fails, the deployment configuration retries until it times out.
Rolling Deployment Strategy
The default deployment strategy, if a deployment strategy is not explicitly specified in the deployment configuration. It performs rolling updates. It supports life-cycle hooks for injecting code into the deployment process.
It consists in the following steps:
- Execute the "pre" life-cycle hook.
- Scale up new deployment by one or more pods, based on maxSurge value, waiting until all readiness checks complete.
- Scale down the old deployment by one or more pods, based on maxUnavailable value.
- Repeat scaling until the new deployment reaches desired replica count and the old deployment has scaled to zero.
- Execute any "post" life-cycle hook.
When scaling down, the strategy waits for pods to become ready, so it can decide if can further scaling would affect availability. If scaled up pods never become ready, the deployment times up and results in a deployment failure.
Recreate Deployment Strategy
It consists in the following steps:
- Execute "pre" life-cycle hook.
- Scale down previous deployment to zero.
- Scale up new deployment.
- Execute "post" life-cycle hook.
During scale up, if the replica count of the deployment is greater than one, the first deployment replica is validated for readiness before fully scaling up the deployment. If this validation fails, the deployment fails.
Custom Deployment Strategy
Allows for custom commands.
The optional "command" array overrides the CMD directive specified in the Dockerfile.
The optional "environment" variables are added to the strategy process' execution.
Deployment Operations
Environment Variables to Use for Strategy Process
- OPENSHIFT_DEPLOYMENT_NAME
- OPENSHIFT_DEPLOYMENT_NAMESPACE
Pipeline
A pipeline is a Jenkins jobs enabled by the pipeline plugin.
Used by the pipeline build.
Region
Zone
High Availability (HA)
Infrastructure HA
See Master HA.
Application HA
OpenShift insures high availability by deploying the same image in multiple containers across multiple hosts and load balancing among them. This technique also provides horizontal scalability for a service packaged into an image.
Installation
There are two installation procedures: RPM and Containerized.
An RPM installation installs all services through package management and configures services to run within the same user space,
A containerized installation installs services using container images and runs separate services in individual containers.
For practical details on installing various OpenShift version, see:
Related:
Installation is performed by Ansible, usually deployed on the environment's support server. Ansible configuration is available under /etc/ansible and the installation logic under /usr/share/ansible.
Security
Metrics
Horizontal Pod Autoscaler (HPA)
Logging
Events
OpenShift events incapsulate information about specific conditions detected in the OpenShift cluster, and allow the OpenShift management facilities to record information about those occurrences in a resource-agnostic manner. They also allow administrators and developers to consume information about system components in a unified way. A list of events can be obtained with:
oc get events [-n <project-name>]
Events for a project can be reviewed by navigating with the web console to the project -> Monitoring -> Right Side.
Events contain:
- type: Normal, Warning
- kind: Configuration, Node, Pod, DaemonSet, Container, Health, Image, Image Manager, System.
- reason
- source
- message
Event Reasons
- FailedScheduling
- OutOfDisk
- MatchNodeSelector
- SuccessfulCreate
DaemonSet
A DaemonSet is a OpenShift component that creates its associated pods and ensures they run on all (or some) nodes of a cluster. It is one of the pod controller types available in OpenShift.
If a node is added to the cluster, the DaemonSet insures that its associated pod will be scheduled on that node. When nodes are removed, the associated pods are shut down. Typical uses for DaemonSets are to run log collection agents (fluentd, logstash), node monitoring agents (collectd) or a cluster storage daemon (glusterd, ceph). Usually, a DaemonSet is needed for each type of daemon.
The DaemonSet decides whether it manages a pod or not based on the label selector expression specified in its definition:
... Selector: component=fluentd,provider=openshift ...
Note that internally, the declared pod template’s label selector must match the label selector above.
When a pod is managed by a DaemonSet, the node the pod is scheduled to run on is selected by the DaemonSet, so the scheduler ignores it. The "unschedulable" field of a node is not respected by the DaemonSet controller. Also, the DaemonSet controller can make pods even if the scheduler has not been started, and this helps with cluster bootstrap. The DaemonSet chooses the nodes to run its pods on based on the node selector specified in its definition:
... Node-Selector: logging=true ...
After a successful placement, the node selector expression is recorded in the pod's definition.
DaemonSet Operations
ConfigMaps
A ConfigMap is a component that holds key/value pairs of configuration data, and that can be consumed by pods, or can be used to store configuration for OpenShift system components such as controllers. It is a mechanism to inject containers with configuration while keeping the containers agnostic of the OpenShift platform. Aside from fine-grained information like individual properties, ConfigMaps can also store coarse-grained information such as entire configuration files or JSON blobs. The ConfigMaps can populate environment variables, set command-line arguments in a container and populate configuration files in a volume.
A ConfigMap is similar to a secret, but designed to be more convenient when working with strings that do not contain sensitive information.
Command Line Tools
Admission Control
Admission control plug-ins intercept requests to the master API, after authentication and authorization was enforced. There's a chain of plug-ins (also known as the admission chain), and if a plug-in rejects the request, the request fails. The plug-in may modify the request object, and related resources. The default list of admission control plug-ins is configured in the master-config.yaml's admissionConfig/pluginConfig section.
Customizable admission control plug-ins:
- BuildDefaults https://docs.openshift.com/container-platform/latest/install_config/build_defaults_overrides.html#ansible-setting-global-build-defaults
- ProjectRequestLimit https://docs.openshift.com/container-platform/latest/admin_guide/managing_projects.html#limit-projects-per-user
- RestrictSubjectBindings https://docs.openshift.com/container-platform/latest/admin_solutions/user_role_mgmt.html#role-binding-restriction
- Pod placement https://docs.openshift.com/container-platform/latest/admin_guide/scheduling/scheduler.html#controlling-pod-placement
- Init containers https://docs.openshift.com/container-platform/latest/architecture/core_concepts/containers_and_images.html#init-containers