OpenShift Concepts TODEPLETE

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

External

Internal

Overview

The OpenShift is a Platform-as-a-Service (PaaS) based on industry standards. It provides development environments on demand. It is a polyglot offering, including a range of languages, frameworks, runtimes and databases. It automates management of the entire application life cycle: build, deploy and retire. It enables collaboration between developers on projects.

OpenShift is PaaS by RedHat: an enterprise-grade, secure, multi-language, auto-scaling, self-service, elastic could application platform, automated with CI/CD and built on Red Hat technologies.

OpenShift is supported anywhere RHEL is: bare metal, virtualized infrastructure (Red Hat Virtualization, vSphere, Hyper-V), OpenStack platform, public cloud providers (Amazon, Google, Azure). It runs on RHEL and Red Hat Atomic.

OpenShift extends Kubernetes to provide a more feature rich development lifecycle platform.

OpenShiftArchitecture.png

OpenShift Workflow

Users or automation make calls to the REST API via the command line, web console or programmatically, to change the state of the system. The master analyses the state changes and acts to bring the state of the system in sync with the desired state. The state is maintained in the store layer. Most OpenShift commands and API calls do not require the corresponding actions to be performed immediately. They usually create or modify a resource description in etcd. etcd then notifies Kubernetes or the OpenShift controllers, which notify the resource about the change. Eventually, the system state reflects the change.

OpenShift functionality is provided by the interaction of several layers:

  • The store layer holds the state of the OpenShift environment, which includes configuration information, current state and the desired state. It is implemented by etcd.
  • The authentication layer provides a framework for collaboration and quota management.
  • Scheduling is OpenShift master's main function and it is implemented in the scheduling layer, which contains the scheduler.
  • The service layer handles internal requests. It provides an abstraction of service endpoints and provides internal load balancing. The pods get IP addresses associated with the service endpoints. Note that the external requests are not handled by the service layer, but by the routing layer.
  • The routing layer handles external requests, to and from the applications. For a description of the relationship between a service and a router, see relationship between a service and router.
  • Application requests are routed to the business logic serving them according to the networking workflow, by the networking layer.
  • The replication layer contains the replication controller, whose role is to ensure that the number of instances and pods defined in the store layer actually exists.

Component Types

OpenShift Component Types

OpenShift Hosts

Master

A master is a RHEL or Red Hat Atomic host that orchestrates and schedules resources.

The master maintains the state of the OpenShift environment.

The master provides the single API all tooling clients must interact with. All OpenShift tools (CLI, web console, IDE plugins, etc. speak directly with the master).

The access is protected via fine-grained role-based access control (RBAC).

The master monitors application health via user-defined pod probes, insuring that all containers that should be running are running. It handles restarting pods that failed probes automatically. Pods that fail too often are marked as "failing" and are temporarily not restarted. The OpenShift service layer sends traffic only to healthy pods.

Kubernetes Master

Masters use the etcd cluster to store state, and also cache some of the metadata in their own memory space.

Master HA

Multiple masters can be present to insure HA. A typical HA configuration involves three masters and three etcd nodes. Such a topology is built by the OpenShift 3.5 ansible inventory file shown here.

An alternative HA configuration consists in a single master node and multiple (at least three) etcd nodes.

The native HA Method

Master API

The master API is available internally (inside the OpenShift cluster) at https://openshift.default.svc.cluster.local

Node

A node is a Linux container host. It is based on RHEL or Red Hat Atomic and provides a runtime environment where applications run inside containers, which run inside pods assigned by the master. Nodes are orchestrated by masters and are usually managed by administrators and not by end users. Nodes can be organized into different topologies. From a networking perspective, nodes gets allocated their own subnets of the cluster network, which then they use to distribute to the pods, and containers, running on them. More details about OpenShift networking is available here.

A node daemon runs on node each node.

Node operations:

TODO:

  • What is the difference between the kubelet and the node daemon?
  • kube proxy daemons.

Infrastructure Node

Also referred to as "infra node".

This is where infrastructure pods run. Metrics, logging, routers are considered infrastructure pods.

The infrastructure nodes, especially those running the the metrics pods and ... should be closely monitored to detect early CPU, memory and disk capacity shortages on the host system.

OpenShift Cluster

All nodes that share the SDN.

Container

https://docs.openshift.com/container-platform/latest/architecture/core_concepts/containers_and_images.html#containers

A container is a kernel-provided mechanism to run one or more processes, in a portable manner, in a Linux environment. Containers are isolated from each other on a host and are initialized from images. All application instances - based on various languages/frameworks/runtimes - as well as databases, run inside containers on nodes. For more details, see

Docker Containers

A pod can have application containers and init containers.

In OpenShift, containers are never restarted. Instead, new containers are spun up to replace old containers when needed. Because of this behavior, persistent storage volumes mounted on containers are critical for maintaining state such as for configuration files and data files.

Init Container

https://docs.openshift.com/container-platform/latest/architecture/core_concepts/containers_and_images.html#init-containers

Docker Support in OpenShift

All containers that are running in pods are managed by Docker servers.

Docker Storage in OpenShift

Each Docker server requires Docker storage, which is allocated on each node in a space specially provisioned as Docker storage backend. The Docker storage is ephemeral and separate from OpenShift storage. The default loopback storage back end for Docker is a thin pool on loopback devices, which is not appropriate for production.

The node Daemon

Pod

A pod is a set of one or more containers, deployed together onto a node, as a single unit.

The defining characteristic of a pod is that all its containers share a virtual network device - an unique IP -, and a set of persistent volumes. Pods also define the security and runtime policy for each container.

The "pod" is a Kubernetes concept, for more details, see Kubernetes Pods. Each pod gets a pod IP address that is routable by default from any other pod in the environment. The default addresses are part of the 10.x.x.x set. The containers on a pod share the IP address and TCP ports, because they share the pod's virtual network device. They also share persistent storage volumes, and other resources allocated to the pod. The pod contains collocated applications that are relatively tightly coupled and run with a shared context. Within that context, an application may have individual cgroups isolation applied. A pod models an application-specific logical host, containing applications that in a pre-container world would have run on the same physical or virtual host, and in consequence, the pod cannot span hosts. The pod is the smallest unit that can be defined, deployed and managed by OpenShift. Complex applications can be made of any number of pods, and OpenShift helps with pod orchestration. Pods do not maintain state, they are expendable.

Pods must not created or managed directly, but by their controllers, which are specified in the pod description.

OpenShift treats pods as largely immutable - changes cannot be made to a pod definition while the pod is running - and expendable, they do not maintain state when they are destroyed and recreated. Therefore, they are managed by controllers, not directly by users.

Pod Definition

The pods for a project are displayed by the following commands:

oc get all
oc get pods

Pods for a project can also be viewed in the web console to the project -> Applications -> Pods.

Controller

A controller is the OpenShift component that creates and manages pods. The controller of a pod is reported by oc describe pod command, under the "Controllers" section:

...
Controllers:		ReplicationController/logging-kibana-1
...

The most common controllers are:

Pod Configuration

Pods are treated as static, and cannot be changed while they are running. To change a pod, the current pod must be terminated, and a new one with a modified base image and/or configuration must be created.

Pod Lifecycle

  • A pod is defined in a pod definition.
  • A pod is instantiated and assigned to run on a node as a result of the scheduling process.
  • The pod runs until its containers exit or the pod is removed.
  • Depending on policy and exit code, may be removed or retained to enable access to their container's logs.

Pod Definition

The definition of an already existing pod can be obtained with oc describe pod.

Pod Name

Pod must have an unique name in their namespace (project). The pod definition can specify a base name and use "generateName" attribute to append random characters at the end of the base name, thus generating an unique name.

Pod Definition File

Pod Definition File

Pod Placement

https://docs.openshift.com/container-platform/3.5/admin_guide/scheduler.html#controlling-pod-placement

Pods can be configured to execute on a specific node, defined by the node name, or on nodes that match a specific node selector.

To assign a pod to a specific node, TODO https://docs.openshift.com/container-platform/3.5/admin_guide/scheduler.html#constraining-pod-placement-labels

To assign a pod to nodes that match a node selector, add the "nodeSelector" element in the pod configuration, with a value consisting in key/value pairs, as described here:

Assigning a Pod to Nodes that Match a Node Selector

After a successful placement, either by a replication controller or by a DaemonSet, the pod records the successful node selector expression as part of its definition, which can be rendered with oc get pod -o yaml:

spec:
  ...
  nodeSelector:
   logging: "true"
  ...


Consolidate with OpenShift_Concepts#Node_Selector

Pod Probe

Users can configure pod probes for liveness or readiness. Can be configured with:

  • initialDelaySeconds
  • timeoutSeconds (default 1)

Liveness Probe

A liveness probe is deployed in a container to expose whether the container is running. Liveness probes: commands executed inside the container, tcpSocket.

Readiness Probe

A liveness probe is deployed in a container to expose whether the container is ready to service requests.

Readines probes: httpGet.

Local Manifest Pod

https://docs.openshift.com/container-platform/latest/install_config/master_node_configuration.html#node-configuration-files

Bare Pod

A pod that is not backed by a replication controller. Bare pods cannot be evacuated from nodes.

Label

Labels are simple key/value pairs that can be assigned to any resource in the system and are used to group and select arbitrarily related objects. Most Kubernetes objects can include labels in their metadata. Labels provide the default way of manage objects as groups, instead of having to handle each object individually. Labels are a Kubernetes concept.

Labels associated with a node can be obtained with oc describe node.

Selector

A selector is a set of labels. It is also referred to as label selector. Selectors are a Kubernetes concept.

Node Selector

A node selector is an expression applied to a node, which, depending on whether the node has or does not have the labels expected by the node selector, allows or prevents pod placement on the node in question during the pod scheduling operation. It is the scheduler that evaluates the node selector expression and decides on which node to place the pod. The DaemonSets also use node selectors when placing the associated pods on nodes.

Node selectors can be associated with an entire cluster, with a project, or with a specific pod. The node selectors can be modified as part of a node selector operation.

Cluster-Wide Default Node Selector

The cluster-wide default node selector is configured during OpenShift cluster installation to restrict pod placement on specific nodes. It is specified in the projectConfig.defaultNodeSelector section of the master configuration file master-config.yml. It can also be modified after installation with the following procedure:

Configure a Cluster-Wide Default Node Selector

Per-Project Node Selector

The per-project node selector is used by the scheduler to schedule pods associated with the project. The per-project node selector and takes precedence over cluster-wide default node selector, when both exist. It is available as "openshift.io/node-selector" project metadata (see below). If "openshift.io/node-selector" is set to an empty string, the project will not have an adminstrator-set node selector, even if the cluster-wide default has been set. This means that a cluster administrator can set a default to restrict developer projects to a subset of nodes and still enable infrastructure or other projects to schedule the entire cluster.

The per-project node selector value can be queried with:

oc get project -o yaml

It is listed as:

...
kind: Project
metadata:
  annotations:
    ...
    openshift.io/node-selector: ""
    ...

The per-project node selector is usually set up when the project is created, as described in this procedure:

Configuring a Per-Project Node Selector during Project Creation

It can also be changed after project creation with oc edit as described in this procedure:

Configuring a Per-Project Node Selector after Project Creation

Per-Pod Node Selector

The declaration of a per-pod node selector can be obtained running:

oc get pod <pod-name> -o yaml

and it is rendered in the "spec:" section of the pod definition:

...
kind: Pod
spec:
  ...
  nodeSelector:
    key-A: value-A


How is the node selector of a pod generated?

TODO: merge with OpenShift_Concepts#Pod_Placement

Once the pod has been created, the node selector value becomes immutable and an attempt to change it will fail. For more details on pod state see Pods.

Precedence Rules when Multiple Node Selectors Apply

TODO

External Resources

Scheduler

https://docs.openshift.com/container-platform/3.5/admin_guide/scheduler.html

Scheduling is the master's main function: when a pod is created, the master determines on what node(s) to execute the pod. This is called scheduling. The layer that handles this responsibility is called the scheduling layer.

The scheduler is a component that runs on master and determines the best fit for running pods across the environment. The scheduler also spreads pod replicas across nodes, for application HA. The scheduler reads data from the pod definition and tries to find a node that is a good fit based on configured policies. The scheduler does not modify the pod, it creates a binding that ties the pod to the selected node, via the master API.

The OpenShift scheduler is based on Kubernetes scheduler.

Most OpenShift pods are scheduled by the scheduler, unless they are managed by a DaemonSet. In this case, the DaemonSet selects the node to run the pod, and the scheduler ignores the pod.

The scheduler is completely independent and exists as a standalone, pluggable solution. The scheduler is deployed as a container, referred to as an infrastructure container. The functionality of the scheduler can be extended in two ways:

  1. Via enhancements, by adding predicates to the priority functions.
  2. Via replacement with a different implementation.

The pod placement process is described here:

Pod Placement

Default Scheduler Implementation

The default scheduler is a scheduling engine that selects the node to host the pod in three steps:

  1. Filter all available nodes by running through a list of filter functions called predicates, discarding the nodes that do not meet the criteria.
  2. Prioritize the remaining nodes by passing through a series of priority functions that assigns each node a score between 0 - 10. 10 signifies the best possible fit to run the pod. By default all priority function are considered equivalent, but they can be weighted differently via configuration.
  3. Sorts the node by score and selects the node with the highest score. If multiple nodes come with the same score, one is chosen at random.

Note that an insight into how the predicates are evaluated and what scheduling decisions are taken can be achieved by increasing the logging verbosity of the master controllers processes.

Predicates

Static Predicates

  • PodFitsPorts - a node is fit fi there are no port conflicts.
  • PodFitsResources - a node is fit based on resource availability. Nodes declare resource capacities, pods specify what resources they require.
  • NoDiskConflict - evaluates if a pod fits based on volumes requested and those already mounted.
  • MatchNodeSelector - a node is fit based on the node selector query.
  • HostName - a node is fit based on the presence of host parameter and string match with host name.

Configurable Predicates

  • ServiceAffinity - filters out nodes that do not belong to the topological level defined by the provided labels.
  • LabelsPresence - checks whether the node has certain labels defined, regardless of value.

Priority Functions

Existing Priority Functions

  • LeastRequestedPriority - favors nodes with fewer requested resources, calculates percentage of memory and CPU requested by pods scheduled on node, and prioritizes nodes with highest available capacity.
  • BalancedResourceAllocation - favors nodes with balanced resource usage rate, calculates difference between consumed CPU and memory as fraction of capacity and prioritizes nodes with the smallest difference. It should always be used with LeastRequestedPriority.
  • ServiceSpreadingPriority - spreads pods by minimizing the number of pods that belong to the same service, onto the same node
  • EqualPriority

Configurable Priority Functions

  • ServiceAntiAffinity
  • LabelsPreference

Scheduler Policy

The selection of the predicates and the priority functions defines the scheduler policy. The default policy is configured in the master configuration file master-config.yml as kubernetesMasterConfig.schedulerConfigFile. By default, it points to /etc/origin/master/scheduler.json. The current scheduling policy in force can be obtained with ?. A custom policy can replace it, if necessary, by following this procedure: Modify the Scheduler Policy.

Scheduler Policy File

The default scheduler policy is configured in the master configuration file master-config.yml as kubernetesMasterConfig.schedulerConfigFile, which by default points to /etc/origin/master/scheduler.json. Another example of scheduler policy file:

Scheduler Policy File

Storage

https://docs.openshift.com/container-platform/latest/dev_guide/persistent_volumes.html
https://docs.openshift.com/container-platform/latest/architecture/additional_concepts/storage.html#persistent-volumes
https://docs.openshift.com/container-platform/latest/install_config/persistent_storage/index.html

Volume

Volumes are mounted filesystems available to pods and their containers. Volumes may be backed by a number of host-local or network attached storage endpoints. The simplest volume type is EmptyDir, which is a temporary directory on a single machine. Administrators may also allow to request and attach Persistent Volumes.

Persistent Volume

Kubernetes Persistent Volume

The persistent volumes and persistent volume claims can be listed with oc get pv and oc get pvc.

Persistent Volume Operations

Persistent Volume Definition File

Persistent Volume Definition File

Persistent Volume Claim

A persistent volume claim is a request for a persistence resource with specific attributes, such as storage size. Persistent volume claims are matched to available volumes and binds the pod to the volume. This process allows a claim to be used as a volume in a pod: OpenShift finds the volume backing the claim and mounts it into the pod. Persistent volume claims are project-specific objects.

The pod can be disassociated from the persistent volume by deleting the persistent volume claim. The persistent volume transitions from a "Bound" to "Released" state.

Persistent Volume Claim Definition

All persistent volume claims for the current project can be listed with:

oc get pvc
Persistent Volume Claim Operations

Temporary Pod Volume

https://docs.openshift.com/container-platform/latest/dev_guide/volumes.html#dev-guide-volumes

When temporary pod volumes are used, the data is written to /var/lib/origin/openshift.local.volumes/pods on the node where the pod is running.

etcd

Implements OpenShift's store layer, which holds the current state, desired state and configuration information of the environment.

OpenShift stores image, build, and deployment metadata in etcd. Old resources must be periodically pruned. If a large number of images/build/deployments are planned, etcd must be placed on machines with large amounts of memory and fast SSD drives.

etcd

etcd and Master Caching

Masters cache deserialized resource metadata to reduce the CPU load and keep the metadata in memory. For small clusters, the cache can use a lot of memory for negligible CPU reduction. The default cache size is 50,000 entries, which, depending on the size of resources, can grow to occupy 1 to 2 GB of memory. The cache size can be configured in master-config.yml.

Image Registries

https://docs.openshift.com/container-platform/latest/install_config/registry/index.html

Integrated Docker Registry

OpenShift contains an integrated Docker image registry, providing authentication and access control to images. The integrated registry should not be confused with the OpenShift Container Registry (OCR), which is a standalone solution, which can be used outside of an OpenShift environment. The integrated registry is installed as part of the cluster installation process. The integrated registry is the default location where users can push images they build. Users push images into registry and whenever a new image is stored in the registry, the registry notifies OpenShift about it and passes along image information such as the namespace, the name and the image metadata. Various OpenShift components reacts by creating new builds and deployments. Third party image registry providers (http://dockerhub.com, or RedHat's own http://registry.access.redhat.com) can also be integrated. In general, any server implementing Docker registry API can be integrated.

The integrated registry is an application deployed within a project; it is deployed as a privileged container (referred to as an infrastructure container). The default integrated registry is deployed as part of the "default" project. It consists of a a "docker-registry" service, a "docker-registry" deployment configuration, a "registry" service account and a role binding that associates the service account to the "system:registry" cluster role. This is the registry's definition generated with oadm registry -o yaml.

Questions':

  • How can I "instruct" an application to use a specific registry. a “lab” application to use registry-lab
  • How do pods that need images go to that registry and not other? The answer to this question lays in the way an application uses the registry.

The default registry comes with a registry console.

The Registry Console

The registry comes with a "registry console" that allows web-based access to the registry. An URL example is https://registry-console-default.apps.openshift.novaordis.io. The registry console is deployed as a registry-console pod part of the "default" project.

More details:

https://docs.openshift.com/container-platform/latest/install_config/registry/deploy_registry_existing_clusters.html#registry-console

Stand-alone Registry

https://docs.openshift.com/container-platform/latest/install_config/install/stand_alone_registry.html#install-config-installing-stand-alone-registry

OpenShift comes with an integrated container registry. However, a stand-alone container registry can be installed and used with the OpenShift cluster.

Registry Operations

Registry Operations

Service

A service represents a group of pods, which may individually come and go, and its primary function is to provide the permanent IP, hostname and port for other applications to use. Each service has a service IP address and a port, which is allocated from the services subnet. The service IP address is sometimes referred to as the "cluster IP", which is consistent with the abstraction provided by the service: a cluster of pods providing the service. The services constitute the service layer.

The service IP is available internally in the cluster. It is not routable (in the IP sense) externally. The service can be exposed externally via a route.

A service resource is an abstraction that defines a logical set of pods and a policy that is used to access the pods. The service layer is how applications communicate with one another. The service is a Kubernetes concept. The service serves as an internal load balancer: it identifies a set of replicated pods and then proxies the connections it receives to those pods (routers provide external load balancing). The service is not a "thing", but an entry in the configuration. Backing pods can be added or removed to or from the service arbitrarily. This way, anything that depends on the service can refer to it as a consistent IP:port pair. The services uses a label selector to find all the running containers associated with it.

Services can be consumed from other pods using the values of the <SERVICE>_HOST environment variables that are injected by the cluster.

Service Definition

Services of a project can be displayed with:

oc get all
oc get svc

Service Endpoint

An <pod IP address>:<port> pair that corresponds to a process running inside of a container, on a pod, on a node. The service endpoint coordinate can be obtained by executing:

oc describe service <service-name>
Name:			logging-kibana
...
Endpoints:		10.129.2.17:3000
...

Relationship between Services and the Pods providing the Service

API

Kubernetes API

Networking

Required Network Ports

https://docs.openshift.com/enterprise/3.1/install_config/install/prerequisites.html#prereq-network-access

Default Network Interface on a Host

Every time OpenShift refers to the "default interface" on a host, it means the network interface associated with the default route. This is the logic that figures it out is here:

Default Network Interface on a Host

SDN, Overlay Network

https://docs.openshift.com/container-platform/latest/architecture/additional_concepts/sdn.html#architecture-additional-concepts-sdn
https://docs.openshift.com/container-platform/latest/install_config/configuring_sdn.html

All hosts in the OpenShift environment are clustered and are also members of the overlay network based on a Software Defined Network (SDN). Each pod gets its own IP address, by default from the 10.128.0.0/14 subnet, that is routable from any member of the SDN network. Giving each pod its own IP address means that pods can be treated like physical hosts or virtual machines in terms of port allocation, networking, naming, service discovery, load balancing, application configuration and migration.

However, it is not recommended that a pod talks to another directly by using the IP address. Instead, they should use services as an indirection layer, and interact with the service, that may be deployed on different pods at different times. Each service also gets its own own IP address from the 172.30.0.0/16 subnet.

The Cluster Network

The cluster network is the network from which pod IPs are assigned. These network blocks should be private and must not conflict with existing network blocks in your infrastructure that pods, nodes or the master may require access to. The default subnet value is 10.128.0.0/14 (10.128.0.0 - 10.131.255.255) and it cannot be arbitrarily reconfigured after deployment. The size and address range of the cluster network, as well as the host subnet size are configurable during installation. Configured with 'osm_cluster_network_cid' at installation.

The master maintains a registry of nodes in etcd, and each node gets allocated upon creation an unused subnet from the cluster network. Each node gets a /23 subnet, which means the cluster can allocate 512 subnets to nodes, and each node has 510 addresses to assign to containers running on it. Once the node is registered with the master, and gets its cluster network subnet, SDN creates on the node three devices:

  • br0 - the OVS bridge device that pod containers will be attached to. The bridge is configured with a set of non-subnet specific flow rules.
  • tun0 - an OVS internal port (port 2 on br0). This gets assigned the cluster subnet gateway address, and it is used for external access. The SDN configures net filter and routing rules to enable access from the cluster subnet to the external network via NAT.
  • vxlan_sys_4789. This is the OVS VXLAN device (port 1 on br0) which provides access to containers on remote nodes. It is referred to as "vxlan0".

Each time a pod is started, the SDN assigns the pod a free IP address from the node's cluster subnet, attaches the host side of the pod's veth interface pair to the OVS bridge br0, adds OpenFlow rules to the OVS database to route traffic addressed to the new pod to the correct OVS port. If a ovs-multitenant plug-in is active, it also adds OpenFlow rules to tag traffic coming from the pod with the pod's VIND, and to allow traffic into the pod if the traffic's VIND matches the pod's VIND, or it is the privileged VIND 0.

Nodes update their OpenFlow rules in response to master's notification in case new nodes are added or leave. When a new subnet is added, the node adds rules on br0 so that packets with a destination IP address the remote subnet go to vxlan0 (port 1 on br0) and thus out onto the network. The ovs-subnet plug-in sends all packets across the VXLAN with VNID 0, but the ovs-multitenant plug-in uses the appropriate VNID for the source container.

The SDN does not allow the master host (which is running the OVS processes) to access the cluster network, so a master does not have access to pods via the cluster network, unless it is also running a node.

When ovs-multitenant plug-in is active, the master also allocates VXLAN VNIDs to projects. VNIDs are used to isolate the traffic.

Packet Flow

TODO: https://docs.openshift.com/container-platform/3.5/architecture/additional_concepts/sdn.html#sdn-packet-flow

Pod IP Address

A cluster network IP address, that gets assigned to a pod.

The Services Subnet

OpenShift uses a "services subnet", also known as "kubernetes services network", in which OpenShift Services will be created within the SDN. This network block should be private and must not conflict with any existing network blocks in the infrastructure to which pods, nodes, or the master may require access to. It defaults to 172.30.0.0/16. It cannot be re-configured after deployment. If changed from the default, it must not be 172.16.0.0/16, which the docker0 network bridge uses by default, unless the docker0 network is also modified. It is configured with 'openshift_master_portal_net', 'openshift_portal_net' at installation and populates the master configuration file servicesSubnet element. Note that Docker expects its insecure registry to available on this subnet.

The services subnet IP address of a specific service can be displayed with:

oc describe service <service-name>

or

oc get svc <service-name>

The Service IP Address

An IP address from the services subnet that is associated with a service. The service IP address is reported as "Cluster IP" by:

oc get svc <service-name>

This makes sense if we think about a service as a cluster of pods that provide the service.

Example:

oc get svc logging-es
NAME         CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
logging-es   172.30.254.155   <none>        9200/TCP   31d

Docker Bridge Subnet

Default 172.17.0.0/16.

Network Plugin

OpenShift Network Plugins

Open vSwitch

Open vSwitch

DNS

OpenShiftDNSConcepts.png

External DNS Server

An external DNS server is required to resolve the public wildcard name of the environment - and as consequence, all public names of various application access points - to the public address of the default router, as per networking workflow. If more than one router is deployed, the external DNS server should resolve the public wildcard name of the environment to the public IP address of the load balancer. For more details about initial configuration details see External DNS Setup.

Optional Support DNS Server

An optional support DNS server can be setup to translate local hostnames (such as master1.ocp36.local) to internal IP addresses allocated to the hosts running OpenShift infrastructure. There is no good reason to make these addresses and names publicly accessible. The OpenShift advanced installation procedure factors it in automatically, as long as the base host images OpenShift is installed in top of is already configured to use it to resolve DNS names via /etc/resolv.conf or NetworkManager. No additional configuration via openshift_dns_ip is necessary.

Internal DNS Server

A DNS server used to resolve local resources. The most common use is to translate service names to service addresses. The internal DNS server listens on the service IP address 172.30.0.1:53. It is a SkyDNS instance built into OpenShift. It is deployed on the master and answers the queries for services. The naming queries issued from inside containers/pods are directed to the internal DNS service by Dnsmasq instances running on each node. The process is described below.

The internal DNS server will answer queries on the ".cluster.local" domain that have the following form:

  • <namespace>.cluster.local
  • <service>.<namespace>.svc.cluster.local - service queries
  • <name>.<namespace>.endpoints.cluster.local - endpoint queries

Service DNS can still be used and responds with multiple A records, one for each pod of the service, allowing the client to round-robin between each pod.

Dnsmasq

Dnsmasq is deployed on all nodes, including masters, as part of the installation process, and it works as a DNS proxy. It binds on the default interface, which is not necessarily the interface servicing the physical OpenShift subnet, and resolves all DNS requests. All OpenShift "internal" names - service names, for example - and all unqualified names are assumed to be in the "namespace.svc.cluster.local", "svc.cluster.local", "cluster.local" and "local" domains and forwarded to the internal DNS server. This is done by Dnsmasq, as configured from /etc/dnsmasq.d/origin-dns.conf:

server=/cluster.local/172.30.0.1 

This configuration tells Dnsmasq to forward all queries for names in the "cluster.local" domain and sub-domains to the internal DNS server listening on 172.30.0.1.

This is an example of such query (where 192.168.122.17 is the IP address of the local node on which Dnsmasq binds, and 172.30.254.155 is a service IP address) :

nslookup logging-es.logging.svc.cluster.local
Server:		192.168.122.17
Address:	192.168.122.17#53

Name:	logging-es.logging.svc.cluster.local
Address: 172.30.254.155

The non-OpenShift name requests are forwarded to the real DNS server stored in /etc/dnsmasq.d/origin-upstream-dns.conf:

server=192.168.122.12

The configuration file /etc/dnsmasq.d/origin-dns.conf is deployed at installation, while /etc/dnsmasq.d/origin-upstream-dns.conf is created dynamically every time the external DNS changes, as it is the case for example when the interface receives a new DHCP IP address.

For more details, see

Dnsmasq

Container /etc/resolv.conf

The container /etc/resolv.conf is created while the container is being assembled, based on information available in node-config.yamll. Also see:

External OpenShift DNS Resources

Routing Layer

OpenShift networking provides a platform-wide routing layer that directs outside traffic to the correct pods IPs and ports. The routing layer cooperates with the service layer. Routing layer components run themselves in pods. They provide automated load balancing to pods, and routing around unhealthy pods. The routing layer is pluggable and extensible. For more details about the overall OpenShift workflow, see "OpenShift Workflow".

Router

https://docs.openshift.com/container-platform/3.5/install_config/router/index.html
https://docs.openshift.com/container-platform/3.5/architecture/core_concepts/routes.html
https://docs.openshift.com/container-platform/latest/install_config/router/index.html

The router service routes external requests to applications inside the OpenShift environment. The router service is deployed as one or more pods. The pods associated with the router service are deployed as infrastructure containers on infrastructure nodes. A router is usually created during the installation process. Additional routers can be created with oadm router.

The router is the ingress point for all traffic flowing to backing pods implementing OpenShift Services. Routers run in containers, which may be deployed on any node (or nodes) in an OpenShift environment, as part of the "default" project. A router works by resolving fully qualified DNS external names ("kibana.apps.openshift.novaordis.io") external requests are associated with into pod IP addresses and proxying the requests directly into the pods that back the corresponding service. The router gets pod IPs from the service, and proxies the requests to pods directly, not through the Service.

Routers directly attach to port 80 and 443 on all interfaces on a host, so deployment of the corresponding containers should be restricted to the infrastructure hosts where those points are available. Statistics are usually available on port 1936.

Relationship between Service and Router

When the default router is used, the service layer is bypassed. The Service is used only to find which pods the service represents. The default router does the load balancing by proxying directly to the pod endpoints.

OpenShiftRouterConcepts.png

Sticky Sessions

The router configuration determines the sticky session implementation. The default HAProxy template implements sticky session using balance source directive.

Router Implementations

HAProxy Default Router

https://docs.openshift.com/container-platform/3.5/architecture/core_concepts/routes.html#haproxy-template-router

By default, the router process is an instance of HAProxy, referred to as the "Default Router". It uses "openshift3/OpenShift Container Platform-haproxy-router" image, and it supports unsecured, edge-terminated, re-encryption terminated and passthrough terminated routes matching on HTTP host and request path.

F5 Router

The F5 Router integrates with existing F5 Big-IP systems.

Router Operations

Information about the existing routers can be obtained with:

 oadm router -o yaml

More details on router operations are available here:

Router Operations

Route

Logically, a route is an external DNS entry, either in a top level domain or a dynamically allocated name, that is created to point to an OpenShift service so that the service can be accessed outside the cluster. The administrator may configure one or more routers to handle those routes, typically through an Apache or HAProxy load balancer/proxy.

The route object describes a service to expose by giving it an externally reachable DNS host name.

A route is a mapping of an FQDN and path to the endpoints of a service. Each route consist of a route name, a service selector and an optional security configuration:

NAME             HOST/PORT                            PATH      SERVICES         PORT      TERMINATION          WILDCARD
logging-kibana   kibana.apps.openshift.novaordis.io             logging-kibana   <all>     reencrypt/Redirect   None

A route can be unsecured or secured.

The secure routes specify the TLS termination of the route, which relies on SNI (Server Name Indication). More specifically, they can be:

  • edge secured (or edge terminated). Occurs at the router, prior to proxying the traffic to the destination. The front end of the router serves the TLS certificates, so they must be configured in the route. If the certificates are not configured in the route, the default certificate of the router is used. Connection from the router into the internal network are not encrypted. This is an example of edge-termintated route.
  • passthrough secured (or passthrough terminated). The encrypted traffic is send to to destination, the router does not provide TLS termination, hence no keys or certificates are required on the router. The destination will handle certificates. This method supports client certificates. This is an example of passthrough-termintated route.
  • re-encryption terminated secured. This is an example of re-encyrption-termintated route.
Route Definition

Routes can be displayed with the following commands:

oc get all
oc get routes

Routes can be created with API calls, an JSON or YAML object definition file or the oc expose service command.

Path-Based Route

A path-based route specifies a path component, allowing the same host to server multiple routes.

Route with Hostname

Routes let you associate a service with an externally reachable hostname. If the hostname is not provided, OpenShift will generate on based on the <routename>-.<namespace>].<suffix> pattern.

Default Routing Subdomain

The suffix and the default routing subdomain can be configured in the master configuration file.

Route Operations

Route Operations

Networking Workflow

The networking workflow is implemented at the networking layer.

  • A user requests a page by pointing the browser to http://myapp.mydomain.com
  • The external DNS server resolves that request to the IP address of one of the hosts that host the default router. That usually requires a wildcard C name in the DNS server pointed to the node that hosts the router container.
  • Default Router selects the pod from the list of pods listed by the service and acts as a proxy for that pod (this is where the port can be translated from the external 443 to the internal 8443, for example).

Resources

  • cpu, requests.cpu
  • memory, requests.memory
  • limits.cpu
  • memory.cpu
  • pods
  • replicationcontrollers
  • resourcequotas
  • services
  • Secrets
  • configmaps
  • persistentvolumeclaims
  • openshif.io/imagestreams

Most resources can be defined in JSON or YAML files, or via an API call. Resources can be exposed via the downward API to the container.

Resource Quota

ResourceQuota object enumerates hard resource usage limits per project.

ClusterResourceQuota object enumerates hard resource usage limits across the cluster.

After a quota is first declared on a project, the system restricts the ability to create new resources that may exceed the quota until usage statistics are calculated. If the resource creation request exceeds the quota, the server will deny the action and will return an error message.

When a resource is created, the quota usage is updated immediately. When a resource is delete, the quota usage is updated during the next full per-project statistics update.

Resource Quota Definition File

Operations:

Project

Projects allows groups of users to work together, define ownership of resources and manage resources - they can be seen as the central vehicle for managing regular users' access to resources. The project restricts and tracks use of resources with quotas and limits. A project is a Kubernetes namespace with additional annotations. A project can contain any number of containers of any kind, and any grouping or structure can be enforced using labels. The project is the closest concept to an application, OpenShift does not know of applications, though it provides a new-app command. A project lets a community of users to organize and manage their content in isolation from other communities. Users must be given access to projects by administrators—unless they are given permission to create projects, in which case they automatically have access to their own projects.

Most objects in the system are scoped by a namespace, with the exception of:

Each project has its own set of objects: pods, services, replication controllers, etc. The names of each resource are unique within a project. Developers may request projects to be create, but administrators control the resources allocated to the projects.

What actions a user can or cannot perform on a project's objects are specified by policies.

Constraints are quotas for each kind of object that can be limited.

New projects are created with:

oc new-project

Current Project

The current project is a concept that applies to oc, and specifies the project oc commands apply to, without to explicitly having to use the -n <project-name> qualifier. The current project can be set with oc project and read with oc status. The current project is part of the CLI tool's current context, maintained in user's .kube/config.

Global Project

In the context of an ovs-multitenant SDN plugin, a project is global if if can receive cluster network traffic from any pods, belonging to any project, and it can send traffic to any pods and services. Is the default project a global project?. A project can be made global with:

oadm pod-network make-projects-global <project-1-name> <project-2-name>

Standard Projects

Default Project

The "default" project is also referred to as "default namespace". It contains the following pods:

  • the integrated docker registry pod (memory consumption based on a test installation: 280 MB)
  • the registry console pod (memory consumption based on a test installation: 34 MB)
  • the router pod (memory consumption based on a test installation: 140 MB)

In case the ovs-multitenant SDN plug-in is installed, the "default" project has VNID 0 and all its pods can send and receive traffic from any other pods.

The "default" project can be used to store a new project template, if the default one needs to be modified. See Template Operations - Modify the Template for New Projects.

"logging" Project

If logging support is deployed at installation or later, the participating pods (kibana, ElasticSearch, fluentd, curator) are members of the "logging" project.

This is the memory consumption based on a test installation:

  • kibana pod: 95 MB
  • elasticsearch pod: 1.4GB
  • curator pod: 10 MB
  • fluentd pods max 130 MB

"openshift-infra" Project

Contains the metrics components:

"openshift" Project

Contains templates.

Other Standard Projects

  • "kube-system"
  • "management-infra"

Projects and Applications

Each project may have multiple applications, and each application can be managed individually.

A new application is created with:

oc new-app

Application Operations:

Application Operations

Project Operations

Project Operations

Template

https://docs.openshift.com/container-platform/latest/dev_guide/templates.html
https://docs.openshift.org/latest/dev_guide/templates.html

A template is a resource that describes a set of objects that can be parameterized and processed, so they can be created at once. The template can be processed to create anything within a project, provided that permissions exist. The template may define a set of labels to apply to every object defined in the template. A template can use preset variables or randomized values, like passwords.

Templates can be stored in, and processed from files, or they can be exposed via the OpenShift API and stored as part of the project. Users can define their own templates within their projects.

The objects define in a template collectively define a desired state. OpenShift's responsibility is to make sure that the current state matches the desired state.

Template Definition

Specifying parameters in a template allows all objects instantiated by the template to see consistent values for these parameters when the template is processed. The parameters can be specified explicitly, or generated automatically.

The configuration can be generated from template with oc process command.

Most templates use pre-built S2I builder images, that includes the programming language runtime and its dependencies. These builder images can also be used by themselves, without the corresponding template, for simple use cases.

New Project Template

The master provisions projects based on the template that is identified by the "projectRequestTemplate" in master-config.yaml file. If nothing is specified there, new projects will be created based on a built-in new project template that can be obtained with:

oadm create-bootstrap-project-template -o yaml

Template Libraries

Template Operations

Template Operations

CI/CD Support

OpenShift enables DevOps. It has built-in support for the Jenkins CI server, providing a native way of doing CICD.

OpenShift can integrate with Git-based source code repositories. When a developer pushes code to the GIt repository, Jenkins pulls the code and performs a project build. After the build is complete, Jenkins invokes the OpenShift master node API to initiate a "source-to-image" (S2I) build. As part of the SI process, the latest project artifact is downloaded from Jenkins and included in the image that runs in the OpenShift node.

More details:

Jenkins and OpenShift
OpenShift CI/CD Operations

Build

https://docs.openshift.com/container-platform/latest/architecture/core_concepts/builds_and_image_streams.html

A build is the process of transforming input parameters into a resulting object. In most cases, that means transforming source code, other images, Dockerfiles, etc. into a runnable image. A build is run inside of a container and has the same restrictions normal pods have. A build usually results in an image pushed to a Docker registry, subject to post-build tests.

Docker can be used to build images directly, but OpenShift supplies additional builders to assist with creating an image, by adding code or configuration to existing images. The build process is defined by its build strategy.

The build operations are executed by OpenShift in privileged containers.

Builds for a project can be reviewed by navigating with the web console to the project -> Builds -> Builds.

Build operations:

Build Operations

Build Configuration

A BuildConfig object is the definition of an entire build. It contains a description of how to build source code and a base image into a new image, the specification of triggers (web-hooks), which specify events for invoking builds, for example when a newer base image is available, or new deployments.

Builds can be:

  • Source based and use builder images for common languages like Java, PHP, Ruby, or Python
  • Docker based and create builds from a Dockerfile.

OpenShift creates Docker containers from build images and pushes the to an integrated registry.

The BuildConfig consists of:

  • Triggers (GitHub webhook, generic webhook, image changes)
  • Parameters (source, strategy, output)

Both oc new-app and oc new-build create BuildConfig objects. oc start-build starts the build.

The build's maximum duration can be set with:

spec:
  completionDeadlineSeconds: 1800

Build Sources

Build Strategy

Docker Build

The Docker build expects repository with Dockerfile and all artifacts required to produce runnable image. It invokes the docker build command, and the result is a runnable image.

Source-to-Image (S2I) Build

https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/builds_and_image_streams.html#source-build

It is a process for building Docker images starting from a base image, known as builder image, possibly binary dependencies and source code, and assembling a new Docker image. The new image incorporates the base image and compiled source code. The image ready to use with the docker run command. S2I supports incremental builds, which reuse previously downloaded dependencies, previously built artifacts, etc.

A Git repository URL, which contains code written in any supported framework, and a builder image, which contains the O/S and the framework to support the code, must be specified to start a S2I build.

S2I scripts can be written to inject application code into almost any existing Docker image. The assemble process can perform a large number of complex operations without creating a new layer at each step, resulting in a fast process. S2I allows to build the application consistently if an underlying image needs a patch due to a security issue.

Builds can be triggered manually or automatically by setting a web hook into the source repository.

The source-to-image runs are driven by the following scripts:

Builder Image

The builder image contain both the build and the runtime environment for that source code.

Source-to-Image Process

  • Pull code from a source code repository.
  • Detect the runtime the source code needs.
  • Starts a pod from a base image specific for that type of runtime.

Incremental Builds

It is possible to reuse artifacts from previously-build images by setting an incremental build.

strategy
  type: "Source"
  sourceStrategy:
    from:
       ...
    incremental: true

Pipeline Build

The pipeline build strategy allows developers to define a Jenkins pipeline for execution by Jenkins pipeline plugin.

Custom Build

The custom build strategy allows to define a specific builder image responsible for the entire build process, which allows you to customize the build process.

The custom builder image is a plain Docker image embedded with build process logic.

Chained Builds

Two builds may be chained: one produces binaries from a builder image and source, and pushes the artifacts into an image stream; the other pulls the binaries produces by the first build from the image stream and places them into a runtime image, based on a Dockerfile, A separated image is thus created.

Build Mechanics

  • A working directory is created.
  • Input is placed in the working directory: the git repository is cloned, files are copied.
  • The build process changes to 'contextDir' - anything that resides outside 'contextDir' is ignored by the build.
  • Inline Dockerfile, if any, is placed in the current directory.

Build can be assigned to specific nodes, by specifying labels in the "nodeSelector" field of the build configuration:

kind: "BuildConfig"
...
spec:
  nodeSelector:
    env: app

Build Resources

Can be setup in the build configuration:

apiVersion: "v1"
kind: "BuildConfig"
...
spec:
  resources:
    limits:
      cpu: ...
      memory: ...

Image

https://docs.openshift.com/container-platform/latest/architecture/core_concepts/containers_and_images.html#docker-images

An image is a layered Linux filesystem that contains all dependencies a container needs to run (application code, application dependencies and any supporting operating system libraries), and the metadata describing its needs and capabilities. An image is identified by a name that can be local to the current cluster - meaning that it is maintained in the internal Docker registry - or point to a remote Docker registry.

For more details about Docker images see:

Docker Image

Images in OpenShift are immutable.

Image Stream

An image stream is similar to a Docker image repository, in that it contains a group of related Docker images identified by image stream tags. Logically it is analogous to a branch in a source code repository.

The image stream presents a single virtual view of related images. The stream may contain images from the OpenShift integrated Docker registry, other image streams, other image repositories.

OpenShift stores complete metadata about each image, including example command, entry point and environment variables.

Builds and deployments can watch an image stream to receive notifications when new images are added and react by performing a build or a deployment.

Image Stream Definition

Operations:

ImageStream Operations

Image Stream Tag

The default tag is called "latest".

A tag may point to an external Docker registry, at other tags in the same stream, or be controlled to directly point to known images.

Images can be pushed to an image stream tag directly via the integrated Docker registry.

Image Pull Policy

When the container is created, the runtime uses the "imagePullPolicy" to determine whether to pull the image prior to starting the container. More details available here:

Pod Definition File - imagePullPolicy

Deployment

https://docs.openshift.com/container-platform/latest/dev_guide/deployments/how_deployments_work.html

A deployment is a replication controller based on a user-defined template called a deployment configuration.

A deployment adds extended support for software development and deployment lifecycle. Deployments are created manually or in response to triggered events.

The deployment system provides a deployment configuration, which is a template for deployments, triggers that drive automated deployments in response to events, user-customizable strategies to transition from the previous deployment to the new deployment, rollback procedure, and manual replication scaling.Deployment configuration version increments each time new deployment is created from configuration. Deployments allow defining hooks to be run before and after the replication controller is created.

Deployments allow rollbacks.

Deployments allow manual replication scaling or autoscaling.

The deployments are triggered with oc deploy.

Deployment Configuration

A deployment configuration is a user-defined template for performing deployments, which result in running applications. The deployment configuration defines the template for a pod. It manages deploying new images or configuration changes whenever those change. A single deployment configuration is usually analogous to a single micro service.

Each deployment is represented as a replication controller. The OpenShift environment creates a replication controller to run the application in response to a deployment configuration. The deployment configuration contains a version number that is incremented each time a replication controller is created from the configuration.

Deployment configurations can support many different deployment patterns, including full restart, customizable rolling updates, and fully custom behaviors, as well as pre- and post-hooks. It supports automatic rolling back to the last successful revision of configuration, in case the current template fails to deploy.

The DeploymentConfig contains:

  • Replication controller definition.
  • Default replica count fo the deployment.
  • Triggers for creating new deployments automatically, in response to events. If no triggers are defined, deployments must be started manually.
  • Strategy for transitioning before deployments.
  • Life-cycle hooks. Every hook has a failure policy (Abort, Retry, Ignore).
DeploymentConfig Definition

The DeploymentConfig for a project can be listed with:

oc get all
oc get dc

Deployment Configuration Triggers

Triggers can be specified in the deployment configuration, and can be modified from command line.

ConfigChange

Results in a new deployment, and a new replication controller being created whenever changes are detected to replication controller template of deployment configuration. In the presence of a ConfigChange trigger, the first replication controller is automatically created when the deployment configuration itself is created.

ImageChange

Results in a new deployment, and a new replication controller being created, whenever value of image stream tag changes.

Replication Controller

A replication controller is one of the pod controller types available in OpenShift. It resides on master, and insures that the specified number of pod replicas defined in the replication controller configuration are running at all times.

Logically, the replication controllers constitute the replication layer.

The definition of a replication controller includes the number of replicas to be maintained, the pod definition for creating the replicated pod, and a selector for identifying managed pods. If pods exit or are deleted, either explicitly or because the node they run on is taken out of service, the replication controller instantiates more pods up to desired number. If there are more pods running than desired, the replication controller deletes as many as necessary. However, it is NOT the replication controller's job to perform autoscaling based on load or traffic. Replication controllers do not exist as physical processes, meaning they do not run in pods, they only exist as entries in etcd, and the master executes the logic.

A replication controller is most commonly used to represent a single deployment of part of an application based on a built image.

The replication controllers of a project can be listed with:

oc get all
oc get rc

Rollout

A rollout is exposed as a replication controller, and the deployment process manages scaling down old replication controllers and scaling up new ones. Implements one of the deployment strategies. The rollout is performed with:

oc rollout

Rollback

Deployments allow rollbacks to previous versions of an application: when one deployment is superseded by another, the previous replication controller is retained, with its number of replicas set to 0. When triggered - the template fails to deploy -, the rollback reverts an application to the last successful deployment. It is done with oc rollback, API or web console.

Deployment Strategy

The deployment configuration defines a deployment strategy. The deployment strategy determines the deployment process and it is defined by the deployment configuration. The deployment strategy uses readiness checks to determine if a new host is ready for use. If the readiness check fails, the deployment configuration retries until it times out. The readiness timeout value is set in deployment configuration.

The deployment strategy is implemented during the rollout process.

Rolling Deployment Strategy

The default deployment strategy, if a deployment strategy is not explicitly specified in the deployment configuration. It performs rolling updates. It supports life-cycle hooks for injecting code into the deployment process.

It consists in the following steps:

  • Execute the "pre" life-cycle hook.
  • Scale up new deployment by one or more pods, based on maxSurge value, waiting until all readiness checks complete.
  • Scale down the old deployment by one or more pods, based on maxUnavailable value.
  • Repeat scaling until the new deployment reaches desired replica count and the old deployment has scaled to zero.
  • Execute any "post" life-cycle hook.

When scaling down, the strategy waits for pods to become ready, so it can decide if can further scaling would affect availability. If scaled up pods never become ready, the deployment times up and results in a deployment failure.

Recreate Deployment Strategy

Recreate strategy is appropriate when the application does not support old versions and new versions running together, or when the application uses ReadWriteOnce volumes that do not support sharing between multiple replicas.

The recreate strategy implies downtime: there is a time interval when on application instance is running.

It consists in the following steps:

  • Execute "pre" life-cycle hook.
  • Scale down previous deployment to zero.
  • Scale up new deployment.
  • Execute "post" life-cycle hook.

During scale up, if the replica count of the deployment is greater than one, the first deployment replica is validated for readiness before fully scaling up the deployment. If this validation fails, the deployment fails.

Custom Deployment Strategy

Allows for custom commands.

The optional "command" array overrides the CMD directive specified in the Dockerfile.

The optional "environment" variables are added to the strategy process' execution.

Deployment Operations

Deployment Operations

Environment Variables to Use for Strategy Process

  • OPENSHIFT_DEPLOYMENT_NAME
  • OPENSHIFT_DEPLOYMENT_NAMESPACE

Pipeline

A pipeline is a Jenkins jobs enabled by the pipeline plugin.

https://jenkins.io/doc/book/pipeline/overview/

Used by the pipeline build.

Region

Zone

High Availability (HA)

Infrastructure HA

See Master HA.

Application HA

OpenShift insures high availability by deploying the same image in multiple containers across multiple hosts and load balancing among them. This technique also provides horizontal scalability for a service packaged into an image.

Installation

There are two installation procedures: RPM and Containerized.

An RPM installation installs all services through package management and configures services to run within the same user space,

A containerized installation installs services using container images and runs separate services in individual containers.

For practical details on installing various OpenShift version, see:

OpenShift Installation

Related:

OpenShift Logging Installation

Installation is performed by Ansible, usually deployed on the environment's support server. Ansible configuration is available under /etc/ansible and the installation logic under /usr/share/ansible.

Security

OpenShift Security Concepts

Metrics

OpenShift Metrics Concepts

Horizontal Pod Autoscaler (HPA)

https://docs.openshift.com/container-platform/latest/dev_guide/pod_autoscaling.html#dev-guide-pod-autoscaling

Logging

OpenShift Logging Concepts

Events

https://docs.openshift.com/container-platform/3.5/dev_guide/events.html

OpenShift events incapsulate information about specific conditions detected in the OpenShift cluster, and allow the OpenShift management facilities to record information about those occurrences in a resource-agnostic manner. They also allow administrators and developers to consume information about system components in a unified way. A list of events can be obtained with:

oc get events [-n <project-name>]

Events for a project can be reviewed by navigating with the web console to the project -> Monitoring -> Right Side.

Events contain:

  • type: Normal, Warning
  • kind: Configuration, Node, Pod, DaemonSet, Container, Health, Image, Image Manager, System.
  • reason
  • source
  • message

Event Reasons

  • FailedScheduling
  • OutOfDisk
  • MatchNodeSelector
  • SuccessfulCreate

DaemonSet

https://docs.openshift.com/container-platform/3.5/dev_guide/daemonsets.html
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

A DaemonSet is a OpenShift component that creates its associated pods and ensures they run on all (or some) nodes of a cluster. It is one of the pod controller types available in OpenShift.

If a node is added to the cluster, the DaemonSet insures that its associated pod will be scheduled on that node. When nodes are removed, the associated pods are shut down. Typical uses for DaemonSets are to run log collection agents (fluentd, logstash), node monitoring agents (collectd) or a cluster storage daemon (glusterd, ceph). Usually, a DaemonSet is needed for each type of daemon.

The DaemonSet decides whether it manages a pod or not based on the label selector expression specified in its definition:

...
Selector:	component=fluentd,provider=openshift
...

Note that internally, the declared pod template’s label selector must match the label selector above.

When a pod is managed by a DaemonSet, the node the pod is scheduled to run on is selected by the DaemonSet, so the scheduler ignores it. The "unschedulable" field of a node is not respected by the DaemonSet controller. Also, the DaemonSet controller can make pods even if the scheduler has not been started, and this helps with cluster bootstrap. The DaemonSet chooses the nodes to run its pods on based on the node selector specified in its definition:

...
Node-Selector:	logging=true
...

DaemonSet.png

After a successful placement, the node selector expression is recorded in the pod's definition.

DaemonSet Operations

ConfigMaps

https://docs.openshift.com/container-platform/3.6/dev_guide/configmaps.html

A ConfigMap is a component that holds key/value pairs of configuration data, and that can be consumed by pods, or can be used to store configuration for OpenShift system components such as controllers. It is a mechanism to inject containers with configuration while keeping the containers agnostic of the OpenShift platform. Aside from fine-grained information like individual properties, ConfigMaps can also store coarse-grained information such as entire configuration files or JSON blobs. The ConfigMaps can populate environment variables, set command-line arguments in a container and populate configuration files in a volume.

A ConfigMap is similar to a secret, but designed to be more convenient when working with strings that do not contain sensitive information.

ConfigMap can be created from directories, files, literal values.

The ConfigMaps must be created before they are consumed in pods. They cannot be shared between projects. If the ConfigMap is updated, it must be redeployed in then pod for the pod to see the changes.

ConfigMap Definition
ConfigMap Operations

Command Line Tools

OpenShift Command Line Tools (CLI)

Admission Control

https://docs.openshift.com/container-platform/latest/architecture/additional_concepts/admission_controllers.html

Admission control plug-ins intercept requests to the master API, after authentication and authorization was enforced. There's a chain of plug-ins (also known as the admission chain), and if a plug-in rejects the request, the request fails. The plug-in may modify the request object, and related resources. The default list of admission control plug-ins is configured in the master-config.yaml's admissionConfig/pluginConfig section.

Customizable admission control plug-ins:

OpenShift and JBoss

OpenShift and JBoss

Jobs

apiVersion: batch/v1
kind: Job

Cron Jobs

apiVersion: batch/v1
kind: CronJob

Configuration Data Externalization

Configuration data must never be stored in application source code, but it must externalized on storage.

Environment Variables

Declared in deployment configurations, replication controllers and build configurations. Set with

oc set env

Some environment variables are automatically exported:

  • <SVCNAME>_SERVICE_HOST
  • <SVCNAME>_SERVICE_PORT

Downward API

The downward API allows containers to consume information about OpenShift objects. The field within a pod are selected using 'fieldRef' API type, which has two types: 'fieldPath', the path of field to select relative to the pod and 'apiVersion'. The downward API exposes the following selectors:

  • metadata.name: the pod name.
  • metadata.namespace: pod namespace.
  • metadata.labels
  • metadata.annotations
  • status.podIP

'resourceFieldRef'/'resource', which refers to the resource entries.

The information can be exposed to pods via environment variables and volumes.