OpenShift Installation Validation

External

• https://docs.openshift.com/container-platform/latest/install_config/install/advanced_install.html#advanced-verifying-the-installation

Internal

• OpenShift Operations
• Verifying an OpenShift 3.5 Installation

On All Nodes

OpenShift Packages

ansible nodes -m shell -a "yum list installed | grep openshift"

The desired OpenShift version must be installed.

OpenShift Version

ansible nodes -m shell -a "/usr/bin/openshift version"

master1.local | SUCCESS | ...
openshift v3.5.5.26
kubernetes v1.5.2+43a9be4
etcd 3.1.0

On Masters

On each master node, run as root:

oc get nodes --show-labels

Output example:

NAME                           STATUS                     AGE       LABELS
infranode1.openshift35.local   Ready                      17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,env=infra,kubernetes.io/hostname=infranode1.openshift35.local,logging-infra-fluentd=true,logging=true
infranode2.openshift35.local   Ready                      17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,env=infra,kubernetes.io/hostname=infranode2.openshift35.local,logging-infra-fluentd=true,logging=true
master1.openshift35.local      Ready,SchedulingDisabled   17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,kubernetes.io/hostname=master1.openshift35.local,logging-infra-fluentd=true,logging=true,openshift_schedulable=False
master2.openshift35.local      Ready,SchedulingDisabled   17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,kubernetes.io/hostname=master2.openshift35.local,logging-infra-fluentd=true,logging=true,openshift_schedulable=False
master3.openshift35.local      Ready,SchedulingDisabled   17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,kubernetes.io/hostname=master3.openshift35.local,logging-infra-fluentd=true,logging=true,openshift_schedulable=False
node1.openshift35.local        Ready                      17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,env=app,kubernetes.io/hostname=node1.openshift35.local,logging-infra-fluentd=true,logging=true
node2.openshift35.local        Ready                      17m       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,cluster=hadron,env=app,kubernetes.io/hostname=node2.openshift35.local,logging-infra-fluentd=true,logging=true

Web Console

At this point the web console should be exposed on the external interface.

https://master.openshift35.external/

https://master.openshift.novaordis.io/

Use the administrative user defined as part of your "identity provider" declaration.

Verify etcd

etcdctl cluster-health
etcdctl member list

Exported Filesystems

On the support node run exportfs and make sure the following filesystems are exported:

exportfs

/support-nfs-storage 192.168.122.0/255.255.255.0
/support-nfs-storage/registry <world> 
/support-nfs-storage/metrics <world>
/support-nfs-storage/logging <world>
/exports/logging-es-ops <world>

Docker Logs

Log into a few nodes and take a look at the docker logs:

journalctl -f -u docker

Docker Startup Paramenters

From the support/installation server, execute as "ansible":

ansible nodes -m shell -a "ps -ef | grep dockerd | grep -v grep"

Make sure "--selinux-enabled" and "--insecure-registry 172.30.0.0/16" are present.

Logging

ElasticSearch

ElasticSearch should be deployed, running, and operational - logs must not contain errors:

oc project logging
oc get pods -l 'component=es'
oc logs -f logging-es-3fs5ghyo-3-88749

fluentd

All nodes should run a fluentd pod, and the fluentd pods should be operational, their logs must not contain errors.

oc project logging
oc get pods -l 'component=fluentd'
NAME                    READY     STATUS    RESTARTS   AGE
logging-fluentd-2r4rt   1/1       Running   0          49m
logging-fluentd-37d72   1/1       Running   0          35m
logging-fluentd-4ljkn   1/1       Running   0          3h
logging-fluentd-74l39   1/1       Running   0          3h
logging-fluentd-7l25h   1/1       Running   0          3h
logging-fluentd-sbh7r   1/1       Running   0          3h
logging-fluentd-w4shg   1/1       Running   0          39m

oc logs -f logging-fluentd-2r4rt 
...

Kibana

ElasticSearch should be deployed, running, and operational - logs must not contain errors:

oc project logging
oc get pods -l 'component=kibana'
oc logs -f -c kibana-proxy logging-kibana-10-sb7sk
oc logs -f -c kibana logging-kibana-10-sb7sk

The Logging Portal

The logging portal should be available:

https://kibana.apps.openshift.novaordis.io/

Metrics

Test https://hawkular-metrics.openshift35.external/hawkular/metrics

oadm Diagnostics

oadm diagnostics