OpenShift Troubleshooting

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

Internal

Overview

The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here:

Change the Log Level for OpenShift Processes

--loglevel=10 seems to work fine.

Then tail the journalctl log. More details on getting logging information:

OpenShift Master and Node Process Logging

General Troubleshooting

Troubleshooting OpenShift Container Platform: Basics:

https://access.redhat.com/solutions/1542293

Master/Node Processes Troubleshooting

OpenShift Runtime

Metrics Troubleshooting

oadm diagnostics MetricsApiProxy

Troubleshooting Pods

oc -n <project-name> get pods
oc -n <project-name> describe po/<pod-name>

...
Name:			logging-fluentd-3kz30
...
Node:			node2/
...
Status:			Failed
Reason:			MatchNodeSelector
Message:		Pod Predicate MatchNodeSelector failed

In-line pod logs:

oc logs -f <pod-name>

Connect to a pod:

oc rsh <pod-name>

Connect to a Pod as Root

Log into the physical node running the pod. The pod - node association can be inferred executing

oc describe <pod-name> 

and extracting the "Node:" value.

On the node, run

docker ps

and identify the container ID.

Then

 docker exec -u 0 -it 3152a2509d92 /bin/bash

For more details on docker exec, see:

docker exec

As root, various diagnostic utilities can be temporarily installed:

yum install bind-utils

Copy Files into Pods

For more details on oc cp:

oc cp

Troubleshooting Routes

Troubleshooting Logging

 oadm diagnostics AggregatedLogging

Troubleshooting Kibana

Troubleshooting Kibana in OpenShift

Cases