OpenShift Troubleshooting: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
(Created page with "=Internal= * OpenShift =Overview= The general technique is to increase the logging level of various OpenShift master (api, controllers) and node proc...")
 
 
(51 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Internal=
=Internal=


* [[OpenShift#Subjects|OpenShift]]
* [[OpenShift TODEPLETE#Subjects|OpenShift TODEPLETE]]


=Overview=
=Overview=


The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here: [[https://kb.novaordis.com/index.php/OpenShift_Change_the_Log_Level_for_OpenShift_Processes|Change the Log Level for OpenShift Processes]]. --loglevel=10 seems to work fine. Then tail the journalctl log:
The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here:  


  journalctl -u atomic-openshift-master-controllers -f
{{Internal|OpenShift_Logging_Levels#Change_the_Log_Level_for_OpenShift_Processes|Change the Log Level for OpenShift Processes}}
 
--loglevel=10 seems to work fine.
 
Then tail the journalctl log. More details on getting logging information:
 
{{Internal|OpenShift_Logging#Master_and_Node_Process_Logging|OpenShift Master and Node Process Logging}}
 
=General Troubleshooting=
 
Troubleshooting OpenShift Container Platform: Basics:
 
{{External|https://access.redhat.com/solutions/1542293}}
 
=Installation Troubleshooting=
 
{{Internal|OpenShift Installation#Installation_Troubleshooting|Installation Troubleshooting}}
 
=Master/Node Processes Troubleshooting=
 
{{External|OpenShift Runtime|OpenShift Runtime}}
 
=Troubleshooting Pods=
 
  [[oc get#pods|oc -n <project-name> get pods]]
 
[[oc describe#pod|oc -n <project-name> describe po/<pod-name>]]
...
<font color=red>Name</font>: logging-fluentd-3kz30
...
<font color=red>Node</font>: node2/
...
<font color=red>Status</font>: Failed
Reason: MatchNodeSelector
Message: Pod Predicate MatchNodeSelector failed
 
In-line pod logs:
 
[[Oc_logs#-f|oc logs]] -f <''pod-name''>
 
Connect to a pod:
 
[[oc rsh]] <''pod-name''>
 
==Connect to a Pod as Root==
 
Log into the physical node running the pod. The pod - node association can be inferred executing
 
oc describe <''pod-name''>
 
and extracting the "Node:" value.
 
On the node, run
 
docker ps
 
and identify the container ID.
 
Then
 
  docker exec -u 0 -it 3152a2509d92 /bin/bash
 
For more details on docker exec, see:
 
{{Internal|docker exec|docker exec}}
 
As root, various diagnostic utilities can be temporarily installed:
 
yum install bind-utils
 
==Copy Files into Pods==
 
For more details on oc cp:
 
{{Internal|oc cp|oc cp}}
 
=Troubleshooting Projects=
 
* [[OpenShift_Project_Operations#Monitor_Project.27s_Event_Log|Monitor project's event log]]
 
=Troubleshooting Routes=
 
=Troubleshooting Logging=
 
  oadm diagnostics AggregatedLogging
 
==Troubleshooting Kibana==
 
{{Internal|Kibana and OpenShift#Troubleshooting|Troubleshooting Kibana in OpenShift}}
 
=Troubleshooting Metrics=
 
oadm diagnostics MetricsApiProxy
 
{{Internal|OpenShift Metrics Concepts|OpenShift Metrics Concepts}}
 
=Cases=
 
* [[OpenShift Kibana Started to Redirect to master1 instead of Public Master URL after Ansible Update]]
* [[OpenShift ElasticSearch pod fails to start with "Unable to load index mapping ..."]]
* [[OpenShift Metrics Installation Failed because Heapster and Hawkular were not starting]]
* [[OpenShift Nexus Pod failed to Start During the CI/CD Pipeline Deployment]]
* [[OpenShift Nexus Kept Getting Restarted]]
* [[OpenShift Nexus and Gogs not Accessible at their Public HTTPS URL]]
* [[OpenShift Jenkins fails to Communicate with Maven slaves]]
* [[OpenShift Gogs Fails to Create Required Postgresql Database]]

Latest revision as of 19:55, 18 November 2020

Internal

Overview

The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here:

Change the Log Level for OpenShift Processes

--loglevel=10 seems to work fine.

Then tail the journalctl log. More details on getting logging information:

OpenShift Master and Node Process Logging

General Troubleshooting

Troubleshooting OpenShift Container Platform: Basics:

https://access.redhat.com/solutions/1542293

Installation Troubleshooting

Installation Troubleshooting

Master/Node Processes Troubleshooting

OpenShift Runtime

Troubleshooting Pods

oc -n <project-name> get pods
oc -n <project-name> describe po/<pod-name>

...
Name:			logging-fluentd-3kz30
...
Node:			node2/
...
Status:			Failed
Reason:			MatchNodeSelector
Message:		Pod Predicate MatchNodeSelector failed

In-line pod logs:

oc logs -f <pod-name>

Connect to a pod:

oc rsh <pod-name>

Connect to a Pod as Root

Log into the physical node running the pod. The pod - node association can be inferred executing

oc describe <pod-name> 

and extracting the "Node:" value.

On the node, run

docker ps

and identify the container ID.

Then

 docker exec -u 0 -it 3152a2509d92 /bin/bash

For more details on docker exec, see:

docker exec

As root, various diagnostic utilities can be temporarily installed:

yum install bind-utils

Copy Files into Pods

For more details on oc cp:

oc cp

Troubleshooting Projects

Troubleshooting Routes

Troubleshooting Logging

 oadm diagnostics AggregatedLogging

Troubleshooting Kibana

Troubleshooting Kibana in OpenShift

Troubleshooting Metrics

oadm diagnostics MetricsApiProxy
OpenShift Metrics Concepts

Cases