OpenShift Troubleshooting: Difference between revisions
Jump to navigation
Jump to search
| (49 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=Internal= | =Internal= | ||
* [[OpenShift#Subjects|OpenShift]] | * [[OpenShift TODEPLETE#Subjects|OpenShift TODEPLETE]] | ||
=Overview= | =Overview= | ||
The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here | The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here: | ||
[[ | {{Internal|OpenShift_Logging_Levels#Change_the_Log_Level_for_OpenShift_Processes|Change the Log Level for OpenShift Processes}} | ||
--loglevel=10 seems to work fine. | |||
Then tail the journalctl log. More details on getting logging information: | |||
{{Internal|OpenShift_Logging#Master_and_Node_Process_Logging|OpenShift Master and Node Process Logging}} | |||
=General Troubleshooting= | |||
Troubleshooting OpenShift Container Platform: Basics: | |||
{{External|https://access.redhat.com/solutions/1542293}} | |||
=Installation Troubleshooting= | |||
{{Internal|OpenShift Installation#Installation_Troubleshooting|Installation Troubleshooting}} | |||
=Master/Node Processes Troubleshooting= | |||
{{External|OpenShift Runtime|OpenShift Runtime}} | |||
=Troubleshooting Pods= | |||
[[oc get#pods|oc -n <project-name> get pods]] | |||
[[oc describe#pod|oc -n <project-name> describe po/<pod-name>]] | |||
... | |||
<font color=red>Name</font>: logging-fluentd-3kz30 | |||
... | |||
<font color=red>Node</font>: node2/ | |||
... | |||
<font color=red>Status</font>: Failed | |||
Reason: MatchNodeSelector | |||
Message: Pod Predicate MatchNodeSelector failed | |||
In-line pod logs: | |||
[[Oc_logs#-f|oc logs]] -f <''pod-name''> | |||
Connect to a pod: | |||
[[oc rsh]] <''pod-name''> | |||
==Connect to a Pod as Root== | |||
Log into the physical node running the pod. The pod - node association can be inferred executing | |||
oc describe <''pod-name''> | |||
and extracting the "Node:" value. | |||
On the node, run | |||
docker ps | |||
and identify the container ID. | |||
Then | |||
docker exec -u 0 -it 3152a2509d92 /bin/bash | |||
For more details on docker exec, see: | |||
{{Internal|docker exec|docker exec}} | |||
As root, various diagnostic utilities can be temporarily installed: | |||
yum install bind-utils | |||
==Copy Files into Pods== | |||
For more details on oc cp: | |||
{{Internal|oc cp|oc cp}} | |||
=Troubleshooting Projects= | |||
* [[OpenShift_Project_Operations#Monitor_Project.27s_Event_Log|Monitor project's event log]] | |||
=Troubleshooting Routes= | |||
=Troubleshooting Logging= | |||
oadm diagnostics AggregatedLogging | |||
==Troubleshooting Kibana== | |||
{{Internal|Kibana and OpenShift#Troubleshooting|Troubleshooting Kibana in OpenShift}} | |||
=Troubleshooting Metrics= | |||
oadm diagnostics MetricsApiProxy | |||
{{Internal|OpenShift Metrics Concepts|OpenShift Metrics Concepts}} | |||
=Cases= | |||
* [[OpenShift Kibana Started to Redirect to master1 instead of Public Master URL after Ansible Update]] | |||
* [[OpenShift ElasticSearch pod fails to start with "Unable to load index mapping ..."]] | |||
* [[OpenShift Metrics Installation Failed because Heapster and Hawkular were not starting]] | |||
* [[OpenShift Nexus Pod failed to Start During the CI/CD Pipeline Deployment]] | |||
* [[OpenShift Nexus Kept Getting Restarted]] | |||
* [[OpenShift Nexus and Gogs not Accessible at their Public HTTPS URL]] | |||
* [[OpenShift Jenkins fails to Communicate with Maven slaves]] | |||
* [[OpenShift Gogs Fails to Create Required Postgresql Database]] | |||
Latest revision as of 19:55, 18 November 2020
Internal
Overview
The general technique is to increase the logging level of various OpenShift master (api, controllers) and node processes as described here:
--loglevel=10 seems to work fine.
Then tail the journalctl log. More details on getting logging information:
General Troubleshooting
Troubleshooting OpenShift Container Platform: Basics:
Installation Troubleshooting
Master/Node Processes Troubleshooting
- OpenShift Runtime
Troubleshooting Pods
oc -n <project-name> get pods
oc -n <project-name> describe po/<pod-name> ... Name: logging-fluentd-3kz30 ... Node: node2/ ... Status: Failed Reason: MatchNodeSelector Message: Pod Predicate MatchNodeSelector failed
In-line pod logs:
oc logs -f <pod-name>
Connect to a pod:
oc rsh <pod-name>
Connect to a Pod as Root
Log into the physical node running the pod. The pod - node association can be inferred executing
oc describe <pod-name>
and extracting the "Node:" value.
On the node, run
docker ps
and identify the container ID.
Then
docker exec -u 0 -it 3152a2509d92 /bin/bash
For more details on docker exec, see:
As root, various diagnostic utilities can be temporarily installed:
yum install bind-utils
Copy Files into Pods
For more details on oc cp:
Troubleshooting Projects
Troubleshooting Routes
Troubleshooting Logging
oadm diagnostics AggregatedLogging
Troubleshooting Kibana
Troubleshooting Metrics
oadm diagnostics MetricsApiProxy
Cases
- OpenShift Kibana Started to Redirect to master1 instead of Public Master URL after Ansible Update
- OpenShift ElasticSearch pod fails to start with "Unable to load index mapping ..."
- OpenShift Metrics Installation Failed because Heapster and Hawkular were not starting
- OpenShift Nexus Pod failed to Start During the CI/CD Pipeline Deployment
- OpenShift Nexus Kept Getting Restarted
- OpenShift Nexus and Gogs not Accessible at their Public HTTPS URL
- OpenShift Jenkins fails to Communicate with Maven slaves
- OpenShift Gogs Fails to Create Required Postgresql Database