Linux Network Troubleshooting
Internal
Overview
This page needs to be reviewed and re-organized.
Organizatorium
- TCP zero windows sent when application receive buffer is empty https://access.redhat.com/solutions/2480661
Network Monitoring
On each node, run the monitor.sh script: https://access.redhat.com/articles/1311173. This script will record OS network stats at a set interval and it will allow monitoring changes over time and correlate these changes with packet capture data.
Network Driver Error Messages
grep vmxnet3 sos_commands/kernel/dmesg [ 5.731844] VMware vmxnet3 virtual NIC driver - version 1.1.30.0-k-NAPI [ 5.731858] vmxnet3 0000:0b:00.0: # of Tx queues : 4, # of Rx queues : 4 [ 5.737730] vmxnet3 0000:0b:00.0: irq 72 for MSI/MSI-X [ 5.737786] vmxnet3 0000:0b:00.0: irq 73 for MSI/MSI-X [ 5.737860] vmxnet3 0000:0b:00.0: irq 74 for MSI/MSI-X [ 5.737891] vmxnet3 0000:0b:00.0: irq 75 for MSI/MSI-X [ 5.737916] vmxnet3 0000:0b:00.0: irq 76 for MSI/MSI-X [ 5.738367] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps [ 8.186233] vmxnet3 0000:0b:00.0 eno16780032: intr type 3, mode 0, 5 vectors allocated [ 8.187854] vmxnet3 0000:0b:00.0 eno16780032: NIC Link is Up 10000 Mbps
Kernel Network Paramenters
cat etc/sysctl.conf
net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_redirects=0 net.ipv4.conf.default.accept_redirects=0 net.ipv4.conf.all.log_martians=1 net.ipv4.conf.default.log_martians=1 net.core.wmem_max = 12582912 net.core.rmem_max = 26214400 net.ipv4.tcp_rmem = 10240 87380 26214400 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_no_metrics_save = 1 net.core.netdev_max_backlog = 5000 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv6.conf.all.disable_ipv6 = 1 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 fs.suid_dumpable = 0
Also see
Inspect packet loss
ethtool -S
awk '($NF !~ "^0$") {print}' sos_commands/networking/ethtool_-S_eno16780032 | egrep -v "[u,m,b]cast|LRO pkts rx|[LR,TS]O byte(s)?|[LR,TS]O pkts|pkts linearized"
NIC statistics:
Tx Queue#: 1
Tx Queue#: 2
Tx Queue#: 3
Rx Queue#: 1
pkts rx OOB: 45
drv dropped rx total: 29
err: 29
Rx Queue#: 2
Rx Queue#: 3
RX Drops
- ifconfig packet drops reported after upgrading to RHEL7 https://access.redhat.com/solutions/2073223
proc/net/dev
cat proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 727497676 2498032 0 0 0 0 0 0 727497676 2498032 0 0 0 0 0 0
eno16780032: 216193874050 702019404 0 2658277 0 0 0 87265004 195315249141 549883330 0 0 0 0 0 0
IP and TCP Diagnostics
Review the IP, TCP OS protocol handler stats.
Check whether IP fragmentation is occurring. This is normal behaviour when an application sends a datagram which exceeds the MTU (1500).
Check number of failures due to fragment loss. TCP divides data into MSS-sized segments which should not require IP fragmentation so fragmentation is likely caused by UDP traffic.
netstat -s
Ip: 702329554 total packets received 0 forwarded 0 incoming packets discarded 699283912 incoming packets delivered 550941269 requests sent out 16 dropped because of missing route 5 fragments dropped after timeout 4372810 reassemblies required 1336065 packets reassembled ok 7 packet reassembles failed 618990 fragments received ok 1856970 fragments created
Check the rate of TCP retransmissions. A low rate is a sign that the network infrastructure is healthy. Problems with packet loss or high latency in the environment for any reason, reflects in a high rate of TCP retransmissions.
Check for socket buffer overflows.
Check for listen queue overflows.
netstat -s | egrep "pruned|collapsed|overflowed"
542 packets pruned from receive queue because of socket buffer overrun 4 packets pruned from receive queue 1197 packets collapsed in receive queue due to low socket buffer
Packet Capture and Analysis
TO Process:
tcpdump -s 0 -i eno16780032 -w /tmp/$HOSTNAME.pcap
TZ=EST5EDT tshark -t a -r test.pcap -Y "tcp.port == 46162" | tail -n 20
TZ=EST5EDT tshark -n -t ad -r test.pcap -Y "frame.number == 543875"
TZ=EST5EDT tshark -n -t ad -r test.pcap -T fields -e frame.number -e frame.time -e ip.src -e ip.dst -e tcp.stream -e col.Info -Y "frame.number == 543875"
TZ=EST5EDT tshark -n -t ad -r test.pcap -Y "tcp.stream == 20 && frame.number >= 543875 && data.len == 21"
TZ=EST5EDT tshark -n -t ad -r test.pcap -Y "frame.number >= 543875 && data.len == 21 && ip.src == 10.10.10.1 && ip.dst == 10.10.10.2" | wc -l
TZ=EST5EDT tshark -n -t ad -r test.pcap -Y "frame.number >= 543875 && data.len == 21 && ip.src == 10.10.10.2 && ip.dst == 10.10.10.1" | wc -l
TZ=EST5EDT tshark -n -t ad -r test.pcap -T fields -e tcp.stream -Y "frame.number >= 543875 && data.len == 21 && ip.src == 10.10.10.1 && ip.dst == 10.10.10.2" | sort -un | wc -l
The following two command lines show the tcp stream number (2nd column) and how many tcp packets with data length of 21 bytes there are on each. The 1st command line shows the communication from 10.10.10.1 to 10.10.10.2, and the 2nd command line shows the other direction:
TZ=EST5EDT tshark -n -t ad -r test.pcap -T fields -e tcp.stream -Y "frame.number >= 543875 && data.len == 21 && ip.src == 10.10.10.1 && ip.dst == 10.10.10.2" | sort -n | uniq --count
2 20
3 21
2 80
25 86
25 87
2 5634
2 5636
2 8030
14 8065
14 8162
TZ=EST5EDT tshark -n -t ad -r test.pcap -T fields -e tcp.stream -Y "frame.number >= 543875 && data.len == 21 && ip.src ==10.10.10.2 && ip.dst == 10.10.10.1" | sort -n | uniq --count
3 21
2 80
25 86
25 87
2 88
2 5634
2 5636
2 8030
15 8065
14 8162