NFS挂载超时排查
问题
/etc/fstab的NFS配置
nfs.example.com:/vol/data /mnt/data nfs rw,soft,intr,vers=3,proto=tcp,rsize=32768,wsize=32768 0 0挂载超时
#mount /mnt/data
mount.nfs: Connection timed out对于NFS v3挂载，默认采用的是UDP协议，所以如果挂载参数中没有指定
proto=tcp则客户端访问采用UDP协议。通常防火墙开启了TCP允许访问，但是UDP端口往往关闭，所以排查时候需要特别注意
排查
这里NFS挂载是TCP协议，所以检查端口
检查NFS端口
[root@nfs-client /root]
#telnet nfs.example.com 2049
Trying 192.168.1.53...
Connected to nfs.example.com.
Escape character is '^]'.
[root@nfs-client /root]
#telnet nfs.example.com 111
Trying 192.168.1.53...
Connected to nfs.example.com.
Escape character is '^]'.看起来端口已经打开
通过verbose方式挂载存储查看:
#mount -v -o tcp,mountproto=tcp,nfsvers=3 -t nfs nfs.example.com:/vol/data /mnt/data
mount.nfs: timeout set for Mon Sep 21 11:08:44 2020
mount.nfs: trying text-based options 'tcp,vers=3,addr=192.168.1.53'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100005 vers 3 prot TCP port 4046
mount.nfs: mount(2): Connection timed out
mount.nfs: trying text-based options 'tcp,vers=3,addr=192.168.1.53'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100005 vers 3 prot TCP port 4046
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed out看起来像是端口 4096 无法打开，但是实际上检查端口也是通的
[root@nfs-client /root]
#telnet nfs.example.com 4046
Trying 192.168.1.53...
Connected to nfs.example.com.
Escape character is '^]'.对比测试了正常的 nfs-client-good
[root@nfs-client-good /etc]
#cat /etc/fstab
nfs.example.com:/vol/data /mnt/data nfs rw,soft,intr,vers=3,proto=tcp,rsize=32768,wsize=32768 0 0
[root@nfs-client-good /etc]
#mount /mnt/data
[root@nfs-client-good /etc]
#df -h
Filesystem Size Used Avail Use% Mounted on
/ 50G 1002M 50G 2% /
/dev/v01d 5.0G 36K 5.0G 1% /dev/shm
/dev/v02d 50G 1002M 50G 2% /home/admin/conf
/dev/v03d 50G 1002M 50G 2% /home/admin/logs
/dev/v04d 50G 1002M 50G 2% /home/staragent/plugins
shm 5.0G 36K 5.0G 1% /dev/shm
nfs.example.com:/vol/data
2.0T 1.6T 436G 79% /mnt/data使用debug模式检查完全没有问题
[root@nfs-client-good /root]
#mount -v -o tcp,mountproto=tcp,nfsvers=3 -t nfs nfs.example.com:/vol/data /mnt/data
mount.nfs: timeout set for Mon Sep 21 03:32:11 2020
mount.nfs: trying text-based options 'tcp,mountproto=tcp,nfsvers=3,addr=192.168.1.53'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.53 prog 100005 vers 3 prot TCP port 4046
[root@nfs-client-good /root]
#df -h
Filesystem Size Used Avail Use% Mounted on
/ 50G 1003M 50G 2% /
/dev/v01d 5.0G 36K 5.0G 1% /dev/shm
/dev/v02d 50G 1003M 50G 2% /home/admin/conf
/dev/v03d 50G 1003M 50G 2% /home/admin/logs
/dev/v04d 50G 1003M 50G 2% /home/staragent/plugins
shm 5.0G 36K 5.0G 1% /dev/shm
nfs.example.com:/vol/data
2.0T 1.6T 436G 79% /mnt/data检查从客户端访问服务器的mountpoint
#showmount -e nfs.example.com
Export list for nfs.example.com:
...
/vol/data 192.0.0.0/8,10.0.0.0/8
...对比检查，我们的异常服务器IP地址是 192.168.1.187/20 已经包含在 /vol/data 的访问允许IP地址段 192.0.0.0/8
检查服务器上的服务:
[root@nfs-client /root]
#rpcinfo -p nfs.example.com
program vers proto port service
100011 1 udp 4049 rquotad
100024 1 tcp 4047 status
100024 1 udp 4047 status
100021 4 tcp 4045 nlockmgr
100021 3 tcp 4045 nlockmgr
100021 1 tcp 4045 nlockmgr
100021 4 udp 4045 nlockmgr
100021 3 udp 4045 nlockmgr
100021 1 udp 4045 nlockmgr
100005 3 tcp 4046 mountd
100003 3 tcp 2049 nfs
100005 2 tcp 4046 mountd
100005 1 tcp 4046 mountd
100003 2 tcp 2049 nfs
100005 3 udp 4046 mountd
100003 3 udp 2049 nfs
100005 2 udp 4046 mountd
100005 1 udp 4046 mountd
100003 2 udp 2049 nfs
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper换一种挂载协议尝试:
mount -v -o udp,mountproto=udp,nfsvers=3 -t nfs nfs.example.com:/vol/data /mnt/data同样也出现访问 4046 端口超时
mount.nfs: timeout set for Mon Sep 21 11:27:27 2020
mount.nfs: trying text-based options 'udp,mountproto=udp,nfsvers=3,addr=192.168.1.53'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 192.168.1.53 prog 100003 vers 3 prot UDP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.1.53 prog 100005 vers 3 prot UDP port 4046
mount.nfs: mount(2): Connection timed out
...不过，根据 telnet 192.168.1.53 4046 可以看到NAS服务器端口是可以访问的，后来根据tcpdump查看，客户端发出端口4046的SYN请求后，NAS服务器端没有返回对应的ACK。
实际上这个抓包对比过程比较曲折，我花费了几天时间才找出导致这个NFS无法挂载的真正原因，涉及的问题和内核相关，使我对NFS以及网络问题排查有了一些心得，我会重新写几篇文档来总结。
tcpdump抓包
使用以下命令抓包
tcpdump -i eth0 -n host 192.168.1.53
tcpdump -i eth0 -nn -vv -e host 192.168.1.53显示信息
16:03:24.164520 IP 192.168.0.187.33763 > 192.168.1.53.acp-proto: Flags [.], ack 30, win 502, options [nop,nop,TS val 1341343592 ecr 766772696], length 0
16:03:24.164838 IP 192.168.0.187.1020 > 192.168.1.53.acp-proto: Flags [S], seq 2722685289, win 64240, options [mss 1460,sackOK,TS val 1341343592 ecr 0,nop,wscale 7], length 0
16:03:25.184372 IP 192.168.0.187.1020 > 192.168.1.53.acp-proto: Flags [S], seq 2722685289, win 64240, options [mss 1460,sackOK,TS val 1341344612 ecr 0,nop,wscale 7], length 0
16:03:27.232376 IP 192.168.0.187.1020 > 192.168.1.53.acp-proto: Flags [S], seq 2722685289, win 64240, options [mss 1460,sackOK,TS val 1341346660 ecr 0,nop,wscale 7], length 0
...抓包对比
tcpdump -w /tmp/nfs.pcap -nni eth0 host 192.168.1.53在wireshark中查看，可以看到 192.168.0.187.33763 > 192.168.1.53.acp-proto 第一次发出 [S](Syn) 之后，NAS服务器端没有任何ACK包，所以后面都是重复的 [S](Syn)
参考
Last updated