Friday, November 18, 2011

Solaris nfs throubleshooting

# showmount -a
     All mount points on local host:
     edcert20.ucs.indiana.edu:/home
     edcert21.ucs.indiana.edu:/usr/local

     # showmount -d
     Directories on local host:
     /home
     /usr/local

     # showmount -e
     Export list on local host
     /home           edcert21.ucs.indiana.edu edcert20.ucs.indiana.edu     
     /usr/local      edcert21.ucs.indiana.edu
# df -F nfs
     Filesystem                      Type  blocks     use   avail %use  Mounted on
     edcert21.ucs.indiana.edu:/home  nfs    68510   55804   12706  81%  /usr/share/help
Use the command nfsstat -s to display NFS activity on the server side. For example:
# nfsstat -s

     Server RPC:
     calls      badcalls   nullrecv   badlen     xdrcall    duphits    dupage
     50852      0          0          0          0          0          0.00    

     Server NFS:
     calls        badcalls     
     50852        0            
     null         getattr      setattr      root         lookup       readlink  
     1  0%        233  0%      0  0%        0  0%        1041  2%     0  0%  
     read         wrcache      write        create       remove       rename    
     49498 97%    0  0%        0  0%        0  0%        0  0%        0  0% 
     link         symlink      mkdir        rmdir        readdir      fsstat 
     0  0%        0  0%        0  0%        0  0%        75  0%       4  0%     
The output may be interpreted using the following guidelines.
  • badcalls > 0 - RPC requests are being rejected by the server. This could indicate authentication problems caused by having a user in too many groups, attempts to access exported file systems as root, or an improper Secure RPC configuration.
  • nullrecv > 0 - NFS requests are not arriving fast enough to keep all of the nfsd daemons busy. Reduce the number of NFS server daemons until nullrecv is not incremented.
  • symlink > 10% - Clients are making excessive use of symbolic links that are on file systems exported by the server. Replace the symbolic link with a directory, and mount both the underlying file system and the link's target on the client.
  • getattr > 60% - Check for non-default attribute caching (noac mount option) on NFS clients.
On the client side use the command nfsstat -c to display the client statistics. For example:
# nfsstat -c

     Client RPC:
     calls      badcalls   retrans    badxid     timeout    wait       newcred
     369003     62         1998       43         2053       0          0 

     Client NFS:
     calls        badcalls     nclget       nclsleep     
     368948       0            368948       0            
     null         getattr      setattr      root         lookup       readlink  
     0  0%        51732 14%    680  0%      0  0%        95069 25%    542  0% 
     read         wrcache      write        create       remove       rename 
     210187 56%   0  0%        2259  0%     1117  0%     805  0%      337  0%   
     link         symlink      mkdir        rmdir        readdir      fsstat    
     120  0%      0  0%        7  0%        0  0%        5510  1%     583  0% 
This output may be interpreted using the guidelines given below.
  • timeout > 5% - The client's RPC requests are timing out before the server can answer them, or the requests are not reaching the server. Check badxid to determine the problem.
  • badxid ~ timeout - RPC requests are being handled by the server, but too slowly. Increase timeo parameter value for this mount, or tune the server to reduce the average request service time.
  • badxid ~ 0 - With timeouts greater than 3%, this indicates that packets to and from the server are getting lost on the network. Reduce the read and write block sizes (mount parameters rsize andwsize) for this mount.
  • badxid > 0 - RPC calls on soft-mounted file systems are timing out. If the server is running, and badcalls is growing, then soft mounted file systems should use a larger timeo or retrans value.