Help with NFS GETATTR Storms

From: Boyd, Chris (NIH/NLM) (boydc@mail.nlm.nih.gov)
Date: Sat Feb 21 2004 - 10:35:57 EST


Sun E4500 directly connected to an Network Appliance 820 via Gigabit Fiber.
No interface errors and everthing seems to be working nomally. I have alot
of experience with Solaris and Netapp but I cannot get past this problem.

The only thing running from the Netapp is an Oracle 8.0.5 database which
isn't "that" busy.

I experience GETATTR storms (far below)from the Sun box to the filer that
are causing many second delays in filer reponse.

I have other Solaris 7/8 systems directly connected the a filer with either
gigabit fiber or copper running Oracle databases. I cannot figure out what
is wrong or different enough to cause this problem.

Below you will find most relevant information regarding my system
configuration. tegument is the Solaris client and na4db10_4 is the Network
Appliance and it is the ge1 interface.

Any and all advice or questions are very much welcome.

Thanks, Chris

uname -a output:
SunOS tegument 5.8 Generic_108528-17 sun4u sparc SUNW,Ultra-Enterprise

Here is my vfstab entry:
na4db10_4:/vol/vol2/tegument/m4 - /m4 nfs - yes
rw,hard,intr,suid,vers=3,proto=udp,actimeo=600

nfsstat -m output:
/m4 from na4db10_4:/vol/vol0/tegument/m4
 Flags:
vers=3,proto=udp,sec=sys,hard,intr,link,symlink,rsize=32768,wsize=32768,retr
ans=5,timeo=11
 Attr cache: acregmin=600,acregmax=600,acdirmin=600,acdirmax=600
 Lookups: srtt=1 (2ms), dev=1 (5ms), cur=0 (0ms)
 Reads: srtt=8 (20ms), dev=3 (15ms), cur=2 (40ms)
 Writes: srtt=8 (20ms), dev=4 (20ms), cur=3 (60ms)
 All: srtt=8 (20ms), dev=4 (20ms), cur=3 (60ms)

netstat -i output:
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis
Queue
lo0 8232 loopback localhost 55654 0 55654 0 0 0

ge0 1500 tegument tegument 2005765 0 3843106 0 0 0

ge1 1500 tegument10_1 tegument10_1 1417269 0 311534 0 0 0

ge2 1500 tegument10_2 tegument10_2 41458 0 150378 0 0 0

hme1 1500 tegument17 tegument17 0 0 0 0 0 0

NFS Patch level:
Patch: 108727-26 Obsoletes: Requires: 108528-01 Incompatibles: Packages:
SUNWcarx, SUNWcsr, SUNWhea

This is an example getattr storm. Notice how many reuests in a very short
amount of time.

snoop -d ge1 -td output:
  0.00038 na4db10_4 -> tegument10_1 RPC R XID=3026953713 Success
  1.89096 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=3225
  0.00020 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00173 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=6B2D
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00046 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=3222
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00034 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=A779
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00416 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=EC6A
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00315 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=FEF4
  0.00013 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00123 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=D7E4
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00031 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=D7E4
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.01963 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=14D0
  0.00014 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00310 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=14D0
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00013 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=14D0
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00169 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00036 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00134 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00024 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00087 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00024 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00044 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00024 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00129 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00024 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00088 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00024 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=7746
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.01007 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=3225
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00152 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=6B2D
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00074 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=3222
  0.00012 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00034 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=A779
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00294 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=FEF4
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00095 tegument10_1 -> na4db10_4 NFS C LOOKUP3 FH=50C5 svr_14756.trc
  0.00013 na4db10_4 -> tegument10_1 NFS R LOOKUP3 No such file or
directory
  0.00063 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=D7E4
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.00031 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=D7E4
  0.00011 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
  0.05342 tegument10_1 -> na4db10_4 NFS C GETATTR3 FH=C87A
  0.00014 na4db10_4 -> tegument10_1 NFS R GETATTR3 OK
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:28:06 EDT