From: Friedman, Jeff (Jeff.Friedman@nordstrom.com)
Date: Thu Jul 07 2005 - 18:50:46 EDT
Hello all,
Environment specifics:
Sunfire 6800 domain running Solaris 9 09/04 Generic_118558-04
Two Emulex LP9002 PCI HBAs in one IO boat installed in 33MHz PCI slots.
2G switched fabric SAN (Cisco MDS 9509) connection to EMC DMX 2000.
8 98G LUNs, two paths to each LUN.
lpfc driver level 6.01f, microcode level 3.92a2
Problem:
Intermittently we see iostat report "wait" values of anywhere from 1 - 100,
and the "wsvc_t" also ranges from 1 - a few hundred, over 1000 on a couple
occasions. This started to occur only on on path at first, but now we see it
on both. Example iostat below. EMC has checked the DMX, and found no
bottlenecks or errors. The problem does escalate as IO grows, but even occurs
during periods of lesser IO volumes. What I am wondering is how I can have
lun queue lengths this high when I have "set sd:sd_max_throttle=20" in
/etc/system, and "lun-queue-depth=20" in lpfc.conf? What device driver is
getting bogged down? sd? lpfc? So far Sun Support says they have never seen
this type of behavior. That does not help too much!
Thanks in advance. - Jeff Friedman
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.3 1.3 2.2 3.5 0.0 0.0 10.6 15.6 2 1 d10
0.1 1.3 1.1 3.5 0.0 0.0 0.0 15.4 0 1 d11
0.1 1.3 1.1 3.5 0.0 0.0 0.0 15.3 0 1 d12
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 d20
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 d21
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 d22
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.1 2.0 1.1 3.9 0.0 0.0 0.0 17.8 0 2 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.1 2.0 1.1 3.9 0.0 0.0 0.0 18.2 0 2 c1t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t8d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t9d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t8d1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t8d2
255.9 34.2 2047.1 320.8 4.9 3.5 16.9 12.2 12 63 c2t8d16
236.4 33.5 1891.3 315.0 4.3 3.5 16.1 12.9 11 63 c2t8d17
237.0 32.8 1895.7 310.3 3.5 3.3 12.9 12.4 11 62 c2t8d18
235.2 31.8 1881.7 303.2 3.4 3.3 12.9 12.4 11 61 c2t8d19
235.2 32.3 1881.3 305.8 3.9 3.4 14.5 12.9 11 63 c2t8d20
234.8 33.7 1878.2 315.0 3.5 3.4 13.1 12.7 11 63 c2t8d21
231.3 34.0 1850.1 320.6 3.7 3.4 13.8 12.8 11 62 c2t8d22
244.1 33.7 1952.5 316.9 3.9 3.3 13.9 12.0 11 61 c2t8d23
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t25d1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t25d2
210.4 35.9 1683.3 336.0 130.9 1.0 531.6 3.9 95 96 c3t25d16
219.5 34.4 1755.9 324.4 114.9 1.0 452.5 3.8 93 96 c3t25d17
222.8 34.8 1782.4 327.0 120.8 1.0 469.0 3.7 95 96 c3t25d18
226.1 33.2 1809.2 314.8 119.8 1.0 461.9 3.7 93 95 c3t25d19
231.7 34.8 1853.7 327.8 302.7 1.0 1135.8 3.7 99 99 c3t25d20
223.5 33.7 1787.7 318.5 171.6 1.0 667.3 3.8 96 97 c3t25d21
226.1 33.8 1808.7 319.8 119.8 1.0 460.8 3.7 95 97 c3t25d22
225.9 33.8 1806.9 319.5 71.8 0.9 276.4 3.5 87 91 c3t25d23
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t25d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:31:04 EDT