From: NKS (sunadmin@norsar.no)
Date: Thu Jan 11 2007 - 12:42:34 EST
Dear all,
I have a SunFire 280R Solaris 9, 2xCPU, 4GB RAM, fiber HBA
connected to a Nexsan Atabeast with 14x300GB disks. Nexsan
disks organized in 8 identical volumes, RAID5. Veritas VM
v.4 used on SUn-server.
Even though the I/O activity is low, the server reports
20-80% of the CPU-time is iowait...
The Atabeast controller is idle; nothing special going on.
I am really stuck here; ideas anyone??
Thanks!
Best regards,
Nils
Below: The output of various commands
munin# uname -a
SunOS munin 5.9 Generic_112233-12 sun4u sparc SUNW,Sun-Fire-280R
munin# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:sliced rootdisk rootdg online
c1t1d0s2 auto:sliced rootmirror rootdg online
c5t0d0s2 auto:cdsdisk NORSARdg01 NORSARdg online
c5t0d1s2 auto:cdsdisk NORSARdg02 NORSARdg online
c5t0d2s2 auto:cdsdisk NORSARdg03 NORSARdg online
c5t0d3s2 auto:cdsdisk NORSARdg04 NORSARdg online
c5t0d4s2 auto:cdsdisk NORSARdg05 NORSARdg online
c5t0d5s2 auto:cdsdisk NORSARdg06 NORSARdg online
c5t0d6s2 auto:cdsdisk NORSARdg07 NORSARdg online
c5t0d7s2 auto:cdsdisk NORSARdg08 NORSARdg online
munin# iostat -xn 5
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.1 0.0 0.0 0.0 0.0 5.0 0 0 c0t6d0
1.4 334.9 6.8 1224.5 0.0 1.0 0.0 2.9 0 22 c5t0d4
1.1 1.9 5.3 8.3 0.0 0.0 0.0 10.3 0 2 c5t0d3
0.9 1.8 4.1 8.1 0.0 0.0 0.0 8.9 0 2 c5t0d2
1.0 1.7 5.2 8.6 0.0 0.0 0.0 12.5 0 2 c5t0d1
1.9 2.2 5.5 9.5 0.0 0.0 0.0 7.1 0 2 c5t0d0
1.4 302.3 6.7 1196.2 0.0 1.0 0.0 3.2 0 20 c5t0d7
1.3 381.4 6.2 1279.5 0.0 1.0 0.0 2.7 0 23 c5t0d6
1.3 318.2 6.6 1215.7 0.0 1.2 0.0 3.8 0 21 c5t0d5
4.7 3.0 30.5 23.7 0.0 0.1 3.2 6.8 0 3 c1t0d0
1.8 1.1 30.5 8.0 0.0 0.0 7.8 11.3 0 1 c1t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 munin:vold(pid831)
(c5t0 beeing the Nexsan volumes, the four disks with i/o are on
a single striped vxfs volume)
munin# top
ast pid: 1495; load averages: 0.77, 0.80, 0.70
18:27:47
37 processes: 36 sleeping, 1 on cpu
CPU states: 44.2% idle, 0.0% user, 32.2% kernel, 23.6% iowait, 0.0% swap
Memory: 4096M real, 3550M free, 79M swap in use, 13G swap free
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
791 root 19 59 0 2520K 2096K sleep 11:21 18.79% nfsd
1495 root 1 59 0 2784K 1712K cpu/0 0:00 0.52% top
1400 root 1 59 0 1480K 1312K sleep 0:00 0.01% csh
388 root 17 59 0 65M 36M sleep 0:03 0.00% vxsvc
14 root 1 60 0 9456K 7952K sleep 0:02 0.00% vxconfigd
314 root 9 59 0 3880K 3424K sleep 0:00 0.00% picld
531 root 17 59 0 5192K 3256K sleep 0:00 0.00% syslogd
470 daemon 2 59 0 5448K 2896K sleep 0:00 0.00% statd
1394 root 1 59 0 4528K 2880K sleep 0:00 0.00% sshd
776 root 2 59 0 3864K 2648K sleep 0:00 0.00% mountd
556 root 24 59 0 3136K 2552K sleep 0:00 0.00% nscd
831 root 3 59 0 2960K 2552K sleep 0:00 0.00% vold
473 root 2 59 0 3728K 2328K sleep 0:00 0.00% automountd
593 root 1 59 0 4432K 2064K sleep 0:00 0.00% sendmail
449 root 1 59 0 2576K 1992K sleep 0:00 0.00% inetd
munin# sar 5 10
SunOS munin 5.9 Generic_112233-12 sun4u 01/11/2007
18:32:51 %usr %sys %wio %idle
18:32:56 0 32 21 47
18:33:01 0 33 15 52
18:33:06 0 31 27 42
18:33:11 0 31 20 49
18:33:16 0 31 17 52
18:33:21 0 27 29 44
18:33:26 0 29 16 55
18:33:31 0 31 26 44
18:33:36 0 35 20 44
18:33:41 0 28 31 41
Average 0 31 22 47
munin# df -h
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/bootdg/rootvol
940M 170M 713M 20% /
/dev/vx/dsk/bootdg/usr
4.8G 1.4G 3.3G 31% /usr
/proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
fd 0K 0K 0K 0% /dev/fd
/dev/vx/dsk/bootdg/var
7.7G 172M 7.4G 3% /var
swap 13G 40K 13G 1% /var/run
dmpfs 13G 0K 13G 0% /dev/vx/dmp
dmpfs 13G 0K 13G 0% /dev/vx/rdmp
swap 13G 0K 13G 0% /tmp
/dev/vx/dsk/bootdg/opt
9.6G 1.0G 8.5G 11% /opt
/dev/vx/dsk/NORSARdg/common
100G 42G 55G 44% /common
/dev/vx/dsk/NORSARdg/guest
100G 85G 14G 87% /guest
/dev/vx/dsk/NORSARdg/ndc
250G 220G 28G 89% /ndc
/dev/vx/dsk/NORSARdg/staff
250G 240G 9.0G 97% /staff
/dev/vx/dsk/NORSARdg/smod
250G 195G 51G 80% /smod
/dev/vx/dsk/NORSARdg/seisproj
140G 91G 46G 67% /seisproj
/dev/vx/dsk/NORSARdg/seismo
500G 239G 245G 50% /seismo
munin# vxprint -rth -g NORSARdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg NORSARdg default default 6000 1110466703.125.munin
dm NORSARdg01 c5t0d0s2 auto 2048 1171781376 -
dm NORSARdg02 c5t0d1s2 auto 2048 1171781376 -
dm NORSARdg03 c5t0d2s2 auto 2048 1171781376 -
dm NORSARdg04 c5t0d3s2 auto 2048 1171781376 -
dm NORSARdg05 c5t0d4s2 auto 2048 1171781376 -
dm NORSARdg06 c5t0d5s2 auto 2048 1171781376 -
dm NORSARdg07 c5t0d6s2 auto 2048 1171781376 -
dm NORSARdg08 c5t0d7s2 auto 2048 1171781376 -
v common - ENABLED ACTIVE 209715200 SELECT common-01 fsgen
pl common-01 common ENABLED ACTIVE 209715200 STRIPE 4/128 RW
sd NORSARdg01-03 common-01 NORSARdg01 471859200 52428800 0/0 c5t0d0 ENA
sd NORSARdg02-03 common-01 NORSARdg02 471859200 52428800 1/0 c5t0d1 ENA
sd NORSARdg03-03 common-01 NORSARdg03 471859200 52428800 2/0 c5t0d2 ENA
sd NORSARdg04-03 common-01 NORSARdg04 471859200 52428800 3/0 c5t0d3 ENA
v guest - ENABLED ACTIVE 209715200 SELECT guest-01 fsgen
pl guest-01 guest ENABLED ACTIVE 209715200 STRIPE 4/128 RW
sd NORSARdg01-04 guest-01 NORSARdg01 524288000 52428800 0/0 c5t0d0 ENA
sd NORSARdg02-04 guest-01 NORSARdg02 524288000 52428800 1/0 c5t0d1 ENA
sd NORSARdg03-04 guest-01 NORSARdg03 524288000 52428800 2/0 c5t0d2 ENA
sd NORSARdg04-04 guest-01 NORSARdg04 524288000 52428800 3/0 c5t0d3 ENA
v ndc - ENABLED ACTIVE 1048576000 SELECT ndc-01 fsgen
pl ndc-01 ndc ENABLED ACTIVE 1048576000 STRIPE 4/128 RW
sd NORSARdg05-01 ndc-01 NORSARdg05 0 262144000 0/0 c5t0d4 ENA
sd NORSARdg06-01 ndc-01 NORSARdg06 0 262144000 1/0 c5t0d5 ENA
sd NORSARdg07-01 ndc-01 NORSARdg07 0 262144000 2/0 c5t0d6 ENA
sd NORSARdg08-01 ndc-01 NORSARdg08 0 262144000 3/0 c5t0d7 ENA
v seismo - ENABLED ACTIVE 1048576000 SELECT seismo-01 fsgen
pl seismo-01 seismo ENABLED ACTIVE 1048576000 STRIPE 4/128 RW
sd NORSARdg05-02 seismo-01 NORSARdg05 262144000 262144000 0/0 c5t0d4 ENA
sd NORSARdg06-02 seismo-01 NORSARdg06 262144000 262144000 1/0 c5t0d5 ENA
sd NORSARdg07-02 seismo-01 NORSARdg07 262144000 262144000 2/0 c5t0d6 ENA
sd NORSARdg08-02 seismo-01 NORSARdg08 262144000 262144000 3/0 c5t0d7 ENA
v seisproj - ENABLED ACTIVE 419430400 SELECT seisproj-01 fsgen
pl seisproj-01 seisproj ENABLED ACTIVE 419430400 STRIPE 4/128 RW
sd NORSARdg05-03 seisproj-01 NORSARdg05 524288000 104857600 0/0 c5t0d4 ENA
sd NORSARdg06-03 seisproj-01 NORSARdg06 524288000 104857600 1/0 c5t0d5 ENA
sd NORSARdg07-03 seisproj-01 NORSARdg07 524288000 104857600 2/0 c5t0d6 ENA
sd NORSARdg08-03 seisproj-01 NORSARdg08 524288000 104857600 3/0 c5t0d7 ENA
v smod - ENABLED ACTIVE 1048576000 SELECT smod-01 fsgen
pl smod-01 smod ENABLED ACTIVE 1048576000 STRIPE 4/128 RW
sd NORSARdg01-02 smod-01 NORSARdg01 209715200 262144000 0/0 c5t0d0 ENA
sd NORSARdg02-02 smod-01 NORSARdg02 209715200 262144000 1/0 c5t0d1 ENA
sd NORSARdg03-02 smod-01 NORSARdg03 209715200 262144000 2/0 c5t0d2 ENA
sd NORSARdg04-02 smod-01 NORSARdg04 209715200 262144000 3/0 c5t0d3 ENA
v staff - ENABLED ACTIVE 838860800 SELECT staff-01 fsgen
pl staff-01 staff ENABLED ACTIVE 838860800 STRIPE 4/128 RW
sd NORSARdg01-01 staff-01 NORSARdg01 0 209715200 0/0 c5t0d0 ENA
sd NORSARdg02-01 staff-01 NORSARdg02 0 209715200 1/0 c5t0d1 ENA
sd NORSARdg03-01 staff-01 NORSARdg03 0 209715200 2/0 c5t0d2 ENA
sd NORSARdg04-01 staff-01 NORSARdg04 0 209715200 3/0 c5t0d3 ENA
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:41:27 EDT