From: Jarkko Airaksinen (JAiraksinen@zed.com)
Date: Tue Sep 26 2006 - 11:03:32 EDT
Dear Managers,
I have a pp450 running a sol8 with oracle 8i. The /opt disk has been
suffering from an I/O problem for a while. The system clearly feels
sluggish when I'm doing any kind of operation on /opt.
At only 30.5 iops and about 4M read per second it goes to 100% busy and
65ms service time:
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
30.0 0.5 3839.9 4.0 0.0 2.0 0.0 65.0 0 100 d5
I don't think that's too much for two sliced, striped and mirrored 72.6G
10krpm disks. The disks are:
0. c0t0d0 <FUJITSU-MAP3735NC-3701 cyl 24345 alt 2 hd 8 sec 737>
/pci@83,4000/FJSV,ulsa@2,1/sd@0,0
1. c1t0d0 <FUJITSU-MAP3735NC-3701 cyl 24345 alt 2 hd 8 sec 737>
/pci@83,4000/FJSV,ulsa@2/sd@0,0
None of the disks are 100% full, neither is /opt:
/dev/md/dsk/d5 57466826 41619254 15272904 74% /opt
The metastat shows that everything's fine, e.g. there's no resync or
anything special going on:
d5: Trans
State: Okay
Size: 116711320 blocks
Master Device: d85
Logging Device: d80
d85: Mirror
Submirror 0: d15
State: Okay
Submirror 1: d25
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default
Size: 116711320 blocks
d15: Submirror of d85
State: Okay
Size: 116711320 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s5 0 No Okay
d25: Submirror of d85
State: Okay
Size: 116711320 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s5 0 No Okay
d80: Logging device for d4 d5
State: Okay
Size: 265062 blocks
d80: Mirror
Submirror 0: d88
State: Okay
Submirror 1: d89
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 265320 blocks
d88: Submirror of d80
State: Okay
Size: 265320 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s3 0 No Okay
d89: Submirror of d80
State: Okay
Size: 265320 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s3 0 No Okay
Of course I'm quite sure that the guilty one is one of the Oracle
processes as seen in top:
load averages: 2.27, 2.14, 1.97
16:46:51
238 processes: 233 sleeping, 2 stopped, 3 on cpu
CPU states: 3.1% idle, 35.1% user, 20.2% kernel, 41.7% iowait, 0.0%
swap
Memory: 16G real, 3161M free, 9878M swap in use, 8200M swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
22428 ora817 23 60 0 485M 465M sleep 4:46 9.90% oracle
28984 ora817 1 59 0 457M 437M cpu/1 3:50 8.32% oracle
28966 ora817 15 0 0 521M 501M cpu/2 3:26 7.13% oracle
On the other hand although Oracle is installed on /opt/ it shouldn't
have many operations there because all the data/archive/log/etc files
are on FC disks. I've installed a few tools that have helped me to find
the exact process of the i/o heaviest operation but it's not much of
help as the highest i/o goes to the FC disks, not on /opt. Also I've
followed the disk read/write operations with some neat perl scripts and
orcallator and such but they haven't really been able to point out
what's causing the /opt/ to wait.
What procedures / tools would you recommend for finding out what eats my
/opt?
Br,
Jarkko Airaksinen
__________________________________________________________________________
La informacion incluida en el presente correo electronico es CONFIDENCIAL,
siendo para el uso exclusivo del/os destinatario/s arriba mencionado/s. Si
usted recibe y lee este correo electronico y no es el destinatario senalado,
el empleado o el agente responsable de entregar el mensaje al destinatario, o
ha recibido esta comunicacion por error, le informamos que esta totalmente
prohibida cualquier divulgacion, distribucion, uso o reproduccion del mismo, y
le rogamos que nos lo notifique inmediatamente respondiendo al mensaje
original a la direccion arriba mencionada y eliminando el mensaje a
continuacion.
The information contained in this e-mail is CONFIDENTIAL and is intended only
for the use of the addressee named above.If the reader of this message is not
the intended recipient or the employee or agent responsible for delivering the
message to the intended recipient, or you have received this communication in
error, please be aware that any diffusion, distribution or duplication of this
communication is strictly forbidden, and please notify us immediately by
return to the original message at the address above eliminating it
afterwards.
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:40:52 EDT