High wait time on a disk

From: Brewer, Edward (BREWERE@OD.NIH.GOV)
Date: Thu Jan 30 2003 - 10:11:07 EST


Admins,

Has anyone ever experienced this...???

I have a disk that contains oracle data files (not index, redo,temp or
archive) for an 8i database.

Information:

We are using Tru64 5.1a with patch kit 3 and the patch for direct I/O.
We are using direct I/O.
The machine is a GS160 with 2 QBB 8 CPU.
We have multiple connections from each QBB to 4 switches and 2 pairs of
HSG80 controllers.
The controllers are in multi-path failover mode with a 45 second flush_cache
timer.
The drive in question is a stripped mirror set, three mirror pairs in a
stripe set.
All of the cache option are chosen for the unit.
The max read and write ahead is set at 256.
The drive in question is the only volume in the domain with one fileset.

This disk during the day (OLTP) traffic is performing well.

Right before the beginning of the work day, I get 2 spikes in performance
from the disk.

Spike 1:

write transactions per second jumps to 1200
read transactions per second hover around 0
read throughput per second jumps to 45000 KB/sec
average service time hovers around 10 milliseconds
average wait time jumps to 2200 millisecond

Spike 2:

write transactions per second jumps to 600
read transactions per second hovers around 0
read throughput per second jumps to 28000 KB/sec
average service time jumps to 250 milliseconds
average wait jumps to 9900 milliseconds.

I have asked the DBA's to examine if any jobs kick off at this time that may
perform massive DML.

My questions:

How can I attribute the poor service performance of the disk during the
second spike? Can it be possible, since it is being address through direct
I/O, that the performance problem is related to oracle. The service
performance during the first peak was acceptable, but the request was high;
thus I think this accounts for the large wait time. But by increasing the
service time by a factor 8-10 on the second the wait time was even more
miserable. Has anyone seen this before?

TIA,
Lee Brewer



This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:49:06 EDT