Oracle Internals Notes
Delayed Write Backlogs
It is possible, merely by copying a few large files simultaneously using several processes,
to flood the file system cache with delayed write buffers.
Shortly afterwards, the operating system will attempt to flush those buffers to disk.
This can create an enormous queue in the device driver or logical volume manager
of maybe tens of thousands of write requests against a single disk.
Somewhere in that queue there may also be a few write requests to update the file system metadata, such as the space allocation maps.
These requests wait their turn in the queue, and while they wait, locks are held on the file system preventing all file system access.
When this happens the entire system may appear to freeze for several seconds or even longer, particularly on large memory systems.
CPU usage falls quickly as most processes block on I/O requests, and only the delayed write target disks remain busy.
Delayed write backlogs seem to be much more severe on some file system types than others.
The following simple test can be used to check whether a particular file system is prone to this problem, or not.
Write a shell script that just does
while :
do
date >> date.txt
sleep 1
done &
|
and then copy a few large files, like some archived log files, into the same file system simultaneously using several background processes.
If you hit a delayed write backlog, you will see that the date shell script failed to run at all for several seconds or longer.
In some cases, this little test can produce a delay of more than a minute.
Oracle itself does not use delayed writes.
Oracle opens all database files explicitly for synchronous write completion.
Nevertheless, Oracle is susceptible to delayed write backlogs if any of its I/O is file system based -- either buffered or direct.
Under several operating systems, including HP-UX and AIX,
it is possible to tune the kernel to limit to number of pending delayed writes per process.
This reduces the risk, but does not remove it entirely.
However, delayed write backlogs do not impact raw I/O, so there is no risk to raw databases.
