From: David R Antoch (dantoch@csc.com)
Date: Wed Sep 14 2005 - 10:50:39 EDT
Admins,
I have a filesystem (VxFS/LVM) that contains a directory with 600,000+
files (avg ~ 20K each... some are larger). Architecturally, an
applicaton no-no (but that's another issue...). As a side project,
we're evaluating a search tool that will search through the files, and
index them into a database. Now, management does not want to risk
evaluating the search on the production system, so I'm attempting to copy
the entire filesystem (51GB, they wanted to search it all) to a
development system. The target disks are new. Both machines are 11.0
patched to recent (within a few weeks) patch versions.
The copy (I've tried ssh and remsh|tar pipe, as well as NFS
find |cpio) gets a to a certain point, then starts thrashing the
target disk. iostat and sar say there's 1.7MB/sec continuous disk IO,
and I see 200+ seeks/second (seems way too much), and only about 1
file (20K) per 10 or 15 seconds gets copied. The issue is definitely
due to the directory size, as anything written into that dir, just grinds
to a crawl.
I used the VXFS defaults when building the filesystem. I was aware of
issues like decreasing the bytes per inode etc...for many small files,
but looking into the VXFS options, I really didnt come up with
anything other than the defaults. (an oversight?, what am I missing?)
Is there anything I can do for a VXFS filesystem, that would render
better performance, when writing large numbers of files into one directory?
Thx,
Dave
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
--
---> Please post QUESTIONS and SUMMARIES only!! <---
To subscribe/unsubscribe to this list, contact majordomo@dutchworks.nl
Name: hpux-admin@dutchworks.nl Owner: owner-hpux-admin@dutchworks.nl
Archives: ftp.dutchworks.nl:/pub/digests/hpux-admin (FTP, browse only)
http://www.dutchworks.nl/htbin/hpsysadmin (Web, browse & search)
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 11:02:49 EDT