filesystem grows out of control

From: Carl Ma (carl_ma@scotiacapital.com)
Date: Wed Jun 25 2003 - 17:52:46 EDT


Hello all,

We have one production server - E4500 with T3+ diskarray running solaris 8 with
latest patch level. It used to be very stable till today.

%df -k -F ufs
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d0 6191949 3293302 2836728 54% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 670536 0 670536 0% /var/run
swap 671664 1128 670536 1% /tmp
/dev/md/dsk/d1 400089861 200071534 200018327 50% /space
/dev/md/dsk/d2 200149000 20987800 179161200 10% /space/data1
/dev/md/dsk/d3 ... ... ... ... /space/data2
...
/dev/md/dsk/dN .. ... ... ... /space/dataN

We can see there are lots of local filesystems mounted under /space.Also
hundreds of files,directories and automount/nfs mount points are also under
/space directory.

Today, the /space usage grows continuously,slowly but constantly. It grows about
10% since noon,which is very abnormal.

I did the following check:

1."du -d -sk /space" every 5 seconds,which shows the usage is growing
2."du -d -sk /space/*" every 5 seconds,which gives me all the directories
capacity information including nfs.(-d doesn't work??)
3."find /space -xdev -type f -mtime -1 -ls | sort -n -k7", it does give me the
list of changed files on /space. There are a lot but none has big size.
4. failed to use tnfdump to analyze since this filesystem has lots of I/O. I am
lost at certain stage. :-)

My questions are:
1.may I just display the size of every directories under /space(d1),excluding
NFS and directories crossing filesytem boundaries(du -dsk /space/* doesn't
work)?
2.is it possible that someone hides data under mount points? how can I find it?

Right now, I am working on a dirty script and would like to hear from others. I
will summarize if solution is available.

thanks a lot,

carl
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:26:39 EDT