summary: filesystem grows out of control

From: Carl Ma (carl_ma@scotiacapital.com)
Date: Fri Jun 27 2003 - 11:45:34 EDT


Hi,

Thanks for the replies from:

Darren Dunham
Hichael Morton
Stephen Cawley
Alex Ugolini
Alan Pae

Finally, I have to bring the machine to single user mode with basic filesystem mounted. There
is one new, big file under one automount point. I still have no idea how such file created
since the consensus is it is impossible to create file under mounted filesystem.

thanks & have a good long weekend,

carl

:> Message: 10
:> Date: Wed, 25 Jun 2003 17:52:46 -0400 (EDT)
:> From: Carl Ma <carl_ma@scotiacapital.com>
:> Subject: filesystem grows out of control
:> To: sunmanagers@sunmanagers.org
:> Message-ID: <200306252155.h5PLt4q05582@sbtormlr2.ScotiaCapital.com>
:> Content-Type: TEXT/plain; charset=us-ascii
:>
:> Hello all,
:>
:> We have one production server - E4500 with T3+ diskarray running
:> solaris 8 with
:> latest patch level. It used to be very stable till today.
:>
:> %df -k -F ufs
:> Filesystem kbytes used avail capacity Mounted on
:> /dev/md/dsk/d0 6191949 3293302 2836728 54% /
:> /proc 0 0 0 0% /proc
:> fd 0 0 0 0% /dev/fd
:> mnttab 0 0 0 0% /etc/mnttab
:> swap 670536 0 670536 0% /var/run
:> swap 671664 1128 670536 1% /tmp
:> /dev/md/dsk/d1 400089861 200071534 200018327 50% /space
:> /dev/md/dsk/d2 200149000 20987800 179161200 10% /space/data1
:> /dev/md/dsk/d3 ... ... ... ... /space/data2
:> ...
:> /dev/md/dsk/dN .. ... ... ... /space/dataN
:>
:> We can see there are lots of local filesystems mounted under /space.Also
:> hundreds of files,directories and automount/nfs mount points are
:> also under
:> /space directory.
:>
:> Today, the /space usage grows continuously,slowly but constantly.
:> It grows about
:> 10% since noon,which is very abnormal.
:>
:> I did the following check:
:>
:> 1."du -d -sk /space" every 5 seconds,which shows the usage is growing
:> 2."du -d -sk /space/*" every 5 seconds,which gives me all the directories
:> capacity information including nfs.(-d doesn't work??)
:> 3."find /space -xdev -type f -mtime -1 -ls | sort -n -k7", it
:> does give me the
:> list of changed files on /space. There are a lot but none has big size.
:> 4. failed to use tnfdump to analyze since this filesystem has
:> lots of I/O. I am
:> lost at certain stage. :-)
:>
:> My questions are:
:> 1.may I just display the size of every directories under
:> /space(d1),excluding
:> NFS and directories crossing filesytem boundaries(du -dsk
:> /space/* doesn't
:> work)?
:> 2.is it possible that someone hides data under mount points? how
:> can I find it?
:>
:> Right now, I am working on a dirty script and would like to hear
:> from others. I
:> will summarize if solution is available.
:>
:> thanks a lot,
:>
:> carl
:>
:
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:26:40 EDT