Arbitrary Data problem (ES40 Cluster)

From: Bona Craig (bona_64unix@yahoo.com)
Date: Wed Jan 29 2003 - 07:13:56 EST


Hi All, I have this peculiar situation

On Sunday /usr suddenly became 99%, as at Monday
Morning it was already 100%.
Monday: We removed some files and were able to get it
down to 98%, but it quickly got up to 100% again. Once
more we removed some more files and got it down to 91%
but it quickly got back to 100%.
 
Tuesday: We removed some files again after thorough
checking and created a file in /usr called
testfile.txt with which we started monitoring the
growth.
 
Below are the steps we took and the output that we
got.
 
We removed some files and got it down to 93% at
12.18pm
 
Server:/usr/share/sysman/web/classLib> df -k |grep usr
cluster_usr#usr 7754288 7133093
550696 93% /usr
 
Here are the subsequent df -k outputs after that:
 
Server:/usr> df - k |grep usr
cluster_usr#usr 7754288 7285845 397944 95%
/usr
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7377877 305912 97%
/usr
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7395853 287936 97%
/usr
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7424933 258856 97%
/usr
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7499853 183936 98%
/usr
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7572789 111000 99%
/usr

At 1.49pm, the df -k output as shown below was still
99% but with decreased disk size available:
 
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7599517 84272 99%
/usr

 
Using the following command
 
find /usr -newer testfile.txt
 
we found that it was the following files that changed
within the period:
 
Server:/usr> find /usr -newer testfile.txt
/usr/users/dpm/bin/nohup.out
/usr/openv/netbackup/bin/bpsched.d/reg_bpsched.lock
/usr/openv/netbackup/bin/bpsched.d/main_bpsched.lock
/usr/openv/netbackup/db/jobs/bpjobd.act.db
 
Here is an ls output of the above named files:
 
Server:/usr/openv/netbackup/db/jobs>ls -l
usr/users/dpm/bin/nohup.out
-rw------- 1 dpm dba 72 Jan 28 12:47
/usr/users/dpm/bin/nohup.out
Server:/usr/openv/netbackup/db/jobs>
ls -l
/usr/openv/netbackup/bin/bpsched.d/reg_bpsched.lock
-rw------- 1 root daemon 19 Jan 28 13:09
/usr/openv/netbackup/bin/bpsched.d/reg_bpsched.lock
Server:/usr/openv/netbackup/db/jobs> ls -l
/usr/openv/netbackup/bin/bpsched.d/main_bpsched.lock
-rw------- 1 root daemon 19 Jan 28 13:09
/usr/openv/netbackup/bin/bpsched.d/main_bpsched.lock
Server :/usr/openv/netbackup/db/jobs> ls -l
/usr/openv/netbackup/db/jobs/bpjobd.act.db
-rw------- 1 root daemon 0 Jan 28 13:09
/usr/openv/netbackup/db/jobs/bpjobd.act.db
 
P.S Just as we were about to send the mail, here's the
latest df -k output (at 2:01pm):
 
Server:/usr> df -k |grep usr
cluster_usr#usr 7754288 7675581 8208 100% /usr

Does anyone know a better way to search for large
files within a directory? Has anyone ever experienced
such? Any assistance wd be much appreciated

Wiz

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com



This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:49:05 EDT