From: Reed, Judith (jreed@navisite.com)
Date: Thu Jul 15 2004 - 10:41:37 EDT
I reported thousands of files in /tmp (subdir of /) which were
"invisible" to "ls -alR /tmp" but which could be listed and accessed if
you knew their actual filename. I ended up calling HP, and we ran
"verify -a root_domain" with the result:
Found entry size too large on page 0; //cluster/members/member0/tmp is
corrupted
HP feels that the tmp directory linked list of files is broken in the
middle somewhere. They recommend I take the system down and run verify
from cdrom to fix the root_domain, and that until we do that we do not
intentionally create any files in /tmp, because that could cause the
"invisible" files to become permanently inaccessible. I was able to move
user files and subdirs to another dir successfully, so that all that is
left there are files we don't care about. This is a prod system so we
are going to have to wait for an opportunity to take it down, but HP
didn't think the problem would spread to other subdirs of / - hope they
are right!
Thanks to those who sent ideas. They were as follows:
* try an "ls -a"
* look for disk i/o errors (none) and advfs errors (none in logs, till I
ran
"verify")
* Derek Haining sent instructions on running fixfdmn:
----------------------------------------------------------------
For "large" directories AdvFS keeps a second directory file which is
hidden from the view. This is the "index directory". The index
directory basically holds pointers to all of the files in the regular
directory, but the entries in the index directory are sorted
alphabetically. When a look-up is done on a large directory, the index
is consulted to find which block of the normal directory contains the
data about the named file.
I suggest running fixfdmn in the "no fix" mode against this filesystem.
If this is part of the root filesystem, then the easiest way to do this
is to boot off of the OS CD-ROM, escape to a shell, and define the
domain.
To define the domain without destroying it...
o determine the device name of the disk that holds the root domain.
For this example, assume that this is dsk29a.
o "cd" to the /etc/fdmns directory. When booted off of the OS CD-ROM
this will end up actually being on a memory file system.
o "mkdir root_domain" in /etc/fdmns
o "cd root_domain". If you were to do a "/bin/pwd" you should see
that
you are not in "/etc/fdnms/root_domain" but somewhere else off of
the
memory file system. Obviously you cannot write to the CD-ROM. :)
o Establish a link to the disk. "ln -s /dev/disk/dsk29a"
Now you should be able to run fixfdmn against the domain. "cd /tmp" and
then "/sbin/advfs/fixfdmn -n root_domain". The "-n" flag tells fixfdmn
to "NOT FIX" anything, but to simply examine the filesystem. Once
fixfdmn completes you should have a log file named something like
fixfdmn.log.root_domain. Read through this log looking for any
significant errors.
I'm don't remember what fixfdmn will do if it finds a discrepancy
between the regular directory and the index directory. My recollection
is that the index directory is deleted. Later, when another change is
made to the regular directory, the index directory will be automatically
created if the regular directory is large enough.
------------------------------------------------------------
* Bob Harris noted that strings shows files that had been deleted, as
well as actual files, and that files of the form "sh57582.Fnwp2" were
probably from shell script EOD inlining. He also suggested that "ls"
might be broken (but I'd tried /usr/ucb/ls also, same result), or that
one of the files in the directory might have non-printing chars in its
name that turned off display of other filenames. This could be seen
with:
ls /tmp/ | cat -vte
-or-
ls /tmp/ >ls.command.saved.output
vi ls.command.saved.output
* Warren Sturm advised checking if the system had been compromised.
Thanks again!
Judith Reed
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:50:03 EDT