From: Bill Bennett (BENNETT@MPGARS.DESY.DE)
Date: Wed Aug 25 2004 - 17:49:15 EDT
Hello Mangers,
Last week I asked for pointers in recovering from a problem in which
a third-party RAID system had come up after a power failure with new
WWIDs for several LUNs, including those providing the member boot
disk and quorum disk of a single-member cluster (5.1B PK3), so that
I could no longer boot from the cluster disks, but could boot from
the pre-cluster stand-alone system disk; I'll append my original
problem description and the responses I got after this summary.
I got four repsonses, and with their help was able to recreate the
harware databases on the cluster disks and reboot the single-member
cluster without recreating it from scratch:
- Bluejay Adametz suggested that it should be possible to copy the
device databases from my stand-alone system to the cluster disks and
pointed me to Ch. 11 of the Cluster Administration manual, which
contains a description of how to do that in the case of recovering a
corrupted cluster root file system. That was a very useful first
response, since I'd decided to start looking at the Hardware Management
manual instead and might not have gotten to the Cluster Admin manual
for some time on my own....
- Iain Barker suggested that I probably wanted to use "dsfmgr -e"
so swap device names, rather than the "dsfmgr -m" I mentioned in my
original message.
- Tomas Blinn suggested that recreating the single-member cluster
would probably be best, and that a RAID system that changes it's WWID's
from time to time possibly wasn't a good choice for TruCluster disks.
- Debra Alpert sent me notes that she had written for colleagues that
outline procedures for recovering from hardware database corruptions;
in the end, these procedures were what allowed me to get the single-
member cluster back up without recreating it.
I decided that would first move the special device files of the LUNs
with new WWIDs back to their original device names, verify all of the
AdvFS filesets on the RAID system, restore any from backup that couldn't
be fixed by verify, and then give the procedure for updating the cluster
hardware database in the Cluster Admin manual a try; the main reason for
choosing this approach was that I believed, incorrectly as it turned out,
that moving the device special files with "dsfmgr -m" would associate the
devices not only with their original device names, but also the original
major and minor device numbers, so that the complete reconstruction of
the devices directory and the member dev directory on the cluster root
disk contained in Debra Alpert's procedure should not be necessary ...
but eventually I did end up doing that.
Possibly what Iain Barker tried to tell me was that "dsfmgr -m" would
not do what I wanted, but I missed the point; and in any case haven't
had a chance yet to check whether the "dsfmgr -e" command he suggested
would have restored the devices to their original minor device numbers.
But in looking at the 5.1B man page for dsfmgr, I did notice that the
example showing how to 'restore previous device special file names
after a conifiguration change' contains a step to delete the kernel
record of the device special files associated with the old component
HWID using "dfsmgr -R" that I didn't recall from the times I've done
this earlier; so what I actually did to restore the original device
names was this:
For example to move the disk that the stand-alone system saw after
the WWID change as dsk15 with HWID 103 back to dsk4 (which had been
the component with HWID 81):
# hwmgr delete component -id 81
hwmgr: Delete operation was successful
# dsfmgr -R hwid 81
# dsfmgr -m dsk15 dsk4
dsk15a=>dsk4a dsk15b=>dsk4b dsk15c=>dsk4c dsk15d=>dsk4d dsk15e=>dsk4e
dsk15f=>dsk4f dsk15g=>dsk4g dsk15h=>dsk4h dsk15a=>dsk4a dsk15b=>dsk4b
dsk15c=>dsk4c dsk15d=>dsk4d dsk15e=>dsk4e dsk15f=>dsk4f dsk15g=>dsk4g
dsk15h=>dsk4h
and so on for the other RAID LUNs whose device names had changed.
I then ran /sbin/advfs/verify on each of the AdvFS domains on the RAID LUNs,
discovering that all three cluster file systems (cluster_root, cluster_usr
and cluster_var) were corrupted and could not be fixed by verify, so I
recreated those domains and their filesets and vrestored them from backup.
Luckily, the root1_domain on LUN containing the member boot disk was found
by verify to be intact, so I had reason to hope that the h partition of
that disk had also not been damaged; from other examples in Chapter 11 of
the Cluster Admin manual and the clu_bdmgr man page, it appears that it is
not possible to recover the h partition of a member boot disk without
booting the clusterized kernel on some cluster member, which would not
have been possible for our single-member cluster.
Although it is not directly relevant to recovering the cluster hardware
database files, I will mention that two RAID LUNs that contained user
data could not be analyzed by verify at all:
# /sbin/advfs/verify common_raid
verify: vfast status not available on domain common_raid; I/O error
Not really expecting it to work, I tried running /sbin/advfs/salvage
on these two domains and was surprised to find that it was able to
recover many of the files modified since the last backup from both
of them.
I then tried updating the hardware database files from the cluster
disks according to the procedure for "recovering the cluster root file
system to a new disk" (Sec. 11.1.8 in the 5.1B Cluster Admin manual),
which involves booting from the stand-alone system disk, copying the
hardware database files from the cluster disks to the stand-alone
system disk, rebooting the stand-alone system in single-user mode,
updating the hardware database with "dsfmgr -v -F", copying the updated
database files back to the cluster disks, and then rebooting from the
member boot disk.
That is more-or-less what I did, but because the procedure in the
manual assumes that only the disk containing the cluster root file
system has changed, the exact sequence of comands shown there did not
work in our case, as the information in the database files from the
cluster disks was incorrect; so once I had performed these steps from
the procedure in the manual:
# mount cluster_root#root /mnt
# cd /mnt/etc
# cp dec_unid_db dec_hwc_cdb dfsc.dat /etc
# cd /mnt/cluster/members/member1/etc
# cp dfsl.dat /etc
# cd /
# umount /mnt
I could no longer perform the next steps, which involve mounting the
member boot disk and copying the rest of the database files from it:
# mount root1_domain#root /mnt
Error: /dev/disk/dsk3a is an invalid device or cannot be opened.
The procedure sent me by Debra Alpert seems to get around this problem
by mounting both the member boot disk (on /mnt) and the disk containing
the cluster root file system (on /mnt1) before copying any database
files; but what I actually did was to first make copies of all of the
database files on the stand-alone system disk:
# cd /etc
# for f in dec_*db ; do cp $f $f.noclu ; done
# cp dfsc.dat dfsc.noclu
# cp dfsl.dat dfsl.noclu
Then I tried the steps shown above, and on seeing the problem, I restored
the stand-alone versions of the database files (and rebooted the stand-
alone system as a precation).
To copy all of the hardware database files from the cluster disks to the
stand-alone system disk, I simply added extensions to the file name of
the copies:
# mount cluster_root#root /mnt
# cd /mnt/etc
# cp dec_unid_db /etc/dec_unid_db.clu
# cp dec_hwc_cdb /etc/dec_hwc_cdb.clu
# cp dfsc.dat /etc/dfsc.dat.clu
# cd /mnt/cluster/members/member1/etc
# cp dfsl.dat /etc/dfsl.dat.clu
# cd /
# umount /mnt
# mount root1_domain#root /mnt
# cd /mnt/etc
# cp dec_devsw_db /etc/dec_devsw_db.clu
# cp dec_hw_db /etc/dec_hw_db.clu
# cp dec_hwc_ldb /etc/dec_hwc_ldb.clu
# cp dec_scsi_db /etc/dec_scsi_db.clu
# cd /
# umount /mnt
Then just before shutting down the stand-alone system to reboot it in
single-user mode, copy the database files from the cluster disks over
the stand-alone versions:
# cp dec_devsw_db.clu dec_devsw_db
# cp dec_hw_db.clu dec_hw_db
# cp dec_hwc_cdb.clu dec_hwc_cdb
# cp dec_hwc_ldb.clu dec_hwc_ldb
# cp dec_scsi_db.clu dec_scsi_db
# cp dec_unid_db.clu dec_unid_db
# cp dfsc.dat.clu dfsc.dat
# cp dfsl.dat.clu ../cluster/members/{memb}/etc/dfsl.dat
and update their backup copies as in the procedure in the manual:
# for f in dec_*db ; do cp $f $f.bak ; done
Then I shut down, rebooted the stand-alone system in single-user mode,
and peformed the next few steps in the manual:
# hwmgr -scan scsi
hwmgr: Scan request successfully initiated
# /sbin/mountroot
Mounting / (root)
usr_cfg_pt: reconfigured
root_mounted_rw: reconfigured
usr_cfg_pt: reconfigured
root_mounted_rw: reconfigured
usr_cfg_pt: reconfigured
dsfmgr: NOTE: updating kernel basenames for system at /
scp kevm tty0 ... dsk0 floppy0 dsk1 mc0 tape0 dsk2 cdrom0
dsfmgr: NOTE: creating device special files for system at /
+dsk14a +dsk14a ... +dsk18h +dsk18h
So not unexpectedly, dsfmgr saw the RAID LUNs with different WWIDs as
being different devices than those found in the device database files
taken from the cluster disks and created new special device files for
them, as had happened earlier when I first booted the stand-alone
system; carrying on with the procedure in the manual, I ran dsfmgr
to update the device databases:
# dsfmgr -v -F
which reported "OK." for all checks; the output of "hwmgr -view devices"
then was the same as that obtained on the stand-alone system before I
had moved the RAID LUNs back to their original device names.
At this point, I used the same commands I had used earlier to move the
RAID LUNs from their new device names, dsk14-dsk18, back to their
original device names, dsk3-7, then checked with "dsfmgr -v" and
"hwmgr -view devices" that things had gone as expected, and was able
to mount those RAID LUNs whose file systems were included in the fstab
of the standalone system with "bcheckrc". It was probably good that I
performed the same device-name manipulations at this point while using
the copies of the hardware database files that I eventually would use
to reboot the cluster, as I could later determine what the major and
minor device number would be for the cluster member by looking at those
found on the stand-alone system.
Before copying the updated database files back to the cluster disks, I
made copies of them with a new extension:
# cd /etc
# cp dec_devsw_db dec_devsw_db.newclu
# cp dec_hw_db dec_hw_db.newclu
# cp dec_hwc_cdb dec_hwc_cdb.newclu
# cp dec_hwc_ldb dec_hwc_ldb.newclu
# cp dec_scsi_db dec_scsi_db.newclu
# cp dec_unid_db dec_unid_db.newclu
# cp dfsc.dat dfsc.dat.newclu
# cp ../cluster/members/{memb}/etc/dfsl.dat /etc/dfsl.dat.newclu
and to make sure that I would be able to reboot the stand-alone system,
I restored the original database files from the stand-alone system disk
and updated the backup copies to be consistent with them:
# cp dec_devsw_db.noclu dec_devsw_db
<...etc...>
# cp dfsl.noclu ../cluster/members/{memb}/etc/dfsl.dat
# for f in dec_*db ; do cp $f $f.bak ; done
Finally, I copied the updated database files back to the cluster disks;
but because of the various versions of the files with different extensions
now on the stand-alone disk, I didn't use the wildcard commands shown in
the manual (intended to copy both the database files and their backup
copies), but copied them individually to rename them in the process:
# mount cluster_root#root /mnt
# cd /etc
# cp dec_unid_db.newclu /mnt/etc/dec_unid_db
# cp dec_unid_db.newclu /mnt/etc/dec_unid_db.bak
# cp dec_hwc_cdb.newclu /mnt/etc/dec_hwc_cdb
# cp dec_hwc_cdb.newclu /mnt/etc/dec_hwc_cdb.bak
# cp dfsc.dat.newclu /mnt/etc/dfsc.dat
# cp dfsl.dat.newclu /mnt/cluster/members/member1/etc/dfsl.dat
# cd /
# umount /mnt
# mount root1_domain#root /mnt
# cd /etc
# cp dec_devsw_db.newclu /mnt/etc/dec_devsw_db
# cp dec_devsw_db.newclu /mnt/etc/dec_devsw_db.bak
# cp dec_hw_db.newclu /mnt/etc/dec_hw_db
# cp dec_hw_db.newclu /mnt/etc/dec_hw_db.bak
# cp dec_hwc_ldb.newclu /mnt/etc/dec_hwc_ldb
# cp dec_hwc_ldb.newclu /mnt/etc/dec_hwc_ldb.bak
# cp dec_scsi_db.newclu /mnt/etc/dec_scsi_db
# cp dec_scsi_db.newclu /mnt/etc/dec_scsi_db.bak
# cd /
# umount /mnt
I then shut down the cluster to the SRM console and tried to boot from
the member boot disk, using the interactive boot command shown in the
manual to specify the cluster root device:
>>> boot -fl "ia"
Enter: <kernel_name> [options]
# vmunix cfs:cluster_root_dev1_maj=19 cfs:cluster_root_dev1_min=238
But again the boot of the cluster failed at the same point it had earlier,
waiting for the "member boot disk to become registered"; so updating the
hardeware database files on the cluster disks alone was not sufficient to
correct the problem.
On looking at the notes provided by Debra Alpert more closely, I saw
that they include instructions for editing the sysconfigtab for the
cluster member to correct the major and minor device numbers for the
member boot disk and quorum disk. Knowing from that what to look for,
I booted again from the stand-alone system disk and mounted the cluster
root disk. I could then see that the device numbers of the member boot
disk in the member's sysconfigtab were inconsistent with both the device
numbers in the member's /dev directory and those in the /dev directory
of the stand-alone system disk; moreover, the device numbers in the
member's /dev directory were not the same as those in the stand-alone
/dev directory.
At that point it was finally clear to me that "dsfmgr -m" hadn't done
what I thought it would do and that the more complete reconstruction of
the device-related files and directories in the procedure suggested by
Debra Alpert would be necessary, so I decided to give it a try; besides,
my only other option by that time was to recreate the single-member
cluster, so it wouldn't matter if I trashed the cluster file systems
in the process...
The notes that Debra Alpert sent me actually contain several database
recovery procedures for different configurations; the one I followed
was "How to rebuild HW-DB for 1st Member in Cluster", starting with
the section "Cleanup Clustermember 1 and Cluster Root"; this procedure
involves removing the contents of the cluster /devices directory and
most of the member's /dev directory, plus the hardware database files
and a few others from the cluster disks, copying the database files
from the stand-alone system to the cluster disks, editing the member's
sysconfigtab to correct the device numbers for the member boot disk
and quorum disk, and then booting the first cluster member to single-
user mode to update the database files and reconstruct the device
directories.
The files replaced in Alpert's procedure that weren't included in the
procedure from the Cluster Admin manual that I tried earlier were
/etc/dccd*, /etc/dcdd* and /cluster/members/member1/etc/cfginfo, all
on the cluster root disk; but I could see from the time stamps and
with cmp that the copies of these files on the stand-alone system disk
and on the cluster root disk were identical on our system, so I did not
bother to replace them.
On the other hand, comparison of the device database files on the
stand-alone disk with the updated versions of the database files I had
copied earlier from the cluster disks with cmp showed that they were
not identical, so I decided to use the updated cluster database files
as the starting point for reconstructing the cluster device directories;
since Alpert's procedure is what worked in the end, this was probably
not necessary, but it seemed to me at the time that it wouldn't hurt to
use the versions of the database files derived from original cluster
database files.
So this is what I actually did after rebooting the stand-alone system
in single-user mode; other than as noted above, I just follow Alpert's
notes as closely as the file names of the updated cluster database
files allow:
# mount root1_domain#root /mnt
# mount cluster_root#root /mnt1
# rm /mnt/etc/dec*
# rm /mnt1/etc/dfsc*
# rm /mnt1/etc/dec_unid_db*
# rm /mnt1/etc/dec_hwc_cdb*
# rm -rf /mnt1/devices/*
# rm /mnt1/cluster/members/member1/.Booted
# rm /mnt1/cluster/members/member1/etc/dfsl*
# rm -rf /mnt1/cluster/members/member1/dev/[a-z]*
# cd /mnt1/cluster/members/member1/dev/; ./MAKEDEV std
Then copy my updated cluster database files to the cluster disks as earlier:
# cd /etc
# cp dec_devsw_db.newclu /mnt/etc/dec_devsw_db
# cp dec_devsw_db.newclu /mnt/etc/dec_devsw_db.bak
# cp dec_hw_db.newclu /mnt/etc/dec_hw_db
# cp dec_hw_db.newclu /mnt/etc/dec_hw_db.bak
# cp dec_hwc_ldb.newclu /mnt/etc/dec_hwc_ldb
# cp dec_hwc_ldb.newclu /mnt/etc/dec_hwc_ldb.bak
# cp dec_scsi_db.newclu /mnt/etc/dec_scsi_db
# cp dec_scsi_db.newclu /mnt/etc/dec_scsi_db.bak
# cp dfsc.dat.newclu /mnt1/etc/dfsc.dat
# cp dfsc.dat.newclu /mnt1/etc/dfsc.bak
# cp dec_unid_db.newclu /mnt1//etc/dec_unid_db
# cp dec_unid_db.newclu /mnt1//etc/dec_unid_db.bak
# cp dec_hwc_cdb.newclu /mnt1/etc/dec_hwc_cdb
# cp dec_hwc_cdb.newclu /mnt1/etc/dec_hwc_cdb.bak
# cp dfsl.dat.newclu /mnt1/cluster/members/member1/etc/dfsl.dat
# cp dfsl.dat.newclu /mnt1/cluster/members/member1/etc/dfsl.bak
Since I have never had to use ed much, and that not for some years, I
decided at this point not to risk editing sysconfigtab in single-user
mode with ed; instead I rebooted the stand-alone system in multi-user
mode, remounted the member boot disk on /mnt and edited
/mnt/etc/sysconfigtab with vi to achieve this:
# cd /mnt/etc
# egrep cluster_ sysconfigtab
cluster_seqdisk_major=19
cluster_seqdisk_minor=348
cluster_qdisk_major=19
cluster_qdisk_minor=394
where (19/348) are the major/minor device numbers of the boot partition
of the member boot disk and (19/394) are the device numbers of the h
partition of the quorum disk on the stand-alone system.
Finally I wanted to boot from the member boot disk in single-user mode to
run "dn_setup -init", etc., as in Alpert's notes, but I made a major
typo in the boot command, specifying the flags "ia" instead of "is":
>>> boot -fl "ia" dkb101
<...>
Enter: <kernel_name> [options]
# vmunix cfs:cluster_root_dev1_maj=19 cfs:cluster_root_dev1_min=238
That of course directed the system to boot to multi-user mode instead
of single-user mode, but it worked anyway, probably because the commands
I intended to issue in single-user mode to rebuild the device directories
and database files (here from Alphert's notes):
mountroot
dn_setup -init
dsfmgr -K
are run as part of the boot process anyway (except "dn_setup -boot",
instead of "dn_setup -init").
In followup checks after logging in on the single-member cluster, I
found that the LAT devices (/dev/lat) had not been recreated by the
procedure above, but that was easily corrected by undoing the LAT
setup and then runnning latsetup again.
There are a few other unusual entries in the startup logs; prpasswd
seems to have a problem recovering some log file on startup:
Aug 23 00:08:26 mynode prpasswdd[525058]: prpasswdd: \
Recovering the log: last valid LSN: file: 1 offset 15661
Aug 23 00:08:26 mynode prpasswdd[525058]: prpasswdd: \
Recovery function for LSN 1 14494 failed
Aug 23 00:08:26 mynode prpasswdd[525058]: now active and \
servicing client requests
and a console message from "clu_bdmgr -b" indicates that it has a problem
saving the h partition of the member boot disk:
*** Error ***
Cannot update boottime cnx configuration file
/.local../boot_partition/etc/clu_bdmgr.conf
Failure to process request
although I can read the contents of the cnx partition with "clu_bdmgr -d"
after the system is up, the .local.. CDSL is correct, and the file
/.local../boot_partition/etc/clu_bdmgr.conf (from the last boot prior
to the power failure) is readable (and the same as what "clu_bdmgr -d"
shows me). Since neither of these problems either prevents the single-
member cluster from booting or hinders normal operation, I will take
my time about figuring out how to fix them ... or if needed, at least
post separate questions about them.
Thanks again to those who responded for their help; their responses
are appended after my signature and original problem description (less
headers, to avoid revealing their email addresses to the spammers).
Regards,
Bill Bennett
----------------------------------------------------------------------------
Dr. William Bennett within Germany International
MPG AG Ribosomenstruktur Tel: (040) 8998-2833 +49 40 8998-2833
c/o DESY FAX: (040) 897168-10 +49 40 897168-10
Notkestr. 85
D-22603 Hamburg Internet: bennett@mpgars.desy.de
Germany
---- my original problem description
Hello Managers,
I have a DS20E that was recently upgraded to 5.1B PK3 and then made a
single-member cluster; the second member has not yet been added to
the cluster. The disks containing the cluster root, usr and var
file systems, member boot disks, quorum disk and a few user disks
are actually all partitions (seen as LUNs of one SCSI ID by the
DS20E) on a single RAID set on a third-party hardware RAID system
(CMD CRD-5500); at the moment, a KZPBA-CB controller in the DS20E
and the CRD-5500 are the only devices on what will eventually be
the shared cluster SCSI bus.
Today we had a short power outage in the computer room; unfortunately,
one of the things I hadn't gotten to yet was to put the RAID system
on a UPS, so it lost power while the DS20E stayed up. Although the
RAID set per se came up undegraded when power was restored, the DS20E
was hung when I found it. On resetting the DS20E, I could see at the
SRM console that it could still find all the LUNs from the RAID system,
but an attempt to boot the DS20E as a single-member cluster failed;
early in the boot output I saw the line:
drd_config_thread: 5 previously unknown devices
and later the cluster reboot got stuck, repeatedly printing the line:
waiting for cluster member boot disk to become registered
I was able to boot from the stand-alone (pre-cluster) system disk; during
the boot of the stand-alone system, a number of new special device files
were created, and after the system was up, I could see that the problem
is that the WWIDs for 5 of the 6 LUNs of the RAID system had changed
somehow ... actually, in each case, four digits of a 32-digit hex string
in the WWID were changed, although the WWIDs remain unique (or at least
different from one another). The LUN of the RAID set whose WWID did not
change was LUN 0, which contained the cluster root, usr and var file
domains, but the WWIDs of the LUNs containing the member boot disks,
quorum disk and user disks did change.
I have no idea what caused that, and since it is clearly not a DEC/
Compaq/HP device, this is probably not the place to find out ... but
if anyone has any insight as to what might cause that or how it can
be avoided in the future, I would be happy to hear them...
But my more immediate problem is how to recover from this situation in
which the first cluster member can't find the it's boot disk because the
WWID of the disk has changed. I can in principle access all the disks
now after booting from the stand-alone system, but I haven't yet ruled
out the possibility that some of the AdvFS domains were corrupted when
the RAID system lost power.
I can imagine, perhaps naively, three ways that it might be possible
to recover from this problem, so as I sit down to look at the hardware
management documentation in more detail, I thought it would be a good
time to ask for pointers ... perhaps someone can at least help me rule
out the bad ideas sooner rather than later.
Given that on booting the stand-alone system, the RAID LUNs with new
WWIDs were assigned new HWIDs and device names (they were dsk3-dsk7
but are now dsk14-dsk18), it seemed to me that it should be possible
to do the following:
1) use the 'hwmgr -delete component -id oldHWID' and 'dsfmgr -m
newdev olddev' commands to restore the wayward LUNs to their previous
device names; this would presumably update the hardware database files
on the stand-alone system to account for the new WWIDs of these LUNs.
Then after verifying the file domains on the RAID LUNs (and where needed
restoring corrupted domains from backup) on the stand-alone system, one
could in principle mount the cluster root filesets temporarily on the
stand-alone system and copy the modified device database files to them.
The problem with this idea is that I don't know enough about how the
Tru64 hardware management works to be certain that the updated database
files from the stand-alone system would be usable by the cluster, or
for that matter, exactly which hardware database files would need to be
copied...
2) leave the new device names as they are and update the links in
/etc/fdmns on the stand-alone system disk to point to the new devices;
after doing that, I could verify the domains and restore from backup
as needed, then temporarily mount the cluster root filesets on the
stand-alone system to update the AdvFS links on them, too, so that the
cluster could find all of its disks on the next boot.
But this would only work if the cluster makes the same new device
name assignments as the stand-alone system, and I'm not sure how good
a bet that is...
3) restore the device assignment of the RAID LUNs on the stand-alone
system as in option 1, then verify and restore from backup only the
user disks so that all then entries in /etc/fstab on the stand-alone
system are working again; then run clu_create to recreate the single-
member cluster again from scratch. Restore modified configuration
files selectively from backup to bring the system back to the state
it was in before the WWIDs changed.
I think that is the most likely option to work, but that last step
might not be as simple as it sounds...
Any suggestions or pointers to relevant documentation would be
greatly appreciated!
Regards,
Bill Bennett
---- response from Bluejay Adametz
You should be able to copy the device database files from your standalone
system to the cluster system. Use the standalone system to determine the
major & minor device numbers for the cluster root disk; you'll need those to
boot the cluster.
Chapter 11 of the cluster admin manual actually details this procedure:
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51B_HTML/ARHGYETE/TIT
LE.HTM
- Bluejay Adametz
---- response from Iain Barker
what you probably want is "dsfmgr -e" to swap device names,
rather than hwmgr to delete and recreate them.
---- response from Thomas Blinn
I hate to say it, but it sounds like maybe your RAID array isn't
designed to be able to work reliably with Tru64 UNIX or TruCluster
software -- because if it loses some or all of the WWID values on
a power cycle, it's going to cause you a world of pain whenever
that happens, and I'm sure you don't want to be there..
That said, I'd say that your "option 3" has the best chance of
working. There is no simple way to get usable hardware mgmt
databases from the standalone (unclustered) environment onto
all the places they need to be, and there are places on the
cluster member boot disks where some of the data is encoded
in ways that are very dependent on internal representations
of device identifiers that you won't find documented anywhere,
so trying to "repair" them by hand is nearly impossible (and
I say this as someone who's close enough to the details to be
in the know if it were possible to do this easily). I'm not
suggesting it can not be done, but rather that you are very
unlikely to find anyone who can give you a cookbook on how to
do it.
Unless you can resolve the RAID array reliability issues, then
putting important data on the device and building a cluster to
access it seems pretty risky to me..
Tom
---- response from Debra Alpert
Hi Bill,
I've been up all night due to a maintenance at our site, so it's hard to
compose a clear response, but please see if the forwarded message that I
wrote up for our systems staff is of use to you. I hope this helps.
Regards,
--Deb
---- and her forwarded message
Hello,
Since corrupted device databases have reared their ugly heads again
today, I thought I should send out this summary describing how to
correct these types of situations. The steps that follow assume a Tru64
5.X cluster, where cluster LSM is not utilized. All of our clusters
fall into this category. The steps under the header "Cleanup the
Installation Disk" may be used to repair device databases on a
standalone node that is not using LSM. On the other hand, we don't have
any of those...
On standalone nodes where LSM is used (these we have), physically remove
one of the rootdg disks, and boot the remaining disk to single-user
mode. You can't actually mount anything except the root partition in
this situation. Once you run "mountroot", you can apply the steps in
the "Cleanup the Installation Disk" section. When the node is rebooted
after these steps are completed, you'll have to deactivate LSM to
proceed further. Assume that the disk you've booted from is
/dev/disk/dskB.
# mountroot
# cd /etc/vol
# mv volboot volboot.sav
# ed /etc/sysconfigtab
/swapdevice/p
swapdevice = /dev/vol/rootdg/swapvol
/swapdevice/s/vol\/rootdg\/swapvol/disk\/dskBb/p
swapdevice = /dev/disk/dskBb
/rootdev/p
lsm_rootdev_is_volume = 1
/rootdev/s/1/0/p
lsm_rootdev_is_volume = 0
w
q
# disklabel -e dskB
(using ed, you'll have to replace all LSM references with AdvFS, swap,
or unused filesystem types, as appropriate)
# ed /etc/inittab
(using ed, you'll have to comment out the two lsm entries and the single
vol entry)
# cd /etc/fdmns/root_domain
# rm *
# ln -s /dev/disk/dskBa
# cd ../usr_domain
# rm *
# ln -s /dev/disk/dskBg
# shutdown -r now
You should now come up in multi-user mode, with no user filesystems
accessible, as LSM is disabled. Once you encapsulate the root disk, the
LSM metadata on the other disks will allow automatic import of the user
diskgroups.
# volencap dskB
# volreconfig
After the system reboots, all filesystems should be mounted and
accessible. Insert the system mirror disk you removed earlier back into
its former slot, run "scu scan edt" so the system recognizes the disk,
run "hwmgr -vi dev" to obtain its new name, and use "dsfmgr" to rename
the disk to its original OS designation. You can now modify the
disklabel on this device, say dskM, and remirror the boot device:
# disklabel -z dskM
# disklabel -r dskB >/tmp/dl
# vi /tmp/dl (mark all partitions unused, and if you enjoyed ed, go for
it again instead of vi!)
# disklabel -rR dskM /tmp/dl
# volrootmir -a dskM
The cluster repair steps follow. For LSM protected standalone nodes,
follow the "Cleanup the Installation Disk" section at the point
indicated.
Have fun!
--Deb
####################################################################
Note:
-the terms "install disk" and "ER disk" and "standalone
disk" are interchangeable
-the procedure is normally done using member1. This is due to the
fact that the local device info propagated to the cluster initially
is from member1. If you use a different member, the local info
restored to the "down" cluster may not match exactly. This can
usually be cleaned up later with various hwmgr/dsfmgr commands.
####################################################################
How to rebuild HW-DB for 1st Member in Cluster
--------------------------------------
- Have a 'hwmgr show scsi -full' from all systems saved
- Take note of the tape, mc and cdrom devices manually
- shutdown the whole cluster
- boot the install disk
- Cleanup Install disks' HWDB here if necessary
#####################################################################.
Cleanup the Installation Disk
-----------------------------
boot -fl s <instdisk>
rm /etc/dec*
rm /etc/dfsc*
rm /etc/dc*
rm -rf /cluster/members/member/dev/[a-z]*
cd /cluster/members/member/dev/; ./MAKEDEV std
rm /cluster/members/member/etc/dfsl*
rm /cluster/members/member/.Booted
rm -rf /devices/*
halt
boot -fl s <instdisk>
mountroot
dn_setup -init
dsfmgr -K
dsfmgr -v # optionally -vF
hwmgr show scsi
# fix if necessary links in /etc/fdmns
mount /usr
mount /var
cdslinvchk # Fix problems NOW
dsfmgr -m/-e # to make old/new device names match
##################################################################
Notes:
-here we're assuming you're fixing member1. Change
accordingly if necessary.
-you may need to manually create the mount directories and links in
/etc/fdmns for cluster_root and root?_domain.
-you should backup all of these files before removing them - best
just to 'mv' them to a new "backup" subdirectory.
Cleanup Clustermember 1 and Cluster Root
----------------------------------------
boot -fl s <instdisk>
mount root1_domain#root /mnt # Do the fdmns links match?
mount cluster_root#root /mnt1 # Do the fdmns links match?
rm /mnt/etc/dec*
rm /mnt1/etc/dfsc*
rm /mnt1/etc/dec_unid_db*
rm /mnt1/etc/dec_hwc_cdb*
rm /mnt1/etc/dccd*
rm /mnt1/etc/dcdd*
rm -rf /mnt1/devices/*
rm /mnt1/cluster/members/member1/.Booted
rm /mnt1/cluster/members/member1/etc/dfsl*
rm /mnt1/cluster/members/member1/etc/cfginfo
rm -rf /mnt1/cluster/members/member1/dev/[a-z]*
cd /mnt1/cluster/members/member1/dev/; ./MAKEDEV std
###################################################################
Setup first Member & Cluster
------------------------------
To member boot:
cp /etc/dec_devsw* /mnt/etc/
cp /etc/dec_hw_db* /mnt/etc/
cp /etc/dec_hwc_ldb* /mnt/etc/
cp /etc/dec_scsi* /mnt/etc/
To cluster root:
cp /etc/dfsc* /mnt1/etc/
cp /etc/dec_unid_db* /mnt1/etc/
cp /etc/dec_hwc_cdb* /mnt1/etc/
cp /etc/dccd* /mnt1/etc/
cp /etc/dcdd* /mnt1/etc/
cp /etc/dfsl* /mnt1/cluster/members/member1/etc/
cp /etc/cfginfo /mnt1/cluster/members/member1/etc/
###################################################################
For ALL Members do :
--------------------
file /dev/disk/dsk?h # Quorum Disk
- Take note of the Major/Minor Number
file /dev/disk/dsk?a # Member Boot Disk
- Take note of the Major/Minor Number
- edit /mnt/etc/sysconfigtab
clubase:
cluster_qdisk_major=19 # From above Quorum Disk
cluster_qdisk_minor=32 # From above Quorum Disk
cluster_seqdisk_major=19 # From above Boot Disk
cluster_seqdisk_minor=64 # From above Boot Disk
vm:
swapdevice=/dev/disk/dsk?b # Use correct swapdevice here !
###################################################################
# reboot first member into 'new' cluster
#boot -fl s <memberbootdisk>
Note: you'll likely need to specify the cluster root maj/min
numbers here. This should automatically update the cnx partitions
everywhere (if you have the qdisk configured).
boot -fl is <memberbootdisk>
vmunix cfs:cluster_root_dev1_min=19 cfs:cluster_root_dev1_maj=XXXX
mountroot
dn_setup -init
dsfmgr -K
dsfmgr -v # optionally -vF
hwmgr show scsi
# fix if necessary links in /etc/fdmns
mount /usr
mount /var
cdslinvchk
###############################################################
Note: You may not need to do this. If you decide to copy the
device DBs to all member boot disks (and if you can easily fix
the local device name/ID issues) this is moot.
# For all remaining members
mount root2_domain#root /mnt # Do the fdmns links match?
rm /cluster/members/member2/.Booted # For all members
rm /cluster/members/member2/etc/dfsl*
rm /cluster/members/member2/etc/cfginfo
rm -rf /cluster/members/member2/dev/[a-z]*
cd /cluster/members/member2/dev/; ./MAKEDEV std
##################################################################
Note: You may not need to do this. If you decide to copy the
device DBs to all member boot disks (and if you can easily fix
the local device name/ID issues) this is moot.
Create genesis databases
------------------------
clu_bdmgr -d dsk0 >/tmp/dsk0.bd #dsk5 should be a bootdisk with
valid cnx partition (example member1 bootdisk)
/usr/sbin/cluster/clu_partmgr -mg /tmp/dsk0.bd dsk5 #dsk5 =
member-boot-disk which should be created
# clu_partmgr initialize cnx partition and creates a valid genesis
# hwdb at /etc for this member
mv /etc/dec_hwc_genesis* /mnt/etc/
#at this point you have to check all relevant files, sysconfigtab,
#cnx partition etc, to be sure that all looks correct
#################################################################
# Boot second member into cluster
cd /
umount /mnt
boot -fl s <bootdisk> # Will fail if you forgot umount
mountroot
dn_setup -init
dsfmgr -K
dsfmgr -v # optionally -vF
hwmgr show scsi
# Very important!
# finally you have to take down the cluster and boot it once again,
# to be sure, that the new created HW-DB is really loaded into
# kernel.
#################################################################
#sometimes additional
---------------------
# If you have problems with cnx partition, shutdown this member to
# create a proper cnx partition
init 0
################################################################
Note: this shouldn't be necessary if booting with the
interactive boot flags specifying the cluster_root maj/min
values. Confirm with a 'clu_bdmgr -d dsk??' on all CNX
partitions to ensure they're pointing to the correct disk.
# Create proper CNX partitions
clu_bdmgr -d dsk1 >/tmp/dsk1.bd # As a backup
mount root2_domain#root /mnt
vdump 0f /tmp/root.vdmp /mnt
umount /mnt
rm -rf /etc/fdmns/root2_domain
clu_bdmgr -c dsk1 2
mount root2_domain#root /mnt
cd /mnt
vrestore xf /tmp/root.vdmp
cd /
umount /mnt
# boot this member now into cluster
###############################################################
--
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:50:07 EDT