root_mirror 1996/04/29 11:30 - jrv - NOTE: This procedure has been verified for the situation when the main and the mirror root disks are NOT the same type/size. I.e., a 2GB root can be mirrored to a 4GB drive. SBM's recommendation is to create two Physical Volume Groups (PVGs), say, "internal" and "external" You can add another two GB drive to PVG vg00/internal and then you have a good matchup: TWO 2GB's mirrored to ONE 4GB. ( note - 2000/03/16 - jrv The following note about the last block may no longer be true ) NOTE: since the LAST Logical Extent is NOT used on a drive, the above scenario will lead to an unused block on the 4GB drive. IE, you have 1 LE from each of the two 2GB drives, for a total of 2 unused LE's, while the 4GB drive has only one unused. This merely means that the total size of the available space will show one less block in PVG internal than PVG external. If you mirror EVERYTHING, you will not be able to use that extra block on the 4GB drive. (see above note - 2000/03/16 - jrv ) Conceptually, we have: PVG PVG internal" "external" .------. --.___________.-- .---------. | .....| __/ \__ |..... | 2GB | .....| __>-----------<__ |..... | | .....| __>-----------<__ |..... | ----|------| \ / | | 4GB drive | | | | | | 2GB | | |---------| | | | | | | | | `------' --' `--- `---------' NOTE that the last lvol spans TWO drives in PVG "internal", but is entirely accomodated by ONE drive in PVG "external". ==================================== Mirroring the system/root disk ==================================== >From "System Administration Tasks", Chapter 8 "Managing Logical Volumes" ======================================================================== Example: Mirroring the Root File System Logical Volume Since root and any mirror copies of root must begin at physical extent 0000 on a bootable disk, root needs to be mirrored on either a new disk or on a disk with the space created for it. The following example shows the addition of a disk to the volume group for the purpose of mirroring root. The example also includes creating a mirror copy of the primary swap logical volume, since that is a logical companion step to the mirroring of root. Primary swap does not need to be on a specific disk or at a specific location, but it does need to be contiguous. The recommended mirror policy for primary swap is to have the Mirror Write Cache and the Mirror Consistency Recovery mechanisms disabled. 1. Create another bootable LVM disk. (SBM NOTE: If this disk has previously been used, then you may get a message stating that the disk is already in a volume group. In this case, add the '-f' option to the pvcreate command. Ex. pvcreate -f -B /dev/rdsk/c3d0s2 ) pvcreate -B /dev/rdsk/c2t1d0 # (make it bootable) Note the r in rdsk; use the disk's character (raw) device file. The -B option creates a region for a LIF volume and boot utilities, and creates a BDRA (Boot Data Reserved Area) on this disk. 2. Include this disk in the current root volume group. vgextend vg00 /dev/dsk/c2t1d0 # (add to vg00) Note /dsk; this is the disk's block device file. 3. Now, make the disk a boot disk by adding boot utilities: mkboot -l /dev/rdsk/c2t1d0 # (write boot info to disk) For LVM disks, the mkboot(1) command requires the character device file. NOTE that above 'mkboot' ONLY copies the DEFAULT set of boot LIF files. You are better off to copy ALL the LIF files, in case any other utilities (like ODE) were installed on the original boot disk: for f in `lifls /dev/rdsk/c0t6d0` ; do lifcp /dev/rdsk/c0t6d0:$f /dev/rdsk/c2t1d0:$f ; done NOTE the $f at the end of BOTH device: names !!! 4. Copy a correct AUTO file into the new LIF area that REMOVES THE DISK QUORUM requirement (we always want to be able to boot!!!). Do this ON BOTH DISKS!!!! (main AND mirror) as follows: for hp-ux 9.x mkboot -a "hpux -lq (;0)/hp-ux" /dev/rdsk/c0d0s2 (change the boot command to remove quorum checks on root disk) mkboot -a "hpux -lq (;0)/hp-ux" /dev/rdsk/c3d0s2 (change the boot command to remove quorum checks on root MIRROR) for 1x.x mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c0t6d0 mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c2t1d0 Note the use of the disk's character device file. Note, "/hp-ux" and "/stand/vmunix" are the name of the kernel that will be booted under 9.x and 1x.x, respectively. For 1x.x, the devic/kernel-name can be omitted: mkboot -a "hpux -lq" /dev/rdsk/c2t1d0 Verify that the correct string has been installed on BOTH disks: lifcp /dev/dsk/c0t6d0:AUTO - | head -1 lifcp /dev/dsk/c2t1d0:AUTO - | head -1 5. Make the new mirror the alternate boot drive: setboot -a 6. Set MWC **OFF** on the swap lvols: First, do primary system swap lvol: reboot (shutdown -ry 0) -- interrupt boot process then "bo" hpux -lm -lq #from the ISL prompt vgchange -a y vg00 lvchange -M n -c n /dev/vg00/lvol2 vgchange -a n vg00 If there are no more swap LVOLs, then just reboot Otherwise, the other swap lvols can be done in single-user mode reboot -- interrupt boot process hpux -is -lq #from the ISL prompt lvchange -M n -c n /dev/vg00/lvoln # on other swap lvols init 3 Once the system is back up, you may notice, while using bdf, that / is not the /dev/vg00/lvol1 filesystem, but is labeled as /dev/root. To correct this, do a: rm /etc/mnttab and then do a: mount -u 7. Do any PVG setup NOW. vi /etc/lvmpvg VG vg00 PVG vg00_INT_A /dev/dsk/c0t6d0 PVG vg00_INT_B /dev/dsk/c2t1d0 Change the mirroring policy to PVG-strict: lvchange -s g /dev/vg00/lvol1 lvchange -s g /dev/vg00/lvol2 ... 8. Now, mirror the system logical volumes to the new bootable disk. Just for "cleanliness", get the LVOL names and order so that the mirror will match and will be in the same order. 99% of the time it simply be lvol1, lvol2, ... in sequential order. pvdisplay -v /dev/dsk/c1t3d0 | grep '^ */dev/vg00' Do the mirrors lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c2t1d0 lvextend -m 1 /dev/vg00/lvol2 /dev/dsk/c2t1d0 ... or more NOTE: some resyncing time is involved above!! NOTE: if there are only 2 disks in vg00 or PVGs were set up, then the device name can be omitted. NOTE: If the LVOLs are simply lvol1, lvol2, ..., then a 'for' loop can make it all happen with one command (but the lvextends will still be done serially -- you cannot do 2 mirrors contemporaneously in one Volme Group). Be sure to use the correct order discovered above: for x in 1 2 3 4 5 6 7 8 do lvextend -m 1 /dev/vg00/lvol$x done Note the use of the block device files. 9. Now, update the boot information contained in the BDRA (Boot Data Reserved Area) of the boot disks in the root volume group with the locations of the mirror copies of root and primary swap. lvlnboot -R #recovers any BDRAs 9.x and 10.01, 10.10, or updates to 10.20, where /stand is not a separate LVOL: lvlnboot -r /dev/vg00/lvol1 #specifies the root LV lvlnboot -s /dev/vg00/lvol2 #specifies the swap LV lvlnboot -d /dev/vg00/lvol2 #specifies the dump LV 10.20,11.x: lvlnboot -r /dev/vg00/lvol3 #specifies the root LV lvlnboot -b /dev/vg00/lvol1 #specifies the boot LV lvlnboot -s /dev/vg00/lvol2 #specifies the swap LV lvlnboot -d /dev/vg00/lvol2 #specifies the dump LV (note the order -- it apparently matters!) lvlnboot -R #recovers any BDRAs lvlnboot -v # display Both disks must be listed for root, boot and swap definitions. Also, you can use -v option to have LVM report its activities. 10. test the mirroring: Run SAM. Go into Disk and File Systems and make sure there is a "1" under mirror copies for lvol1-lvol8. Shut down the system "reboot -h" and remove power from the root drive. Then power the system back on and interrupt the boot process. Type "sea" and boot from the mirror copy "bo p*". Interact with ISL> "n" The system should boot fine. Now shut down the system "reboot -h" and plug the power back into the root drive. Turn the system back on and it should boot from the primary boot path. There will be resyncing when the system comes up. Now that you have created mirror copies of the root logical volume and the primary swap logical volume, should either of the disks fail, the system can use the copy of root or of primary swap on the other disk and continue. When the failed disk comes back online, it will be automatically recovered, provided the system has not been rebooted. If the system is rebooted before the disk is back online, you will need to reactivate the disk and update the LVM data structures that track the disks within the volume group. See the following: ------------------------------------------------------------ Date: Sun, 9 Feb 1997 11:17:51 -0500 Submitted By: bobv@sbm-inc.com ========================== To replace a faulty root_mirror disk: ========================== Shutdown the system, have the customer engineer replace the faulty disk, and then boot the system in single user mode from the alternate boot disk. If you only have two disks in the root volume group, then you will need to override quorum as you boot. For our example: ISL> hpux -is (52.5.0;0)/stand/vmunix OR ISL> hpux -is -lq (52.5.0;0)/stand/vmunix (if you need to override quorum.) [Step 4.2] Restore the LVM configuration/headers onto the replaced disk from your backup of the LVM configuration: # vgcfgrestore -n /dev/rdsk/c0tXd0 where X is the Logical unit number of the disk that has been replaced. For our example: # vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t0d0 NOTE: You must have performed the command vgcfgbackup to save off the headers prior to the disk failure ( Refer to Appendix A ). [Step 4.3] Reactivate the volume group so that the new disk can be attached, since it wasn't configured in at boot time. This will also resync any mirrors that resided on the faulty disk. # vgchange -a y For our example, the volume group vg00 will already be activated, but it will not know of the replaced disk. Therefore, this step is still required so that LVM will now know that the disk is again available and the resync will occur: # vgchange -a y /dev/vg00 [Step 4.4] If you have any logical volumes that resided on the faulty disk that were NOT mirrored, you will need to recreate them and restore from backup. We typically oly have system lvols on the root disk and mirror them ALL, so this is not usually a problem. [Step 4.5] Bring system up into multiuser mode, allowing users on to access system. To do this, find out your normal run level ( first line of the /etc/inittab file ), and then perform a telinit command: # more /etc/inittab init:4:initdefault: .... .... # telinit 4 At this stage, your system should be fully recovered. [InternetShortcut] URL=3Dhttp://us-support.external.hp.com/atq/bin/doc.pl/sid=3D7ef89c0a06fb= ============================================================================= SBM NOTES Notes on disk quorum. LVM enforces a disk quorum policy such that if 50% or more physical volumes(disks) in a volume group are offline, the volume group will not be activated automatically at boot time. An example of this could be as simple as having lost a power supply in an external disk subsystem that housed the drives of one half of a mirror. If this should happen, the volume group can still be activated with the following command: vgchange -a y -q n /dev/vg01 The '-q n' option to the vgchange command tells LVM that we wish to ignore the quorum requirement. If quorum is not met on the Root Volume Group, then the system will halt at boot time after displaying the following message: lv_vgconf: root VG activation failed not enough PVs available You must reset the system and interact with the IPL in order to override the quorum requirement on the Root Volume Group. 1. Interrupt the autoboot process by pressing the spacebar on the console during the 10 second autoboot timout period. 2. Choose the boot device and answer 'Y' to the question "Do you want to interact with IPL?" 3. At the ISL> prompt type the following boot string: hpux -lq (52.6.0;2)/hp-ux You should substitute your appropriate hardware path for the value 52.6.0 used in the above example. The '-lq' option to the hpux utility causes LVM to ignore the quorum requirement for the Root Volume Group. These steps will be necessary at each boot until the missing disks are made available again. See "Chapter 8 Logical Volume Manager (LVM) Problems" of the book "Solving HP-UX Problems" for more information on this and other LVM related problems. ============================================================================= SBM NOTES Notes on physical volume groups The idea behind physical volume groups is to divide a set of disks that all belong to a single volume group into two subsets, where typically each subset is connected to a different SCSI bus than any other subsets. This makes it easier to configure the disk mirroring across multiple SCSI controllers. The HP documentation show how to create Physical Volume Groups with '-g' option of the vgcreate and vgextend commands. The information describing Physical Volume Groups is however maintained in a text file that can be edited whenever you wish to add disks into a PVG, not just at the time the volume group is created. See also lvmpvg(4). The following /etc/lvmpvg file divides the volume group /dev/vg00 into the two Physical Volume Groups scsi_1 and scsi_2. It is assumed here that disk units 0 and 1 are connected to one scsi controller and units 2 and 3 to a second controller. VG /dev/vg00 PVG scsi_1 /dev/dsk/c0d0s2 /dev/dsk/c1d0s2 PVG scsi_2 /dev/dsk/c2d0s2 /dev/dsk/c3d0s2 ============================================================ Date: Tue, 4 Nov 1997 12:39:15 -0500 Submitted By: rickg@sbm.com 1997/10/31 -rrg- Zeus mirror disk failure. It was discovered that one of the 4GB drives on the system Jamaica box was bad. The following log entry was discovered in syslog: Oct 10 14:27:40 zeus vmunix: LVM : Failure in attaching PV (8/8.12.0) to the root volume group. After attempts to restore the Logical Volumn information were unsuccessful, a new 4GB DE drive was pulled from stock and installed in place of the bad drive. As per HP support, the following commands were used to resync the backup system drive: vgconfgrestore -n /dev/vg00 /dev/rdsk/c3t12d0 mkboot /dev/rdsk/c3512d0 mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c3512d0 vgchange -a y /dev/vg00 vgsync /dev/vg00