Re: HACMP, hdisk with no VG assignment.

From: jkstevenson (jkstevenson@micron.com)
Date: Tue Oct 29 2002 - 12:14:22 EST


James,

There is a way of doing this but you must be exact in you key strokes.
Basically, It comes down to rebuilding the ODM to such a point that importvg
-L will work. But you are correct,in your current situation the only way to
re-import your vg is to varyoff the volume group on the service node.

If you want to leave it varyied on you will need to do the following steps:
If you are not comfortable with these commands or are unsure .. DO NOT DO
THEM... use the varyoffvg ... importvg .. it is the safest and supported.

#prod node

        varyonvg -bu 'volume_group_name'

        this will release the ssa locks placed against the current drives
        you will need to gather the VGID / vg name /PVID for one disk
currently in the disk group

#dev node

        you have already exported the volume group so the system should be
clean

        1) re-create the volume group name in the odm

        putlvodm -v volume_group_name -o -Q n volume_group_id

        2) re-create an initial volume group and pvid relationship

        putlvodm -p volume_group_id disk_pvid

        3) re-create the device file in the dev tree

        mknod /dev/volume_group_name c volumegroup_major_number 0 #use
same volumegroup_major_number as prod.

        4) At this point an lspv should show one hdisk assigned to the
volume group

           now redefine all the physical hdisk to the volume group

           redefinevg -d /dev/{hdisk # in lspv output} volume_group_name

           #after this step all the hdisk associated with the volume group
should now appear in the lspv output

        5) sync up the logical volume components

           importvg -L volume_group_name (hdisk # in lspv output}

        6) reset the HACMP lazy update setting ..

           clvgdats /dev/{hdisk # in lspv output} >
/usr/sbin/cluster/etc/vg/volume_group_name

#prod node:

        re-lock down the shared drives ..

        1) varyonvg volume_group_name

We have a 24X7 shop so taking down time is a complete killer ....... I have
written a complete script around this procedure to sync up drives after
people have worked on them ... If you check the HACMP documentation they
should have a procedure for LAZY UPDATE .. which would have worked before
the exportvg took place.

Jon Stevenson

-----Original Message-----
From: James Jackson [mailto:James.Jackson@MAIL.STATE.AR.US]
Sent: Tuesday, October 29, 2002 9:46 AM
To: aix-l@Princeton.EDU
Subject: Re: HACMP, hdisk with no VG assignment.

Simon,

Thanks for your thorough response. Your deductions regarding the cause of
the problem appear to be right on the money.

I'm guessing the only way to import the VG on the Dev. node is to vary it
off on the Prod. node, correct? Would it be absolutely necessary to stop
the cluster to do this? I've already exported the VG on the Dev. node, and
I'm stuck right now, since I apparently can't import it while it's active on
the Prod. node; mark it off to my lack of HACMP experience.

I'm doing all of this to prepare for a takeover test in advance of adding
some ESS SAN storage. I want to make sure everything is stable before I
start mucking with maintenance levels and device drivers.

I've found the HACMP manuals to be pretty useful, as well, though I wish
they went a bit deeper on troubleshooting.

As always, thanks for your well-reasoned responses to this list. I'll look
into the Yahoo! HACMP group.

Regards,

James Jackson

-----Original Message-----
From:
Sent: None
Subject: RE: HACMP, hdisk with no VG assignment.

X-Spam-Report:
CARRIAGE_RETURNS,EXCHANGE_SERVER,QUOTED_EMAIL_TEXT,SPAM_PHRASE_01_02
Message-ID: <D4AE64E989E7D411A8540001FAD48E2D07721070@KJSGBCHESHREXC2>
Newsgroups: bit.listserv.aix-l
Date: Tue, 29 Oct 2002 12:22:28 +0100
Reply-To: IBM AIX Discussion List <aix-l@Princeton.EDU>
Sender: IBM AIX Discussion List <aix-l@Princeton.EDU>
From: "Green, Simon" <SGreen@KRAFTEUROPE.COM>
Subject: Re: HACMP, hdisk with no VG assignment.
X-To: IBM AIX Discussion List <aix-l@Princeton.EDU>
To: aix-l@Princeton.EDU
Precedence: list
Return-Path: owner-aix-l@Princeton.EDU
X-OriginalArrivalTime: 29 Oct 2002 12:06:56.0898 (UTC)
FILETIME=[AD6DB620:01C27F43]

At some point in the past, someone has added another disk to the volume
group on the Production node. The volume group definitions now need to be
synchronised on each node in the cluster.

In all likelihood, in the event of a takeover "Lazy Update" would work:
HACMP determines that the VGDA has changed and re-imports it on the standby
node.
This adds a little time to the takeover. There are a few situations it
can't handle, though, so it's better to synchronise them when you have a
chance.

The simplest way is to stop your cluster, vary off the VG on your Production
node then import it on the Development node, using importvg -L VGName HDisk
where HDisk is one of the disks in the VG, (not the new one).

If you're feeling particularly paranoid - sorry; cautious - you could do an
exportvg on your Development node, then a normal importvg. If you do this,
be sure to specify the VG Major Number, (-V option).

It's worth checking the volume group at each end to make sure that they're
not set to vary on automatically at startup. I know a normal importvg sets
this
on; I'm not sure about importvg -L.

If it's mirrored, check the quorum, too.

Since you've just picked this up, it might be worth checking a few other
things. Make sure that the VG Major Number for each VG is the same on both
nodes, (ls -l /dev/VGName). No reason to think that they won't be, but
someone's been a bit sloppy about keeping them synchronised in the past, so
it won't hurt to check.

Also, do a full verify of the cluster. Easiest to do this through smit.
smitty hacmp; Cluster Configuration; Cluster Verification; Verify Cluster

Hopefully you have some documentation. If so, you ought to check that it's
accurate: network adapters defined as described etc. If it doesn't exist
you could make some now.

I'd quite like to do a takeover as well, to check that it works correctly
and make sure that I'd seen one from start to finish; it's good to see a
controlled takeover, so that if it happens for real you'll be able to
recognise anything out of the ordinary.

There are some HACMP Redbooks, but to be honest I haven't found them to be
very useful. Still; might be worth a look. The normal HACMP manuals are
quite good. The Configuration and Administration guide(s) are the most
useful.

If you start having to do much with this, there's a Yahoo! HACMP group which
you might find interesting. Not very active, but some useful expertise.

Simon Green
Philip Morris ITSC Europe

AIX-L Archive at http://marc.theaimsgroup.com/?l=aix-l&r=1&w=2
AIX FAQ at http://www.faqs.org/faqs/aix-faq/

N.B. Unsolicited email from vendors will seldom be appreciated.

-----Original Message-----
>From: James Jackson [mailto:James.Jackson@MAIL.STATE.AR.US]
>Sent: 28 October 2002 22:44

> I have inherited a two-node HACMP cluster with SSA disks. Two of the
volume
> groups on the Production node contain filesystems that are configured to
> failover to the Development node. However, one of the hdisks on the
> Development node does not have a corresponding reference to the volume
group
> on the Production node. Rather than belonging to VG "backvg", the hdisk
has
> a VG reference of "none". The hdisk does, however, reference the correct
> PVID. How should I go about correcting this?



This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 22:16:18 EDT