Poor Man's Guide to Failover

A Poor Man's Guide to Fail-Over.

or
How to Fail Over Systems Using Shared Storage but not Using Clustering Software

Conributed by Brad Leak

Manual Fail Over Steps:

Assumptions:
Two hosts with shared storage. The hosts are named hostA and hostB.
Volume Management using Veritas File System and Volume Manager
Two disk groups: "rootdg" and "data_dg"

rootdg contains normal local system volumes (rootvol, swapvol, var, etc.) data_dg contains one volume named data01_vol
data01_vol will be mounted on /opt/data01

The local (maintenance) IP of hostA is 192.68.13.10.
The local (maintenance) IP of hostB is 192.68.13.11.
Another virtual IP and hostname is reserved for access. The IP is 192.68.13.12. The hostname is hostV.

This simply maps a diskgroup failover. I'll include something like NFS and Oracle later.

===SUMMARY OF STEPS:===
/* On failed hostA: */
Stop application(s) in question.
Unmount volume(s).

Stop volume(s).
Deport diskgroup(s).
Un-configure Virtual IP address. /* On hostB: */
Configure Virtual IP address.
Import diskgroup(s).
Start volume(s).
fsck the volume(s).
Mount volume(s).
Start application(s) in question.

===DETAILED STEPS:===

=BEFORE FAIL OVER=
=hostA:=

ifconfig -a

hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500

inet 192.68.13.10 netmask fffffe00 broadcast 10.0.5.255

hme0:1: flags=843<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.68.13.12 netmask fffffe00 broadcast 10.0.5.255

cat /etc/hosts
127.0.0.1 localhost loghost
192.68.13.10 hostA
192.68.13.11 hostB
192.68.13.12 hostV
~
~
~
uname -n
hostA
vxdg list
NAME STATE ID

rootdg enabled 963868050.1025.hostA

data_dg enabled 963925176.1319.hostA
vxprint -g data_dg

TY NAME          ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg data_dg       data_dg      -        -        -        -        -       -

dm data_dg01     c1t0d0s2     -        35282944 -        -        -       -

v  data01_vol    fsgen       ENABLED   70578176 -        ACTIVE   -       -
pl data01_vol-01 data01_vol  ENABLED   70578176 -        ACTIVE   -       -
sd data_dg-01    data01_vol-01 ENABLED 70578176 0        -        -       -

df -k /opt/data01

Filesystem kbytes used avail capacity Mounted on

/dev/vx/dsk/data_dg/data01_vol 8796160 370208 8360136 5% /opt/data01
ping hostV
hostV is alive
format
Searching for disks...done
AVAILABLE DISK SELECTIONS:

0. c0t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>

/pci@1f,4000/scsi@3/sd@0,0

1. c1t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>

/pci@1f,2000/scsi@1/sd@0,0

=hostB:=

ifconfig -a
hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
inet 192.68.13.11 netmask fffffe00 broadcast 10.0.5.255
cat /etc/hosts
127.0.0.1 localhost loghost
192.68.13.10 hostA
192.68.13.11 hostB
192.68.13.12 hostV
~
~
~
uname -n
hostB
vxdg list

NAME STATE ID

rootdg enabled 963868050.1025.hostB
df -k /opt/data01

Filesystem kbytes used avail capacity Mounted on

/dev/vx/dsk/rootvol 482824 308484 126058 71% /

/* Notice that since /opt/data01 is not currently a mount point, rootvol is reported. */
ping hostV
hostV is alive
format
Searching for disks...done
AVAILABLE DISK SELECTIONS:

0. c0t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>

/pci@1f,4000/scsi@3/sd@0,0

1. c1t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>

/pci@1f,2000/scsi@1/sd@0,0

=START FAIL OVER=

hostA:
/*      Stop application(s) in question.        */
# umount /opt/data01

# vxvol -g data_dg stopall

# vxdg deport data_dg

# vxdg list


NAME          STATE           ID
rootdg        enabled  963868050.1025.hostA

ifconfig hme0:1 unplumb
ifconfig -a hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500 inet 192.68.13.10 netmask fffffe00 broadcast 10.0.5.255
ping hostV
no response from hostV

hostB:

ifconfig hme0:1 plumb
ifconfig hme0:1 inet 192.68.13.12 netmask 255.255.254.0 broadcast + up
ifconfig -a
hme0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
inet 192.68.13.11 netmask fffffe00 broadcast 10.0.5.255
hme0:1: flags=843<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.68.13.12 netmask fffffe00 broadcast 10.0.5.255
ping hostV
hostV is alive
vxdg import data_dg
vxdg list

NAME STATE ID
rootdg enabled 963868050.1025.hostB

data_dg enabled 963925176.1319.hostA
vxvol -g data_dg startall
fsck -F vxfs /dev/vx/rdsk/data_dg/data01_vol
vxprint -g data_dg

TY NAME          ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg data_dg       data_dg      -        -        -        -        -       -

dm data_dg01     c1t0d0s2     -        35282944 -        -        -       -

v  data01_vol    fsgen       ENABLED   70578176 -        ACTIVE   -       -
pl data01_vol-01 data01_vol  ENABLED   70578176 -        ACTIVE   -       -
sd data_dg-01    data01_vol-01 ENABLED 70578176 0        -        -       -

# mount -F vxfs /dev/vx/dsk/data_dg/data01_vol /opt/data01

/*      Start application(s) in question.               */

NAME STATE ID
rootdg	enabled	963868050.1025.hostA
data_dg	enabled	963925176.1319.hostA

Filesystem	kbytes	used	avail	capacity	Mounted on
/dev/vx/dsk/data_dg/data01_vol	8796160	370208	8360136	5%	/opt/data01

NAME	STATE	ID
rootdg	enabled	963868050.1025.hostB
data_dg	enabled	963925176.1319.hostA