Configuration issues with a Hitachi 5800 fiber attached array and a Sun server running Solaris 2

Configuration issues with a Hitachi 5800 fiber attached array and a Sun server running Solaris 2.6 with Veritas Volume Manager 3.02 (supposedly the same for 3.0x less than 3.04).

The issue here is Volume Manager and Hitachi dealing with dual paths. With this model of the array (58xx) and Volume Manager 3.0x using DMP there is a problem that requires load balancing be turned off (the paths need to be configured as active/passive). This allows DMP to fail to the secondary path in the event that the primary fails but does not allow load balancing between the paths.

The Hitachi configuration for active/passive use on Solaris/Volume Manager is set with the following values in the array configuration (this was performed by Hitachi so I don’t have details on why or what most of the settings are). The information below was retrieved from Sommet and the page numbers are the same as shown in the Sommet GUI. Items that have BOX to the right are dialog boxes in the GUI. The indentations show boxes within a box and CHK to the left of an item indicates that this item was checked (selected) in the GUI (options not “checked” are left out below).

page 1

SYSTEM STARTUP ATTRIBUTE BOX

CHK Dual Active Mode

Data Share Mode BOX

CHK Not Used

Host Connection Mode BOX

CHK TRESPASS Mode

page 3

Option 1 BOX

SCSI/FIBRE-channel Common Option BOX

CHK VxVM mode enable

CHK SUN Solaris2.5.1 mode enable

page 8

INQUIRY Information BOX

Vendor Type "HITACHI"

Product Type "DF400"

Command Queuing BOX

CHK ON

Cache Mode BOX

CHK ALL OFF

page 9

Controller 0

Port Type BOX

CHK Normal

page 10

Controller 1

Port Type BOX

CHK Normal

To confuse the issue a little more, Veritas automatically sets up active/active communication if the array is fiber attached. So if the array is fiber attached you must lie and say that is a SCSI array in the model number information (drop the trailing f, for example DF400F becomes DF400.

Indications of a setup problem:

# vxdisk list c3t0d0

Device: c3t0d0s2

devicetag: c3t0d0

type: sliced

....

.... lots of other info

.... then at the end of the output are the interesting items for this issue

....

Multipathing information:

numpaths: 2

c3t0d0s2 state=enabled

c4t0d0s2 state=enabled

In the above output there is no indication of which path is primary and which is secondary. Below is what it should look like if properly configured:

# vxdisk list c3t0d0

Device: c3t0d0s2

devicetag: c3t0d0

type: sliced

....

.... lots of other info

.... then at the end of the output are the interesting items for this issue

....

Multipathing information:

numpaths: 2

c3t0d0s2 state=enabled type=primary

c4t0d0s2 state=enabled type=secondary

Another indication of a problem is the performance. I cp'd a 6GB directory from one volume configured on the array to another volume on the array while running iostat -xn (below).

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0

0.0 21.5 0.0 1341.9 0.0 2.6 0.2 120.3 0 96 c3t0d0

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d1

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d2

0.0 20.4 0.0 1293.4 0.0 2.6 0.0 127.1 0 96 c4t0d0

In addition to the bad performance note that both paths are being used. Below is an example of a proper setup.

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0

0.0 317.0 0.0 20253.2 0.0 0.6 0.0 2.0 0 54 c3t0d0

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d1

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c3t0d2

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0

In addition to the performance boost note that a single path is in use.

For fiber attached arrays, the following command can be used to verify that Solaris/Volume Manager see’s the array as SCSI instead of fiber.

# format -e

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c1t0d0 <SUN18G cyl 7506 alt 2 hd 19 sec 248>

/pci@3,2000/scsi@2/sd@0,0

. more stuff

12. c3t0d0 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,0

13. c3t0d1 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,1

14. c3t0d2 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,2

15. c3t0d3 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,3

16. c3t0d4 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,4

17. c3t0d5 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@2,2000/fibre-channel@2/sd@0,5

18. c4t0d0 <HITACHI-DF400F-0000 cyl 3637 alt 2 hd 50 sec 768>

/pci@6,2000/fibre-channel@2/sd@0,0

.more stuff

Specify disk (enter its number): 12

selecting c3t0d0

[disk formatted]

FORMAT MENU:

disk - select a disk

type - select (define) a disk type

partition - select (define) a partition table

current - describe the current disk

format - format and analyze the disk

repair - repair a defective sector

label - write label to the disk

analyze - surface analysis

defect - defect list management

backup - search for backup labels

verify - read and display labels

save - save new disk/partition definitions

inquiry - show vendor, product and revision

scsi - independent SCSI mode selects

volname - set 8-character volume name

!<cmd> - execute <cmd>, then return

quit

format> scsi

Warning: these functions are intended for expert use only, for

debugging disk devices and for unusual configuration settings.

It is recommended that you do not use this menu for normal disk

configuration and formatting, unless you have explicit instructions,

or know exactly what you are doing.

SCSI MENU:

p<n> - display a mode sense page

p<n> b<n> <op> [~]<n> - change a byte and issue mode select

b<n> <op> [~]<n> - add an operation to the mode select list

for the current page

where: p<n> specifies the page with page code <n>

b<n> specifies byte <n> of the page

<op> can be one of the following operators:

= (set specified value)

|= (bitwise OR with current value)

&= (bitwise AND with current value)

<n> can be a decimal value in the range 0-255,

or two hexadecimal digits, in the form 0x<xx>.

[~] complements the specified value

apply - apply mode select list

cancel - cancel mode select list

display - display mode select list

all - display all supported mode sense pages

default p<n> - mode select page <n> to default values

default all - mode select all pages to default values

format - format without standard mode selects

inquiry - display device's inquiry response

list - list common SCSI-2 mode pages

!<cmd> - execute <cmd> , then return

quit

scsi> inquiry 44 46 34 30 30 20 20 20 20 20 20 20 20 20 20 20 DF400

30 30 30 30 31 30 38 31 33 2d 30 30 30 30 30 000010813-00000

Vendor: HITACHI

Product: DF400

Revision: 0000

Removable media: no

Device type: 0

ISO version: 0

ECMA version: 0

ANSI version: 2

Async event notification: no

Terminate i/o process msg: no

Response data format: 2

Additional length: 43

Relative addressing: no

32 bit transfers: no

16 bit transfers: yes

Synchronous transfers: yes

Linked commands: yes

Command queueing: yes

Soft reset option: no

The key information is the Product line (note it is DF400 instead of DF400F).