From: Bruce Shaw (Bruce.Shaw@gov.ab.ca)
Date: Thu Dec 28 2006 - 13:46:35 EST
The problem turned out to be that the Qlogic cards being used by Solstice
disksuite were being hi-jacked by mpxio.
Here's the appropriate entries from the various files involved (all in
/kernel/drv). In this case we're attaching an HP SAN, presenting multiple
LUNs up to 6. There's two Qlogic fibre cards controlling the internal
drives, and they display on separate controllers. If you're using the
optional wiring/configuration where they all appear on one, you only need
one entry in qlc.conf.
#################################fp.conf#################################
name="fp" class="fibre-channel" port=0;
name="fp" class="fibre-channel" port=1;
name="fp" class="fibre-channel" port=2;
name="fp" class="fibre-channel" port=3;
name="fp" class="fibre-channel" port=4;
name="fp" class="fibre-channel" port=5;
#
# List of ULP modules for loading during port driver attach time
#
load-ulp-list="1","fcp";
scsi-binding-set="fcp";
#################################end
fp.conf#################################
#################################bottom of
qlc.conf#################################
# the following lines disable multiplexing on the internal FC-AL drives,
otherwise
# half of them disappear on reboot
name="qlc" parent="/pci@8,600000" unit-address="2" mpxio-disable="yes";
name="qlc" parent="/pci@8,700000" unit-address="4" mpxio-disable="yes";
#################################end
qlc.conf#################################
#################################scsi_vhci.conf#############################
####
name="scsi_vhci" class="root";
mpxio-disable="no";
load-balance="round-robin";
auto-failback="enable";
device-type-scsi-options-list =
"HP HSV200", "symmetric-option";
symmetric-option = 0x1000000;
#################################end
scsi_vhci.conf#################################
For some reason the last four lines have to be pretty much verbatim. I had
a comment
for the second last line and it failed. For some reason the blank line
needs to be there.
With this files in place, do a boot -r. If you see a whole bunch of disks
in 'format' instead of what' you've presented from the SAN (usually four
times what you'd expect to see) mpxio didn't start correctly, usually
because of a configuration problem. If there's SAN resources missing, its
generally because they're on an unrecognized LUN, hence the mods to fp.conf.
If some of your local hard drives disappear, it's a problem with Qlogic
configuration.
-----Original Message-----
From: sunmanagers-bounces@sunmanagers.org
[mailto:sunmanagers-bounces@sunmanagers.org] On Behalf Of Bruce Shaw
Sent: 13 December 2006 01:23
To: 'sunmanagers@sunmanagers.org'
Subject: configuring mpxio for emulex HBA's on a V880
I'm attempting to install Emulex HBA's on a V880 under Solaris 8 using
SUNWsan.
Boot disks etc. are mirrored using Solstice disksuite.
If I configure /kernel/drv/scsi_vhci.conf with:
################################################################
mpxio-disable="no";
load-balance="round-robin";
auto-failback="enable";
device-type-scsi-options-list =
"HP HSV200", "symmetric-option";
symmetric-option = 0x1000000;
################################################################
...half my local hardware disks disappear from format, Solstice
disksuite fails and files get corrupted and cannot be removed and
generate I/O errors.
I'm presuming I have to configure scsi_vhci.conf with
mpxio-disable="yes"
and individually configure multiplex devices within /kernel/drv/fp.conf.
Format reports:
0. c1t0d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68e801,0
1. c1t1d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68e737,0
2. c1t2d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf279fb8,0
3. c1t3d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68e0b8,0
4. c1t4d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68e928,0
5. c1t5d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68df99,0
6. c1t8d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf6824ed,0
7. c1t9d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf682648,0
8. c1t10d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf2795e3,0
9. c1t11d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf681059,0
10. c1t12d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf681ce0,0
11. c1t13d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf681ea5,0
for internal FC/AL drives.
As it stands, fp.conf contains:
#################################################
name="fp" class="fibre-channel" port=0;
name="fp" class="fibre-channel" port=1;
load-ulp-list="1","fcp";
scsi-binding-set="fcp";
#################################################
/usr/platform/sun4u/prtdiag -v reports:
########################################################################
####
########
========================= IO Cards =========================
Bus Max
IO Port Bus Freq Bus Dev,
Brd Type ID Side Slot MHz Freq Func State Name Model
---- ---- ---- ---- ---- ---- ---- ---- -----
-------------------------------- ----------------------
I/O PCI 8 B 1 33 33 4,0 ok pci-pci8086,b154.0/network
(netw+ PCI-BRIDGE
I/O PCI 8 B 1 33 33 0,0 ok network-pci108e,abba.20
SUNW,pci-ce/pci-bridge
I/O PCI 8 B 0 33 33 5,0 ok
scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 8 B 0 33 33 5,1 ok
scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 9 B 6 33 33 2,0 ok
fibre-channel-pci10df,fa00.10df.+ LP10000
I/O PCI 9 B 5 33 33 3,0 ok
fibre-channel-pci10df,fa00.10df.+ LP10000
########################################################################
####
########
prtpicl -v reports:
########################################################################
####
########
fibre-channel (obp-device, 19000001ecf)
:DeviceID 0x2
:UnitAddress 2
:vendor-id 0x10df
:device-id 0xfa00
:revision-id 0x1
:subsystem-vendor-id 0x10df
:subsystem-id 0xfa00
:min-grant 0xff
:max-latency 0
:cache-line-size 0x10
:latency-timer 0x40
:interrupts 00 00 00 01
:class-code 0xc0400
:devsel-speed 0x1
:fast-back-to-back
:66mhz-capable
:compatible (19000001eedTBL)
| pci10df,fa00.10df.fa00.1 |
| pci10df,fa00.10df.fa00 |
| pci10df,fa00 |
| pci10df,fa00.1 |
| pci10df,fa00 |
| pciclass,0c0400 |
| pciclass,0c04 |
:reg
00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00
03 00 10 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20 00
03 00 10 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20 00
01 00 10 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01 00
:assigned-addresses
83 00 10 10 00 00 00 00 00 12 40 00 00 00 00 00 00 00
20 00
83 00 10 18 00 00 00 00 00 12 60 00 00 00 00 00 00 00
20 00
81 00 10 20 00 00 00 00 00 00 03 00 00 00 00 00 00 00
01 00
:num-nodes 512
:pm-support 0
:adisc-support 1
:assign-alpa 0
:cr-count 1
:cr-delay 0
:ack0 0
:link-speed 0
:topology 0
:network-on 1
:ub-bufs 1000
:num-iocbs 1024
:log-errors -1
:log-warnings -1
:log-notices -1
:console-errors 0
:console-warnings 0
:console-notices 0
:model LP10000
:devfs-path /pci@9,700000/fibre-channel@2
:driver-name emlxs
:binding-name pci10df,fa00
:bus-addr 2
:instance 0
:_class obp-device
:name fibre-channel
fp (devctl, 19000001f03)
:scsi-binding-set fcp
:load-ulp-list (19000001f13TBL)
| 1 |
| fcp |
:class fibre-channel
:port 0
:pm-components (19000001f0cTBL)
| NAME=FC Port |
| 0=Port Down |
| 1=Port Up |
:devfs-path /pci@9,700000/fibre-channel@2/fp@0,0
:driver-name fp
:binding-name fp
:bus-addr 0,0
:instance 1
:_class devctl
:name fp
fibre-channel (obp-device, 19000001f17)
:DeviceID 0x3
:UnitAddress 3
:vendor-id 0x10df
:device-id 0xfa00
:revision-id 0x1
:subsystem-vendor-id 0x10df
:subsystem-id 0xfa00
:min-grant 0xff
:max-latency 0
:cache-line-size 0x10
:latency-timer 0x40
:interrupts 00 00 00 01
:class-code 0xc0400
:devsel-speed 0x1
:fast-back-to-back
:66mhz-capable
:compatible (19000001f35TBL)
| pci10df,fa00.10df.fa00.1 |
| pci10df,fa00.10df.fa00 |
| pci10df,fa00 |
| pci10df,fa00.1 |
| pci10df,fa00 |
| pciclass,0c0400 |
| pciclass,0c04 |
:reg
00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00
03 00 18 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20 00
03 00 18 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20 00
01 00 18 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01 00
:assigned-addresses
83 00 18 10 00 00 00 00 00 12 80 00 00 00 00 00 00 00
20 00
83 00 18 18 00 00 00 00 00 12 a0 00 00 00 00 00 00 00
20 00
81 00 18 20 00 00 00 00 00 00 04 00 00 00 00 00 00 00
01 00
:num-nodes 512
:pm-support 0
:adisc-support 1
:assign-alpa 0
:cr-count 1
:cr-delay 0
:ack0 0
:link-speed 0
:topology 0
:network-on 1
:ub-bufs 1000
:num-iocbs 1024
:log-errors -1
:log-warnings -1
:log-notices -1
:console-errors 0
:console-warnings 0
:console-notices 0
:model LP10000
:devfs-path /pci@9,700000/fibre-channel@3
:driver-name emlxs
:binding-name pci10df,fa00
:bus-addr 3
:instance 1
:_class obp-device
:name fibre-channel
fp (devctl, 19000001f4b)
:scsi-binding-set fcp
:load-ulp-list (19000001f5bTBL)
| 1 |
| fcp |
:class fibre-channel
:port 0
:pm-components (19000001f54TBL)
| NAME=FC Port |
| 0=Port Down |
| 1=Port Up |
:devfs-path /pci@9,700000/fibre-channel@3/fp@0,0
:driver-name fp
:binding-name fp
:bus-addr 0,0
:instance 2
:_class devctl
:name fp
########################################################################
####
########
So what I presumably need to do in fp.conf is to explicitly disable the
onboard fibre controller so it doesn't multiplex the internal disks, but
enable the specific entries for the SAN.
The documentation example shows:
mpxio-disable="yes";
name="fp" parent="/pci@13,2000/pci@2/SUNW,qlc@5" port=0
mpxio-disable="no";
but just to be on the safe side (yes, after trying to back out of the
last install I'm paranoid) I'd like to explicitly disable the onboard
internal drives eg.
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cf68e801,0
but I don't understand what to use for a port number.
The SAN is going to be broadcasting on LUN 1 and 2. Does that translate
into "port=1" and "port=2"?
Or, in short, what should my fp.conf entries look like to explicity
disable multiplexing on the internal hard drives, but enable it on the
SAN connections?
Finally, is there any way of getting mpxio going without a
reconfiguration reboot? It's almost impossible to get downtime on this
machine.
-- This communication is intended for the use of the recipient to which it is addressed, and may contain confidential, personal and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communication received in error, or subsequent reply, should be deleted or destroyed. _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers This communication is intended for the use of the recipient to which it is addressed, and may contain confidential, personal and or privileged information. Please contact us immediately if you are not the intended recipient of this communication, and do not copy, distribute, or take action relying on it. Any communication received in error, or subsequent reply, should be deleted or destroyed. _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:41:24 EDT