Problem with 3510 FC Array State Replica's

From: Thor Newman (Thor@airg.com)
Date: Thu Mar 17 2005 - 19:49:20 EST

Next message: Darren Brechman-Toussaint: "Ufsdump to remote tape device"
Previous message: Ryan Krenzischek: "Summary: IPMP on Boot comes up with Failed Interfaces (Sol9)..."
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Hello managers,

I am in the process of building a SunFire E2900 with 3510FC RAID for
external storage, connected via 2GB Host Adapter (SG-XPCI2FC-QF2).
Solaris itself is installed on the two internal HDD which the E2900 has
; the RAID's are intended for database partitions.

Each 3510 provides a hardware RAID LUN to a separate controller in the
E2900, I then use Solaris VM to mirror the LUN's.

The problem I am having is not with the mirroring itself, this is fine,
it's with 'metadb' state replicas put on 3510 LUN's.

I have state replicas on the two internal HDD, and these are fine. I set
aside a small slice (20mb or so) on each LUN for additional state
replicas, and added them with metadb, and there was no problem with
this. The replicas were added, the VM objects were created, the
partitions mounted, everyone was happy, and all was well.

The problem arose after rebooting the server; the 3510 state replicas
failed or were otherwise reported as corrupted. The server would not
boot unaided, but prompted for single user mode to perform maintenance.
I had to delete state replicas on the 3510 devices in order to clear the
error and continue with a successful boot. I went through this twice and
verified patches, drivers, etc, each time.

- The replicas on the E2900 internal HDD are always fine.

- The internal drives themselves are mirrored with VM as well
as the RAID LUN's.

- There are no issues with the setup as long as I don't add
state replicas to the 3510 devices; if I do, all is well until a reboot.

- When I reboot, the server prompts for maintenance and reports
problems with the 3510 objects and corruption issues with 3510 state
replicas. Deleting the 3510 state replicas corrects the issues, and the
server continues booting. The 3510 metadevices mount correctly, no data
is lost or corrupted.

So the problem appears to be the state replicas only. The 3510 is
supposed to interop with Solaris VM, but perhaps only when configured
with JBOD? Is their a problem putting a state replica on a slice on a HW
RAID LUN? Driver issues? Any suggestions / notes / recommendations /
links / etc will be greatly appreciated.

Here's the sort of errors I get:

Mar 17 19:39:44 e2900-01 genunix: [ID 936769 kern.info] ssd1 is
/ssm@0,0/pci@18,700000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff884c46,1

Mar 17 19:39:44 e2900-01 scsi: [ID 799468 kern.info] ssd0 at fp0: name
w216000c0ff884c46,0, bus address a7

Mar 17 19:39:44 e2900-01 genunix: [ID 936769 kern.info] ssd0 is
/ssm@0,0/pci@18,700000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff884c46,0

Mar 17 19:39:44 e2900-01 genunix: [ID 408114 kern.info]
/ssm@0,0/pci@18,700000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff884c46,1 (ssd1)
online

Mar 17 19:39:44 e2900-01 genunix: [ID 454863 kern.info] dump on
/dev/md/dsk/d60 size 8193 MB

Mar 17 19:39:46 e2900-01 pseudo: [ID 129642 kern.info] pseudo-device:
devinfo0

Mar 17 19:39:46 e2900-01 genunix: [ID 936769 kern.info] devinfo0 is
/pseudo/devinfo@0

Mar 17 19:39:46 e2900-01 genunix: [ID 511416 kern.notice]
e_ddi_get_dev_info: Illegal major device number <-1>

Mar 17 19:39:46 e2900-01 md_mirror: [ID 437521 kern.info] NOTICE: md:
d69: B_FAILFAST I/O disabled

Mar 17 19:39:46 e2900-01 md_mirror: [ID 976326 kern.warning] WARNING: md
d70: open error on (Unavailable)

Mar 17 19:39:46 e2900-01 genunix: [ID 511416 kern.notice]
e_ddi_get_dev_info: Illegal major device number <-1>

Mar 17 19:39:46 e2900-01 md_mirror: [ID 437521 kern.info] NOTICE: md:
d69: B_FAILFAST I/O disabled

Mar 17 19:39:46 e2900-01 md_mirror: [ID 976326 kern.warning] WARNING: md
d70: open error on (Unavailable)

Mar 17 19:39:46 e2900-01 genunix: [ID 511416 kern.notice]
e_ddi_get_dev_info: Illegal major device number <-1>

Mar 17 19:39:46 e2900-01 md_mirror: [ID 437521 kern.info] NOTICE: md:
d79: B_FAILFAST I/O disabled

Mar 17 19:39:46 e2900-01 md_mirror: [ID 976326 kern.warning] WARNING: md
d80: open error on (Unavailable)

Mar 17 19:39:46 e2900-01 genunix: [ID 511416 kern.notice]
e_ddi_get_dev_info: Illegal major device number <-1>

Mar 17 19:39:46 e2900-01 md_mirror: [ID 437521 kern.info] NOTICE: md:
d79: B_FAILFAST I/O disabled

Mar 17 19:39:46 e2900-01 md_mirror: [ID 976326 kern.warning] WARNING: md
d80: open error on (Unavailable)

Mar 17 19:39:46 e2900-01 pseudo: [ID 129642 kern.info] pseudo-device:
sgfru0

Mar 17 19:39:46 e2900-01 genunix: [ID 936769 kern.info] sgfru0 is
/pseudo/sgfru@0