From: Steve Edberg (sbedberg@ucdavis.edu)
Date: Mon Jun 06 2005 - 09:18:47 EDT
To reiterate my problem:
I had a misconfigured mirror metadevice under Solaris 8/DiskSuite -
the /opt filesystem was mounted on a raw partition making up the
mirror (c1t0d0s5) instead of the metadevice (d5) itself. I had tried
simply unmounting /opt and remounting it on the mirror device, but
that resulted in a corrupted directory structure; doing a subsequent
metadetach & metattach did not help.
My setup was: metadevice d5 was a mirror, consisting of two
submirrors d15 (c1t0d0s5) and d25 (c1t0d1s5).
As it turns out, I was correct in the commands I needed but wrong
about the sequence. What I did successfully was to (1) detach
submirror d25, leaving only the single submirror d15 (which was the
disk slice I had been using); (2) unmount /opt, remount onto the
mirror d5; and (3) attach d25 back onto d5. After a few minutes of
resyncing, all looks OK.
Now, I have to do this for /usr. My plan is to (1) edit vfstab so
that /usr points to the mirror, (2) detach the second submirror, (3)
reboot, (4) reattach the second submirror. After, of course, a
verified backup. I also have to do this for swap.
Thanks to:
Ken Rossman
Wesley W. Garland
Peter Junker
Thomas M. Payerle
Alex Stade
Darren Dunham
In addition, I found this
http://www.blacksheepnetworks.com/security/resources/labs/disksuite.html
- which summarizes disksuite usage well. And this -
http://www.physics.umd.edu/rgroups/pnce/pcs-docs/Glue/disksuite/disksuite-boot-mirror.html
- which calls out some warnings about what I did wrong in the first
place (not detaching mirrors before remounting onto the mirror).
steve edberg
=======================
= Summary of replies: =
=======================
The thing to do here is to break the mirror (metadetach), then reattach it
(metattach). You have to decide which side of the mirror is more correct
in this case, and leave that submirror attached to the mirror metadevice,
while detaching the other submirror. But I'd make the assumption in this
case that it was the physical device that *was* mounted directly that you'd
consider correct in this case.
---------------------------------------------------------------
This should do the trick:
metareplace -e d5 c1t1d0s5
then "metastat | grep %" -- if it says "Resyncing, 0% done" you're golden.
PS: Backup first!!
---------------------------------------------------------------
How to get issues with volumes under Solstice Diksmanager fixed lookup
up the documentation.
If you don't have it, the lookit up under http://docs.sun.com
Dealing with the file systems /, /usr, /opt and swap needs
special steps, otherwise there is going to be file system corruption.
---------------------------------------------------------------
I thinkyour bestbet is to get all problematic filesystems "unmirrored",
mounted using the physical device, then metadetach all submirrors but the
one corresponding to the physical device you were using, then metaclear
the mirror metadevice.
Then just "remirror"; make a submirror metadevice for the physical device in
use (this may be redundant, as probably still around) (if you do make
it, use the -f option as it should be already mounted). Create the
mirror
metadevice with only a single submirror (the one corresponding to the mounted
physicsal device). Now you must remount the filesystem so the mirror
metadevice
is the mount point. Then you can attach other submirrors.
---------------------------------------------------------------
I'll explain briefly what I believe has happened, and then tell you
my conclusion.
When you first do a;
metainit -f d15 1 1 c1t0d0s5
metainit d5 -m d15
You're telling SDS/SVM that you've create a mirror and defined which
drive to mirror _from_. When you then;
metainit d25 1 1 c1t1d0s5
metattach d5 d25
SDS/SVM knows which device to sync from. However, once sync is
completed, things change. Unless you specified -g and/or -r when you
created the mirror (from metainit(1M));
The following read options for mirrors are supported:
-g Enables the geometric read option, which
results in faster performance on sequential
reads.
-r Directs all reads to the first submirror. This
should only be used when the devices comprising
the first submirror are substantially faster
than those of the second mirror. This flag can-
not be used with the -g flag.
You'll end up with the default, which is a round-robin style read
(from metainit(1M));
If neither the -g nor -r flags are specified, reads are
made in a round-robin order from all submirrors in the
mirror. This enables load balancing across the submir-
rors.
So. My theory is that you're reading in a round-robin fashion, where
the data on d25 is considerably older than the data on d15.
As for what you can do, I would (since you have a good backup)
attempt to detach d25 from d5 and re-attach it;
metadetach d5 d25
metattach d5 d25
That should sync up the mirror from d15.
---------------------------------------------------------------
<SNIP>
> My question is: is there an easy way to refresh a metadevice from a
> physical slice so that I can simply remount onto the mirror device?
Yes. metadetach the side that's not mounted, remount onto the mirror,
then metaattach the component back.
If you do this later, then corruption will be rewritten onto both sides
of the mirror.. *ouch*.
I
> know I could simply do a newfs on the d5 metadevice and restore the
> backup, but I'd like to avoid that (especially since the /usr
> partition is misconfigured this way as well). I assume pointing swap
> at a metadevice will be no problem by editing /etc/vfstab & rebooting
> will not be a problem since there's nothing persistent on swap, but
> perhaps I'm missing something.
The most you could do at this point is to resync to one side (sounds
like you've done that) and fsck. But that will only make everything
consistent, not necessarily "right". The problem is that the corruption
has already occured, and probably been rewritten onto the disk....
</SNIP>
=====================
= Original message: =
=====================
Hi -
I have a Sparc/Solaris 8 system with some mirror metadevices set up
using Solaris Disksuite. However, some of the filesystems were
mounted via /etc/vfstab onto one of the physical disk slices that
made up the mirror, rather than the mirror metadevice.
To remedy this, I first tested by unmounting the /opt filesystem
(because my /opt was small, I had a current verified backup and there
was nothing critical there), and remounting it onto the mirror
device. In summary:
/opt originally mounted on /dev/dsk/c1t0d0s5
c1t0d0s5 one part of mirror d5; metastat -i d5 reports all ok
umount /opt
mount -F ufs /dev/md/dsk/d5 /opt
Now, I took a quick look at /opt and things *seemed* ok. However,
problems soon became evident; eg, ls and ls -l reported different
files/directories, trying to cd into some subdirectories of /opt gave
me an IO error, recent files disappeared. I remounted /opt back onto
the physical partition c1t0d0s5, and everything was OK again. It was
as if the directory structure of the metadevice had not been updated,
so anything (directories & files) that was older than approximately a
year was visible. I am currently unable to trace things back to *why*
the metadevice directory was updated then; it was over a year since
the metadevices had been created, and much older than the last
reboot; it *may* have been about the same time that the filesystems
last needed fscking (thank <deity> for UFS logging!).
I had checked metastat -i, and mirror were reported all ok, so I
assumed that things would be fine. I was wrong. I tried again doing a
metadetach and metattach in hopes that would resync the mirror's
directory, that didn't help either.
My question is: is there an easy way to refresh a metadevice from a
physical slice so that I can simply remount onto the mirror device? I
know I could simply do a newfs on the d5 metadevice and restore the
backup, but I'd like to avoid that (especially since the /usr
partition is misconfigured this way as well). I assume pointing swap
at a metadevice will be no problem by editing /etc/vfstab & rebooting
will not be a problem since there's nothing persistent on swap, but
perhaps I'm missing something.
Thanks in advance; will summarize -
steve edberg
Some details:
uname -a
========
SunOS pgfjupiter 5.8 Generic_117000-03 sun4u sparc SUNW,Sun-Fire-280R
Excerpt from current /etc/vfstab:
=================================
# device device mount FS fsck mount mount
# to mount to fsck point type pass at boot options
################# ################## ############ ##### # ###
#############
/dev/dsk/c1t0d0s1 - - swap - no -
/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no logging
/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /usr ufs 1 no logging
/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /opt ufs 2 yes logging
metastat -p d5
==============
d5 -m d15 d25 1
d15 1 1 c1t0d0s5
d25 1 1 c1t1d0s5
(swap, /, /opt and /usr were all intended to be mounted on mirrors
d1, d0, d5 & d6 respectively; only / is actually currently running on
the mirror metadevice).
metastat -i
===========
d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8389656 blocks
d10: Submirror of d0
State: Okay
Size: 8389656 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s0 0 No Okay
d20: Submirror of d0
State: Okay
Size: 8389656 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t1d0s0 0 No Okay
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d21
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 1048707 blocks
d11: Submirror of d1
State: Okay
Size: 1048707 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s1 0 No Okay
d21: Submirror of d1
State: Okay
Size: 1048707 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t1d0s1 0 No Okay
d3: Mirror
Submirror 0: d13
State: Okay
Submirror 1: d23
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2889 blocks
d13: Submirror of d3
State: Okay
Size: 2889 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s3 2889 Yes Okay
d23: Submirror of d3
State: Okay
Size: 2889 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t1d0s3 2889 Yes Okay
d4: Mirror
Submirror 0: d14
State: Okay
Submirror 1: d24
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 2889 blocks
d14: Submirror of d4
State: Okay
Size: 2889 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s4 2889 Yes Okay
d24: Submirror of d4
State: Okay
Size: 2889 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t1d0s4 2889 Yes Okay
d5: Mirror
Submirror 0: d15
State: Okay
Submirror 1: d25
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8389656 blocks
d15: Submirror of d5
State: Okay
Size: 8389656 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t0d0s5 0 No Okay
d25: Submirror of d5
State: Okay
Size: 8389656 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c1t1d0s5 0 No Okay
<SNIP>
-- +--------------- my people are the people of the dessert, ---------------+ | Steve Edberg http://pgfsun.ucdavis.edu/ | | UC Davis Genome Center sbedberg@ucdavis.edu | | Bioinformatics programming/database/sysadmin (530)754-9127 | +---------------- said t e lawrence, picking up his fork ----------------+ _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:30:49 EDT