How do I manage multi

November 2, 2009 8:07 PM

Seemingly arcane Solaris I/O Multipathing (MPxIO) question - how do I remove a physical device on a MPxIO-managed controller?

I am a Solaris newbie. I have a Solaris 10u8 machine that is running an attached J4400 and some internal drives. We're using multipathed SAS I/O (enabled via stmsboot), so the device mount points have been moved off from their "normal" c0t5d0 to long strings -- in the case of c0t5d0, it's now /dev/rdsk/c6t5000CCA00A274EDCd0. (I can see the cross-referenced devices with stmsboot -l0.)

Normally, when replacing a disk on a Solaris system, I would run cfgadm -c unconfigure c0::dsk/c0t5d0. However, cfgadm -l does not list c6, nor does it list any disks. In fact, running cfgadm gets me the following:

bash# cfgadm -l /dev/rdsk/c0t5d0
Ap_Id Type Receptacle Occupant Condition
/dev/rdsk/c0t5d0: No matching library found

bash# cfgadm -l /dev/rdsk/c6t5000CCA00A274EDCd0
cfgadm: Attachment point not found

bash# cfgadm -l c6t5000CCA00A274EDCd0
Ap_Id Type Receptacle Occupant Condition
c6t5000CCA00A274EDCd0: No matching library found

So -- how do I administer and remove physical devices that are in multipath-managed controllers on Solaris 10u8?

posted by SpecialK to computers & internet (5 comments total)

A wizard friend writes:

"so the best way is to use cfgadm -c configure c6 #which will find all devices on C6
then devfsadm -C #this will clean up the old devices and add any new devices"
posted by iamabot at 9:24 PM on November 2

Disclaimer! I've only dealt with a J4400 array as part of a 7000-series storage appliance, which is to say: I haven't really dealt with a J4400 array. You will probably destroy all of your data if you listen to anything from me. That said:

You definitely have SAS drives, not SATA drives in your SAS-attached array?

Agree that you probably want to run devfsadm to make sure what's there is what's seen by the OS. Side note: `devfsadm -C -v` is preferable in my book, since you can actually see what changes devfsadm is making to your device tree.

Output of `cfgadm -al`, `stmsboot -L`, and `mpathadm list lu` might be helpful here.
posted by jmcmurry at 6:34 AM on November 3

And wouldn't using the "-s -v" arguments to `devfsadm` let him see what is going to happen without actually executing it?
posted by wenestvedt at 7:00 AM on November 3

Sorry about the slow response.

There are SATA drives in the attached array and a mix of SATA (SSDs) and 300 GB SAS drives in the primary system. They seem to be happy playing with one another and I've been writing data to them constantly.


# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::es/ses0 ESI connected configured unknown
c0::sd9 disk connected configured unknown
c0::smp/expd0 smp connected configured unknown
c4 scsi-bus connected configured unknown
c4::es/ses1 ESI connected configured unknown
c4::smp/expd1 smp connected configured unknown
c5 scsi-bus connected configured unknown
c5::es/ses2 ESI connected configured unknown
c5::smp/expd2 smp connected configured unknown
pcie32 etherne/hp connected configured ok
sata0/0 sata-port empty unconfigured ok
sata0/1 sata-port empty unconfigured ok
sata0/2 sata-port empty unconfigured ok
sata0/3 sata-port empty unconfigured ok
sata0/4 sata-port empty unconfigured ok
sata0/5 sata-port empty unconfigured ok
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
usb1/1 usb-device connected configured ok
usb1/2 usb-device connected configured ok
usb2/1 unknown empty unconfigured ok
usb2/2 unknown empty unconfigured ok
usb3/1 unknown empty unconfigured ok
usb3/2 unknown empty unconfigured ok
usb4/1 unknown empty unconfigured ok
usb4/2 unknown empty unconfigured ok
usb4/3 unknown empty unconfigured ok
usb4/4 unknown empty unconfigured ok
usb4/5 unknown empty unconfigured ok
usb4/6 usb-hub connected configured ok
usb4/6.1 unknown empty unconfigured ok
usb4/6.2 unknown empty unconfigured ok
usb4/6.3 unknown empty unconfigured ok
usb4/6.4 unknown empty unconfigured ok
usb4/7 unknown empty unconfigured ok
usb4/8 unknown empty unconfigured ok


notes: c0 is the onboard controller, c4 and c5 are the LSI-MPT controllers. Note that c6 is not seen.


bash-3.00# mpathadm list lu
/dev/rdsk/c6t5000CCA00A2A9398d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A29EE2Cd0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2BDBFCd0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2A8E68d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A0537ECd0s2
Total Path Count: 1
Operational Path Count: 1
mpathadm: Error: Unable to get configuration information.
mpathadm: Unable to complete operation


... Ok, that choked on the disk that I yanked and replaced. I restarted the system. NOTE: This is what I was talking about as a problem. :-P

Here's the output after the restart:

bash-3.00# mpathadm list lu
/dev/rdsk/c6t5000CCA00A2A9398d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A29EE2Cd0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000C50014F2C20Fd0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D783d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D6DDd0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2E180d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D5F3d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D6D6d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2AFEEd0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2EA24d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D3EDd0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2EC5Bd0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D2F8d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000C50014F2D476d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c6t5000CCA00A2BDBFCd0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2A8E68d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A0537ECd0s2
Total Path Count: 1
Operational Path Count: 1
/scsi_vhci/disk@g5000c5000a9147a7
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A274FECd0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A29A994d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2B3088d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2B4E74d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A2751D4d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A275584d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A298248d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t500151795879D11Ed0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t500151795879D1C1d0s2
Total Path Count: 1
Operational Path Count: 1




bash-3.00# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c0t0d0 /dev/rdsk/c6t5000CCA00A2A9398d0
/dev/rdsk/c0t1d0 /dev/rdsk/c6t5000CCA00A29EE2Cd0
/dev/rdsk/c0t4d0 /dev/rdsk/c6t5000CCA00A0537ECd0
/dev/rdsk/c0t5d0 /dev/rdsk/c6t5000CCA00A274EDCd0
/dev/rdsk/c0t6d0 /dev/rdsk/c6t5000CCA00A274FECd0
/dev/rdsk/c0t7d0 /dev/rdsk/c6t5000CCA00A29A994d0
/dev/rdsk/c0t8d0 /dev/rdsk/c6t5000CCA00A2B3088d0
/dev/rdsk/c0t9d0 /dev/rdsk/c6t5000CCA00A2B4E74d0
/dev/rdsk/c0t10d0 /dev/rdsk/c6t5000CCA00A2751D4d0
/dev/rdsk/c0t11d0 /dev/rdsk/c6t5000CCA00A275584d0
/dev/rdsk/c0t12d0 /dev/rdsk/c6t5000CCA00A298248d0
/dev/rdsk/c0t2d0 /dev/rdsk/c6t5000CCA00A2BDBFCd0
/dev/rdsk/c0t3d0 /dev/rdsk/c6t5000CCA00A2A8E68d0
/dev/rdsk/c0t14d0 /dev/rdsk/c6t500151795879D11Ed0
/dev/rdsk/c0t15d0 /dev/rdsk/c6t500151795879D1C1d0
/dev/rdsk/c4t0d0 /dev/rdsk/c6t5000C50014F2D476d0
/dev/rdsk/c4t1d0 /dev/rdsk/c6t5000C50014F2C20Fd0
/dev/rdsk/c4t2d0 /dev/rdsk/c6t5000C50014F2D783d0
/dev/rdsk/c4t3d0 /dev/rdsk/c6t5000C50014F2D6DDd0
/dev/rdsk/c4t4d0 /dev/rdsk/c6t5000C50014F2E180d0
/dev/rdsk/c4t5d0 /dev/rdsk/c6t5000C50014F2D5F3d0
/dev/rdsk/c4t6d0 /dev/rdsk/c6t5000C50014F2D6D6d0
/dev/rdsk/c4t7d0 /dev/rdsk/c6t5000C50014F2AFEEd0
/dev/rdsk/c4t8d0 /dev/rdsk/c6t5000C50014F2EA24d0
/dev/rdsk/c4t9d0 /dev/rdsk/c6t5000C50014F2D3EDd0
/dev/rdsk/c4t10d0 /dev/rdsk/c6t5000C50014F2EC5Bd0
/dev/rdsk/c4t11d0 /dev/rdsk/c6t5000C50014F2D2F8d0

posted by SpecialK at 10:56 AM on November 3

For anyone else curious -- the bug trail starts here:

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6776330

... and goes downhill from there. The general gist is that it's fixed in snv_126 / nevada, but not in Sol10 yet.
posted by SpecialK at 5:24 PM on November 5