Replacing a disk in Veritas Volume Manager

Replacing a disk in VxVM
--------------------------------------------------------

This has always been a difficult task while replacing a disk. The main worry which runs in our mind that data should not be lost while performing the activity, it should be intact otherwise you know the pain of facing other team.

Here i will be explaining very simple steps which will make you comfortable in doing such critical activities.

I will be talking about concatenated volume where you don't have redundant copies of data later i will cover the mirrored volume.

Disk replacement in concatenated volume

Here is the vxprint -htq output

bash-3.00# vxprint -htq
Disk group: appdg

dg appdg        default      default  4000     1347953587.80.vcs1

dm disk03       disk_3       auto     65536    2027168  -
dm disk04       disk_4       auto     65536    2027168  -

v  appvol       -            ENABLED  ACTIVE   3121152  SELECT    -        fsgen
pl appvol-01    appvol       ENABLED  ACTIVE   3121152  CONCAT    -        RW
sd disk03-01    appvol-01    disk03   0        2027168  0         disk_3   ENA
sd disk04-01    appvol-01    disk04   0        1093984  2027168   disk_4   ENA

In the above appdg diskgroup first subdisk is disk03-01 which is associated with disk03 and the plex is a concatenated type.

In the above scenario, if in case disk03 which is the first disk added to diskgroup and on which the Veritas has created the first subdisk for "appvol" volume. There is no way to replace the disk, if you try to do so, you will end up in corrupting the filesytem and the data.

But if you want to replace the second disk(flag - failing disk status)  which is absolutely possible without data loss but during this activity volume would not accessible. 

First we will use vxdiskadm to replace the disk, later we will try with only command prompt.

Connect the new disk in an empty slot and initialize it.

Run vxdiskadm from the command prompt, it will open up a list of menu items.


-> Select option no 4 - Remove a disk for replacement

Select the correct disk which you want to replace

(Caution - Do not select first the disk in the disgroup)

Once you have removed the disk it will prompt to select a disk with which you want to replace.  Select the disk which you have initialized.http://www.blogger.com/blogger.g?blogID=7967882172610062116#editor/target=post;postID=2988077713332921354

Once you are done with the above steps, you will find the plex is in disabled and recover steps

# vxprint -htqv
Disk group: appdg

v  appvol       -            DISABLED ACTIVE   3121152  SELECT    -        fsgen
pl appvol-01    appvol       DISABLED RECOVER  3121152  CONCAT    -        RW
sd disk03-01    appvol-01    disk03   0        2027168  0         disk_3   ENA
sd disk04-01    appvol-01    disk04   0        1093984  2027168   disk_5   ENA
#

You need to correct the above

Below sequence of commands needs to executed to fix the abovfe

#vxmend -g appdg fix stale appvol-01

#vxmend -g appdg fix clean appvol-01

#vxvol -g appdg start appvol

That's it you are done. Mount the volume and you are ready to use the volume.


Replacing a failed disk in mirrored volume

Here is my volume status

# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mir01        disk_3       auto     65536    2027168  -
dm mir02        disk_4       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
pl appvolmir-02 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir02-01     appvolmir-02 mir02    0        1024000  0         disk_4   ENA

Where appvolmir is mirrored volume in mir diskgroup, having 2 plexes appvolmir-01 and appvolmir-02

Change the plex status to offline

#vxmend -g mir off appvolmir-02

# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mir01        disk_3       auto     65536    2027168  -
dm mir02        disk_4       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
pl appvolmir-02 appvolmir    DISABLED OFFLINE  1024000  CONCAT    -        RW
sd mir02-01     appvolmir-02 mir02    0        1024000  0         disk_4   ENA
# 

Disassociates the offline plex from the volume

#vxplex -g mir dis  appvolmir-02

# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mir01        disk_3       auto     65536    2027168  -
dm mir02        disk_4       auto     65536    2027168  -

pl appvolmir-02 -            DISABLED -        1024000  CONCAT    -        RW
sd mir02-01     appvolmir-0 mir02    0        1024000  0         disk_4   ENA

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
#
 
 Remove the plex

#vxedit -g mir -r rm appvolmir-02

 # vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mir01        disk_3       auto     65536    2027168  -
dm mir02        disk_4       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
#
Once plex is removed, removed the disk from veritas control

#vxdg -g mir rmdisk mir02 




# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mir01        disk_3       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
#


So disk removal is done, Now its time to attach a new disk and sync the data. Identify a new disk should be of similar or more size.

Here in this example i would be using the disk_5 as a replacement disk.

# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mirreplacedisk disk_5     auto     65536    41764864 -
dm mir01        disk_3       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
#

So once the new disk is initialized and added to the diskgroup , mirror the volume. Here is the vxdisk list ouput

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
c0d0s2       auto:sliced     ibmdg01      ibmdg        online
disk_3       auto:cdsdisk    mir01        mir          online
disk_4       auto:cdsdisk    -            -            online
disk_5       auto:cdsdisk    mirreplacedisk  mir          online
disk_6       auto:SVM        -            -            SVM
disk_7       auto:SVM        -            -            SVM
disk_8       auto:ZFS        -            -            ZFS
disk_9       auto:ZFS        -            -            ZFS
disk_10      auto:ZFS        -            -            ZFS
disk_11      auto:cdsdisk    -            -            online
disk_12      auto:cdsdisk    -            -            online

"mirrreplacedisk" is the replaced disk

#vxassist -g mir make mirror appvolmir mirreplacedisk

# vxprint -htqg mir
dg mir          default      default  4000     1348198265.15.vcs1

dm mirreplacedisk disk_5     auto     65536    41764864 -
dm mir01        disk_3       auto     65536    2027168  -

v  appvolmir    -            ENABLED  ACTIVE   1024000  SELECT    -        fsgen
pl appvolmir-01 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mir01-01     appvolmir-01 mir01    0        1024000  0         disk_3   ENA
pl appvolmir-02 appvolmir    ENABLED  ACTIVE   1024000  CONCAT    -        RW
sd mirreplacedisk-01 appvolmir-02 mirreplacedisk 0 1024000 0      disk_5   ENA
#
 

Driver versus firmware

The difference of firmware and drivers is the application of this piece of software.
The most electronic hardware only needs firmware to run basicly. E.G. a DVD-player needs firmware to know how to read a CD or DVD. But to have it accessed under a operating system like Windows you need a drivhttp://www.blogger.com/blogger.g?blogID=7967882172610062116#editor/target=post;postID=5967573526606548086er to access the CD or DVD or watch a movie.
To visualize: A car needs an engine to drive at all, but needs a person (driver!)to control it

Firmware is the software that runs on the device. A driver is the software that tells your operating system how to communicate with the device. Not all devices have firmware--only devices with some level of intelligence.
Read more: http://wiki.answers.com/Q/What_is_the_difference_between_firmware_and_drivers#ixzz25ieOoLEf

ALOM reset.

1. ALOM console is hunged

I am able to access the console but not able to type in the username and password. As per my finding from the internet, ALOM can be reset by completely poweroff the box means to say remove the power cable.

Second option i could find is, it can be done at the OS level. Here is the command

#/usr/platform/`uname -i`/sbin/scadm resetrsc

Veritas Dyanmic Multipathing

Veritas Dynamic Multipathing:

As the name indicates, veritas dynamic multipathing provides multipathing functionality for operating system native device.
It represents a multiple path of a device using DMP metadevice also known as DMP nodes .

Important points:

You can not create veritas filessytem on DMP metadevice, only way to  have the filesystem is create VxVM volume and create VxFS filesystem on it.

VxVM volume and ZFS volume can coexist together but the device which has VxVM label can not be used in ZFS and vice versa.

Feature of Multipathing:

It provides the following feature:

1. Availability
2. Reliability
3. Performance 

It does those by using path failover and load balancing.

Two kernel thread performs the checking the status of HBA and path restoration.

1. errord
2. restored

This can be checked using #vxdmpadm stat

Support for ZFS
---------------------

By default VxVM does not support zfs. To enable it, you need to turn on dmp_native_support tunable

#vxdmpadm settune dmp_native_support=on

To check the status 

#vxdmpadm gettune dmp_native_support


Here is sequence of commands to enable VxDMP support and create a zpool followed by adding a new device to the zpool.

1.Enable the dmp_native_support

#vxdmpadm settune dmp_native_support=on

2.Check the dmp_native_support status

#vxdmpadm gettune dmp_native_support

Once the above steps are done, you can go ahead and add create zpool.

3. Bring the disk under Veritas Volume manager

#vxdctl enable or #vxdisk scandisks

4. Check the disks status

#vxdisk list

5.Now its time to create zpool using the device

#zpool create testpool <device_name>

6. Check the zpool status

#zpool status

7.Add a new device

#zpool add testpool <device name>

------------------------------------------------------------------

To see all the subpaths of the dmpnode name

#vxdmpadm getsubpaths dmpnodename=<name> 

# vxdmpadm getsubpaths dmpnodename=disk_8
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c2t5d0       ENABLED(A)    -          c2         Disk         disk       
      -

where disk_8 is the dmpnodename

Now the next is when we have device details, wants to find out the corresponding dmpnodename


# vxdmpadm getdmpnode nodename=c2t5d0
NAME                 STATE        ENCLR-TYPE   PATHS  ENBL  DSBL  ENCLR-NAME
==============================================================================
disk_8  

Others useful commands

#vxdmpadm list dmpnode all

#vxdmpadm list dmpnode dmpnodename=<dmp node name>

# vxdmpadm list dmpnode dmpnodename=disk_4
dmpdev          = disk_4
state           = enabled
enclosure       = disk
cab-sno         = DISKS
asl             = scsi3_jbod
vid             = VMware,
pid             = VMware Virtual S
array-name      = Disk
array-type      = Disk
iopolicy        = MinimumQ
avid            = -
lun-sno         = 6000C29A8511F1C593A6B3235534E5EA
udid            = VMware%2C%5FVMware%20Virtual%20S%5FDISKS%5F6000C29A8511F1C593A6B3235534E5EA
dev-attr        = -
###path         = name state type transport ctlr hwpath aportID aportWWN attr
path            = c2t1d0s2 enabled(a) - SCSI c2 /pci@0,0/pci15ad,1976@10 - - -
#

Configuring network management port to access console

Console can be accessed remotely either using Serial management port or Network management port. Mostly all the Solaris servers are having both the option.

Here, I will be showing, how to set up the Network Management port.

Access the console locally. Once you are connected you will be prompted to enter username and password. Use the below default username and password.

Username : root

password : changeme

Once you are logged in you will be placed in ILOM service processor prompt "->"

Note: Assign a new password during the initial system configuration

Initially network management port is configured to use DHCP.

Follow the below steps to configure the Network Mgmt port with static IP address.

Step 1:

-> set /SP/network state=enabled

Set ’state’ to ’enabled’

Step 2:

-> set /SP/network pendingipaddress=xx.xx.xx.xx

Step 3:

-> set /SP/network pendingipdiscovery=static

Step 4:

-> set /SP/network pendingipnetmask=xx.xx.xx.xx

Step 5:

-> set /SP/network pendingipgateway=xxx.xxx.xx.xxx

Step 6:

Final steps to commit the changes.

-> set /SP/network commitpending=true

Step 7:

Verify the network settings.

Show /SP/network

That's it you are done. Connect your network Mgmt port using ssh