Articles from IT World

Veritas Volume Manager (VxVM) command line

examples
2012-06-15 15:06:02 Yannick

Table Of Contents
1. Preamble
2. Dynamic Multipathing (DMP)
3. Basic commands
1. Disk groups
2. Volumes
3. Plexes
4. Subdisks
4. Performance monitoring
5. References

Preamble
Before entering in Veritas Volume Manager (VxVM) details we must go in fine detail
of our hardware configuration. Lets first confirm at Linux level that I see my 2 fiber
channel (FC) cards with each 2 FC ports:
[root@server1
09:00.0 Fibre
09:00.1 Fibre
0c:00.0 Fibre
0c:00.1 Fibre

~]# lspci | grep -i fibre

Channel: QLogic Corp. ISP2432-based
Channel: QLogic Corp. ISP2432-based
Channel: QLogic Corp. ISP2432-based
Channel: QLogic Corp. ISP2432-based

[root@server1 ~]#
lrwxrwxrwx 1 root
lrwxrwxrwx 1 root
lrwxrwxrwx 1 root
lrwxrwxrwx 1 root

4Gb
4Gb
4Gb
4Gb

Fibre
Fibre
Fibre
Fibre

Channel
Channel
Channel
Channel

to
to
to
to

ll /sys/class/fc_host/host*/device
root 0 Jun 5 17:49 /sys/class/fc_host/host0/device
root 0 Jun 5 17:51 /sys/class/fc_host/host1/device
root 0 Jun 5 17:51 /sys/class/fc_host/host2/device
root 0 Jun 5 17:51 /sys/class/fc_host/host3/device

PCI
PCI
PCI
PCI
->
->
->
->

Express
Express
Express
Express

HBA
HBA
HBA
HBA

(rev
(rev
(rev
(rev

03)
03)
03)
03)

../../../devices/pci0000:00/0000:00:03.0/0000:09:00.0/host0
../../../devices/pci0000:00/0000:00:03.0/0000:09:00.1/host1
../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.0/host2
../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.1/host3

We will see this later but we need correspondence between SCSI chain number
and its World Wide Name (WWN), here it is:
[root@server1 ~]# for host in `ls /sys/class/fc_host/`; do echo "$host WWN: `cat /sys/class/fc_host/$host/port_name`"; done
host0 WWN: 0x50060b0000c35e0c
host1 WWN: 0x50060b0000c35e0e
host2 WWN: 0x50060b0000c35e08
host3 WWN: 0x50060b0000c35e0a
So if everything is fiber wired (FC cards/ports to FC switches) I should see any disk
array (enclosure) device/LUN as 4 equivalent name. Please note that this is
absolutely not mandatory, whats really important in my configuration is to have each
disk visible through my 2 FC cards (so by one port minimum so each disk/LUN will
have 2 names).
Please note that this post is more for DBAs to see and understand what have been
done by you system teammates i.e. Im not entering in the creation/modification of
VxVM disk/disk group/controller and so on
All commands have been executed on a Red Hat Enterprise Linux Server release
5.5 (Tikanga) server configured with Veritas Volume Manager release VRTSvxvm5.1.132.000-SP1RP2_RHEL5.
To avoid putting real server name Im using the below tips for my prompt
customization:
export PS1="[\u@server1 \W]# "

Dynamic Multipathing (DMP)

Veritas Dynamic Multi-Pathing (DMP or VxDMP) is the VxVM functionality
implemented to support disk availability from multiple paths. Even if it looks complex
to start with this functionality it is unfortunately mandatory before doing the more
simple things as it is based configuration on which all other components will be
created (subdisks, plexes, volumes).
Configured disk arrays or enclosures accessible from this server:
[root@server1 ~]# vxdmpadm listenclosure
ENCLR_NAME
ENCLR_TYPE
ENCLR_SNO
STATUS
ARRAY_TYPE
LUN_COUNT
=======================================================================================
emc0
EMC
000290102696
CONNECTED
A/A
54
emc1
EMC
000290105264
CONNECTED
A/A
1
other_disks
OTHER_DISKS
OTHER_DISKS
CONNECTED
OTHER_DISKS
1
I/O policy default configuration of this enclosure:
[root@server1 ~]# vxdmpadm getattr enclosure emc0
ENCLR_NAME
ATTR_NAME
DEFAULT
CURRENT
============================================================================
emc0
iopolicy
MinimumQ
MinimumQ
emc0
partitionsize
512
512
emc0
use_all_paths
emc0
failover_policy
Global
Global
emc0
recoveryoption[throttle]
Nothrottle[0] Nothrottle[0]
emc0
recoveryoption[errorretry]
Timebound[300] Timebound[300]
emc0
redundancy
0
0
emc0
dmp_lun_retry_timeout
0
0
emc0
failovermode
Disk default naming convention of VxVM:
[root@server1 ~]# vxddladm get namingscheme
NAMING_SCHEME
PERSISTENCE
LOWERCASE
USE_AVID
============================================================
Enclosure Based
Yes
Yes
Yes
List of configured disk on your system:
[root@server1 ~]# vxdisk -e
DEVICE
TYPE
cciss/c0d0
auto:none
emc0_00f3
auto:cdsdisk
emc0_3d6b
auto:cdsdisk
emc0_3d7b
auto:cdsdisk
emc0_3d73
auto:cdsdisk
emc0_3ebb
auto:cdsdisk
emc0_3eb3
auto:cdsdisk
emc0_3ecb
auto:cdsdisk
emc0_3ec3
auto:cdsdisk
.
.

list
DISK
vgp1148_01
vgp1148_03
vgp1148_02
vgp1118_02
vgp1118_01
vgp1318_01
vgp1118_03

GROUP
vgp1148
vgp1148
vgp1148
vgp1118
vgp1118
vgp1318
vgp1118

STATUS
online
online
online
online
online
online
online
online
online

clone_disk
clone_disk
clone_disk
clone_disk
clone_disk
clone_disk
clone_disk

OS_NATIVE_NAME
cciss/c0d0
sdbg
sdbn
sdbp
sdbo
sdca
sdbx
sdcc
sdcb

ATTR
std
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1

clone_disk
clone_disk
clone_disk
clone_disk
clone_disk
clone_disk
clone_disk
clone_disk

OS_NATIVE_NAME
sdax
sdaz
sdbd
sdbh
sdbi
sdbj
sdbb
sdbc

ATTR
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1
srdf-r1

invalid

Disks of a disk group (please note it makes relationship between VxVM disks and
the ones you see at system level):
[root@server1 ~]# vxdisk -g
DEVICE
TYPE
emc0_084b
auto:cdsdisk
emc0_084f
auto:cdsdisk
emc0_085b
auto:cdsdisk
emc0_526f
auto:cdsdisk
emc0_527b
auto:cdsdisk
emc0_527f
auto:cdsdisk
emc0_0853
auto:cdsdisk
emc0_0857
auto:cdsdisk

vgp1417 -e list
DISK
GROUP
vgp1417_04
vgp1417
vgp1417_05
vgp1417
vgp1417_08
vgp1417
vgp1417_09
vgp1417
vgp1417_10
vgp1417
vgp1417_11
vgp1417
vgp1417_06
vgp1417
vgp1417_07
vgp1417

STATUS
online
online
online
online
online
online
online
online

Main problem with above command is that you dont see multipathing details of
VxVM disk, to go deeper in detail use:

[root@server1 ~]# vxdisk list emc0_084b

Device:
emc0_084b
devicetag: emc0_084b
type:
auto
hostid:
server1
disk:
name=vgp1417_04 id=1325653859.61.server1
group:
name=vgp1417 id=1318435685.29.server1
info:
format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:
online ready private autoconfig noautoimport imported clone_disk
pubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3
guid:
udid:
EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:
version:
3.1
iosize:
min=512 (bytes) max=1024 (blocks)
public:
slice=3 offset=65792 len=70539872 disk_offset=0
private:
slice=3 offset=256 len=65536 disk_offset=0
update:
time=1335608893 seqno=0.17
ssb:
actual_seqno=0.0
headers:
0 240
configs:
count=1 len=51360
logs:
count=1 len=4096
Defined regions:
config
priv 000048-000239[000192]: copy=01 offset=000000 enabled
config
priv 000256-051423[051168]: copy=01 offset=000192 enabled
log
priv 051424-055519[004096]: copy=01 offset=000000 enabled
lockrgn priv 055520-055663[000144]: part=00 offset=000000
Multipathing information:
numpaths:
2
sdax
state=enabled
sdh
state=enabled
Remark:
You may also use the opposite command:
[root@server1 ~]# vxdisk list sdax
Device:
emc0_084b
devicetag: emc0_084b
type:
auto
hostid:
server1
disk:
name=vgp1417_04 id=1325653859.61.server1
group:
name=vgp1417 id=1318435685.29.server1
info:
format=cdsdisk,privoffset=256,pubslice=3,privslice=3
flags:
online ready private autoconfig noautoimport imported clone_disk
pubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3
guid:
udid:
EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:
version:
3.1
iosize:
min=512 (bytes) max=1024 (blocks)
public:
slice=3 offset=65792 len=70539872 disk_offset=0
private:
slice=3 offset=256 len=65536 disk_offset=0
update:
time=1335608893 seqno=0.17
ssb:
actual_seqno=0.0
headers:
0 240
configs:
count=1 len=51360
logs:
count=1 len=4096
Defined regions:
config
priv 000048-000239[000192]: copy=01 offset=000000 enabled
config
priv 000256-051423[051168]: copy=01 offset=000192 enabled
log
priv 051424-055519[004096]: copy=01 offset=000000 enabled
lockrgn priv 055520-055663[000144]: part=00 offset=000000
Multipathing information:
numpaths:
2
sdax
state=enabled
sdh
state=enabled
Using vxdmpadm command you can also display complete multipathing information
and obviously also confirm that each VxVM disk is accessible through 2 paths:
[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=emc0_084b
NAME
STATE[A]
PATH-TYPE[M] CTLR-NAME ENCLR-TYPE
ENCLR-NAME
ATTRS
================================================================================
sdax
ENABLED(A)
c2
EMC
emc0
sdh
ENABLED(A)
c0
EMC
emc0
[root@server1 ~]# vxdmpadm -v getdmpnode nodename=sdax
NAME
STATE
ENCLR-TYPE
PATHS ENBL DSBL ENCLR-NAME
SERIAL-NO
ARRAY_VOL_ID
========================================================================================================
emc0_084b
ENABLED
EMC
2
2
0
emc0
960084B000
084B
[root@server1 ~]# vxdmpadm getdmpnode enclosure=emc0
NAME
STATE
ENCLR-TYPE
PATHS ENBL DSBL ENCLR-NAME
==============================================================================
emc0_00f3
ENABLED
EMC
2
2
0
emc0
emc0_3d6b
ENABLED
EMC
2
2
0
emc0
emc0_3d7b
ENABLED
EMC
2
2
0
emc0
emc0_3d73
ENABLED
EMC
2
2
0
emc0
emc0_3ebb
ENABLED
EMC
2
2
0
emc0
emc0_3eb3
ENABLED
EMC
2
2
0
emc0
emc0_3ecb
ENABLED
EMC
2
2
0
emc0
emc0_3ec3
ENABLED
EMC
2
2
0
emc0
.
.
[root@server1 ~]# vxdmpadm list dmpnode dmpnodename=emc0_084b
dmpdev
= emc0_084b
state
= enabled
enclosure
= emc0
cab-sno
= 000290102696
asl
= libvxemc.so
vid
= EMC
pid
= SYMMETRIX
array-name
= EMC
array-type
= A/A
iopolicy
= MinimumQ
avid
= 084B
lun-sno
= 960084B000
udid
= EMC%5FSYMMETRIX%5F000290102696%5F960084B000
dev-attr
= srdf-r1
###path
= name state type transport ctlr hwpath aportID aportWWN attr
path
= sdax enabled(a) - FC c2 c2 7b-b 50:06:04:84:52:a7:6a:36 path
= sdh enabled(a) - FC c0 c0 10b-b 50:06:04:84:52:a7:6a:39 [root@server1 ~]# vxdmpadm getlungroup dmpnodename=sdax
NAME
STATE
ENCLR-TYPE
PATHS ENBL DSBL ENCLR-NAME
=========================================================================
emc0_084b
ENABLED
EMC
2
2
0
emc0
[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=sdax
NAME
STATE[A]
PATH-TYPE[M] CTLR-NAME ENCLR-TYPE
ENCLR-NAME
ATTRS
================================================================================
sdax
ENABLED(A)
c2
EMC
emc0
sdh
ENABLED(A)
c0
EMC
emc0
Remark:
Please note the two controllers that are linked to the FC card we have identified in
previous chapter:
[root@server1 ~]# vxdmpadm getsubpaths enclosure=emc0 ctlr=c0
NAME
STATE[A]
PATH-TYPE[M] DMPNODENAME ENCLR-NAME
CTLR
ATTRS
================================================================================
sdo
ENABLED(A)
emc0_00f3
emc0
c0
sdz
ENABLED(A)
emc0_0837
emc0
c0
sdh
ENABLED(A)
emc0_084b
emc0
c0
sdi
ENABLED(A)
emc0_084f
emc0
c0
sdj
ENABLED(A)
emc0_0853
emc0
c0
sdk
ENABLED(A)
emc0_0857
emc0
c0
sdl
ENABLED(A)
emc0_085b
emc0
c0
.
.
The vxdisk command can also display the multipathing information:
[root@server1 ~]# vxdisk path -g vgp1417

SUBPATH
sdax
sdh
sdaz
sdi
sdbd
sdl
sdbh
sdp
sdbi
sdq
sdbj
sdr
sdbb
sdj
sdbc
sdk

DANAME
emc0_084b
emc0_084b
emc0_084f
emc0_084f
emc0_085b
emc0_085b
emc0_526f
emc0_526f
emc0_527b
emc0_527b
emc0_527f
emc0_527f
emc0_0853
emc0_0853
emc0_0857
emc0_0857

DMNAME
vgp1417_04
vgp1417_04
vgp1417_05
vgp1417_05
vgp1417_08
vgp1417_08
vgp1417_09
vgp1417_09
vgp1417_10
vgp1417_10
vgp1417_11
vgp1417_11
vgp1417_06
vgp1417_06
vgp1417_07
vgp1417_07

GROUP
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417
vgp1417

STATE
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED

Lets come back to the controller and display the ones configured on my system
(each enclosure is configured with two controllers):
[root@server1 ~]# vxdmpadm listctlr all
CTLR-NAME
ENCLR-TYPE
STATE
ENCLR-NAME
=====================================================
c0
EMC
ENABLED
emc0
c2
EMC
ENABLED
emc0
c0
EMC
ENABLED
emc1
c2
EMC
ENABLED
emc1
c360
OTHER_DISKS
ENABLED
other_disks
We can re-conciliate WWN identified in previous chapter with VxDMP commands:
[root@server1 ~]# vxdmpadm getctlr c0
LNAME
PNAME
VENDOR
CTLR-ID
=============================================================================================
c0
c0
Qlogic-Corp.
50:06:0b:00:00:c3:5e:0c
[root@server1 ~]# vxdmpadm getctlr c2
LNAME
PNAME
VENDOR
CTLR-ID
=============================================================================================
c2
c2
Qlogic-Corp.
50:06:0b:00:00:c3:5e:08
And confirm, as anticipated, that each of my FC card is wired with only one FC
cable. Each of my FC or Host Bus Adapter (HBA) cards has two ports and only one
is used:
[root@server1 ~]# vxddladm list hbas
HBA-ID DRIVER
FIRMWARE
DISCOVERY
STATE
ADDRESS
======================================================================================
c1
8.03.07.03.05.07-k 5.03.16 (496)
fabric
Offline 50:06:0b:00:00:c3:5e:0e
c3
8.03.07.03.05.07-k 5.03.16 (496)
fabric
Offline 50:06:0b:00:00:c3:5e:0a
c360
Online c0
8.03.07.03.05.07-k 5.03.16 (496)
fabric
Online 50:06:0b:00:00:c3:5e:0c
c2
8.03.07.03.05.07-k 5.03.16 (496)
fabric
Online 50:06:0b:00:00:c3:5e:08
To obtain the configured ports:
[root@server1 ~]# vxddladm list ports
PORT-ID
HBA-ID
STATE
ADDRESS
==========================================
c0_p3
c0
Online
50:06:04:84:52:a7:6a:39
c0_p2
c0
Online
50:06:04:84:52:a7:6a:08
c0_p1
c0
Online
50:06:04:84:52:a9:ec:09
c2_p3
c2
Online
50:06:04:84:52:a7:6a:36
c2_p2
c2
Online
50:06:04:84:52:a7:6a:07
c2_p1
c2
Online
50:06:04:84:52:a9:ec:06
[root@server1 ~]# vxddladm list targets
TARGET-ID
ALIAS
HBA-ID
STATE
ADDRESS
=====================================================
c0_p3_t0
c0
Online
50:06:04:84:52:a7:6a:39
c0_p2_t0
c0
Online
50:06:04:84:52:a7:6a:08
c0_p1_t0
c0
Online
50:06:04:84:52:a9:ec:09
c2_p3_t0
c2
Online
50:06:04:84:52:a7:6a:36
c2_p2_t0
c2
Online
50:06:04:84:52:a7:6a:07
c2_p1_t0
c2
Online
50:06:04:84:52:a9:ec:06
[root@server1 ~]# vxddladm list targets hba=c0
TARGET-ID
ALIAS
HBA-ID
STATE
ADDRESS
=====================================================
c0_p3_t0
c0
Online
50:06:04:84:52:a7:6a:39
c0_p2_t0
c0
Online
50:06:04:84:52:a7:6a:08
c0_p1_t0
c0
Online
50:06:04:84:52:a9:ec:09
To understand which devices are using which ports and the name of the VxVM
driver:
[root@server1 ~]# vxddladm list devices
DEVICE
TARGET-ID
STATE
DDL-STATUS (ASL)
===============================================================
cciss/c0d0
Online sdg
c0_p3_t0
Online SKIPPED
sdi
c0_p3_t0
Online CLAIMED (libvxemc.so)
sdf
c0_p3_t0
Online SKIPPED
sdh
c0_p3_t0
Online CLAIMED (libvxemc.so)
sde
c0_p3_t0
Online SKIPPED
sdd
c0_p3_t0
Online SKIPPED
sdr
c0_p3_t0
Online CLAIMED (libvxemc.so)
.
.
[root@server1 ~]# vxddladm list devices target=c0_p1_t0
DEVICE
TARGET-ID
STATE
DDL-STATUS (ASL)
===============================================================
sdbl
c0_p1_t0
Online CLAIMED (libvxemc.so)
sdbk
c0_p1_t0
Online SKIPPED
To get the list of supported enclosures:
[root@server1 ~]# vxddladm listsupport all
LIBNAME
VID
PID
=================================================================================================
libvxpurple.so
SUN
T300
libvxveritas.so
VERITAS
All
libvxhillsannet2.so DotHill
SANnet II FC, SANnet II SCSI, SANnet II SATA, SANnet II U320
libvxhds.so
HITACHI
All
libvxpp.so
EMC, DGC
All
libvxFJTSYe8k.so
FUJITSU
All
libvxfje3k4ka.so
FUJITSU
E3000, E400A
libvxibmds6k.so
IBM
1750
libvxsun7x10.so
SUN
Sun Storage 7410, Sun Storage 7310, Sun Storage 7210, Sun Storage 7110, ZFS Storage 7120, ZFS Storage 7320, ZFS Storage 7420, ZFS Storage 7720
libvxfsc.so
FSC
FibreCAT_SX1
libvxxp1281024.so
HP
All
libvxlsiall.so
IBM, SUN, STK, SGI All
libvxnipnyis.so
NEC
iStorage 1000, iStorage 2000, iStorage 4000
libvxtsbaf.so
TOSHIBA
AF_AF3500, AF_AF1500, AF2_AF7000, AF2_AF2000, AF3_AF7500, AF3_AF2500
libvxfusionio.so
FIO
ioDrive
libvxhpalua.so
HP, COMPAQ
HSV101, HSV111 (C)COMPAQ, HSV111, HSV200, HSV210, HSV300, HSV400, HSV450
libvxxp12k.so
HP
All
libvxhitachi.so
HITACHI
DF350, DF400, DF400F, DF500, DF500F
libvxpillaraxiom.so Pillar
Axiom 300, Axiom 500, Axiom 600
libvxFJTSYe6k.so
FUJITSU
E6000
libvxxiv.so
XIV, IBM
NEXTRA, 2810XIV
libvxxp256.so
HP
All
libvxramsan.so
TMS
RamSan 400
libvxhdsalua.so
HITACHI
DF600, DF600-V, DF600F, DF600F-V
libvxhpmsa.so
HP
MSA VOLUME
libvxhds9980.so
HITACHI
All
libvxhdsusp.so
HITACHI
All
libvxCLARiiON.so
DGC
All
libvxnetapp.so
NETAPP
All
libvxeqlogic.so
EQLOGIC
100E-00
libvxemc.so
EMC
SYMMETRIX
libvxddns2a.so
DDN
S2A 9550, S2A 9900, S2A 9700
libvxfjtsye2k.so
FUJITSU
E2000, ETERNUS_DXL
libvxmsa2kfc_sa.so HP
MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSI

libvxmsa2kfc_sa.so
libvxdothill.so
libvxcompellent.so
libvxibmds8k.so
libvxmsa2k.so
libvxxiotechE5k.so
libvxcopan.so
libvxibmsvc.so
libvxhuawei.so
libvx3par.so
libvxhpsvsp.so
libvxshark.so

HP
DotHill
COMPELNT
IBM
HP
XIOTECH
COPANSYS
IBM
HUAWEI, HS
3PARdata
HP
IBM

MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSI
R/Evo 2730-2R, R/Evo 2530-2R, R/Evo 2330-2R, R/Evo 2130-2RX, R/Evo 2130-2J, R/Evo 5730-2R
Compellent Vol
2107
MSA2012fc, MSA2212fc, MSA2012i
ISE1400
8814, 8818
2145, 2062
S5300, S5500, S5600, S2300, S2100, V1800, V1500, VIS6000, S2300E, S2600, S6800E, S8000-I, S8000
VV
HSVX740
2105

And more information (Active/Active mode) on the one I use:

[root@server1 ~]# vxddladm listsupport libname=libvxemc.so
ATTR_NAME
ATTR_VALUE
=================================================================================================
LIBNAME
libvxemc.so
VID
EMC
PID
SYMMETRIX
ARRAY_TYPE
A/A
ARRAY_NAME
EMC

Basic commands
Disk groups
How to determine if a disk is associated to a disk group:
[root@server1 ~]# vxdisk -s list sdax
Disk:
emc0_084b
type:
auto
flags: online ready private autoconfig noautoimport imported clone_disk
guid:
udid:
EMC%5FSYMMETRIX%5F000290102696%5F960084B000
site:
diskid: 1325653859.61.server1
dgname: vgp1417
dgid:
1318435685.29.server1
hostid: server1
info:
format=cdsdisk,privoffset=256,pubslice=3,privslice=3
Disk groups list:
[root@server1 ~]# vxdg list
NAME
STATE
vgp1417a
enabled,cds
vgoem3
enabled,cds
vgptws02a
enabled,cds
.
.

ID
1318432411.19.server1
1320851456.27.server1
1318847636.93.server1

More information of a disk group:

[root@server1 ~]# vxdg list vgp1417
Group:
vgp1417
dgid:
1318435685.29.server1
import-id: 1024.62
flags:
cds
version:
160
alignment: 8192 (bytes)
ssb:
on
autotagging:
on
detach-policy: global
dg-fail-policy: dgdisable
copies:
nconfig=default nlog=default
config:
seqno=0.1556 permlen=51360 free=51325 templen=17 loglen=4096
config disk emc0_084b copy 1 len=51360 state=clean online
config disk emc0_084f copy 1 len=51360 state=clean online
config disk emc0_085b copy 1 len=51360 state=clean online
config disk emc0_526f copy 1 len=51360 disabled
config disk emc0_527b copy 1 len=51360 disabled
config disk emc0_527f copy 1 len=51360 disabled
config disk emc0_0853 copy 1 len=51360 state=clean online
config disk emc0_0857 copy 1 len=51360 state=clean online
log disk emc0_084b copy 1 len=4096
log disk emc0_084f copy 1 len=4096
log disk emc0_085b copy 1 len=4096
log disk emc0_526f copy 1 len=4096 disabled
log disk emc0_527b copy 1 len=4096 disabled
log disk emc0_527f copy 1 len=4096 disabled
log disk emc0_0853 copy 1 len=4096
log disk emc0_0857 copy 1 len=4096
Free space of a disk group:
[root@server1 ~]# vxdg -g vgp1417 free
DISK
DEVICE
TAG
vgp1417_04
emc0_084b
emc0_084b
vgp1417_05
emc0_084f
emc0_084f
vgp1417_09
emc0_526f
emc0_526f
vgp1417_10
emc0_527b
emc0_527b
vgp1417_11
emc0_527f
emc0_527f

OFFSET
56623104
64389600
17592064
0
0

LENGTH
13916768
6150272
52947808
70539872
70539872

FLAGS
-

Remark:
So how much at the end ? For vgp1417_04 (emc0_084b) it is 13916768*512 (sector
size)/(1024 * 1024) MB i.e. 6795MB free (I have already seen more user friendly
command output)
Even if not written in official VxVM documentation there is an alternative command
with a much better output:
[root@server1 ~]# vxassist -g vgp1417 maxsize
Maximum volume size: 109616431104 (104538Mb)
Complete information of a disk group (subdisks, plexes, ):
[root@server1 ~]# vxprint -g vgp1417
TY NAME
ASSOC
KSTATE
dg vgp1417
vgp1417
-

LENGTH
-

PLOFFS
-

STATE
-

TUTIL0
-

PUTIL0
-

dm
dm
dm
dm
dm
dm
dm
dm

70539872
70539872
70539872
70539872
70539872
70539872
70539872
70539872

ENABLED
ENABLED
ENABLED

6291456
6291456
6291456

ACTIVE
ACTIVE
-

LENGTH
LENGTH
LENGTH
LENGTH
LENGTH

READPOL
LAYOUT
[COL/]OFF
[COL/]OFF
[COL/]OFF

vgp1417_04
vgp1417_05
vgp1417_06
vgp1417_07
vgp1417_08
vgp1417_09
vgp1417_10
vgp1417_11

emc0_084b
emc0_084f
emc0_0853
emc0_0857
emc0_085b
emc0_526f
emc0_527b
emc0_527f

v lvol1
fsgen
pl lvol1-01
lvol1
sd vgp1417_04-01 lvol1-01
.
.

Volumes
Volumes of a disk group:
[root@server1 ~]# vxprint -hvt -g vgp1417
V NAME
RVG/VSET/CO KSTATE
STATE
PL NAME
VOLUME
KSTATE
STATE
SD NAME
PLEX
DISK
DISKOFFS
SV NAME
PLEX
VOLNAME NVOLLAYR
SC NAME
PLEX
CACHE
DISKOFFS
DC NAME
PARENTVOL
LOGVOL
SP NAME
SNAPVOL
DCO
EX NAME
ASSOC
VC

PERMS

PREFPLEX
NCOL/WID
DEVICE
AM/NM
DEVICE

MODE

UTYPE
MODE
MODE
MODE
MODE

STATE

v lvol1
pl lvol1-01
lvol1
sd vgp1417_04-01 lvol1-01

ENABLED ACTIVE
ENABLED ACTIVE
vgp1417_04 0

6291456
6291456
6291456

v
.
.

ENABLED

20971520 SELECT

[root@server1 ~]# vxprint -g vgp1417 -t lvol1

V NAME
RVG/VSET/CO KSTATE
STATE

LENGTH

READPOL

PREFPLEX UTYPE

6291456

SELECT

[root@server1 ~]# vxprint -pt -g vgp1417

PL NAME
VOLUME
KSTATE
STATE

LENGTH

LAYOUT

NCOL/WID MODE

pl
pl
pl
pl
pl
pl
pl
pl
pl
pl
pl

6291456 CONCAT
20971520 CONCAT
2097152 CONCAT
6291456 CONCAT
4194304 CONCAT
14680064 CONCAT
44040192 CONCAT
169869312 CONCAT
12582912 CONCAT
27262976 CONCAT
41943040 CONCAT

lvol2

ACTIVE

SELECT
CONCAT
0

fsgen
RW
emc0_084b ENA
fsgen

More information of a particular volume:

lvol1

ENABLED

ACTIVE

fsgen

Plexes
Plexes of a disk group:

lvol1-01
lvol2-01
lvol3-01
lvol4-01
lvol5-01
lvol6-01
lvol7-01
lvol8-01
lvol9-01
lvol10-01
lvol11-01

lvol1
lvol2
lvol3
lvol4
lvol5
lvol6
lvol7
lvol8
lvol9
lvol10
lvol11

ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED
ENABLED

ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE
ACTIVE

RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW

More information of a plex:

[root@server1 ~]# vxprint -g vgp1417 -l lvol1-01
Plex:
lvol1-01
info:
len=6291456
type:
layout=CONCAT
state:
state=ACTIVE kernel=ENABLED io=read-write
assoc:
vol=lvol1 sd=vgp1417_04-01
flags:
busy complete
mediatype: hdd

Subdisks
Subdisks of a particular disk group:
[root@server1 ~]# vxprint -st -g vgp1417
SD NAME
PLEX
DISK
DISKOFFS LENGTH
SV NAME
PLEX
VOLNAME NVOLLAYR LENGTH
SC NAME
PLEX
CACHE
DISKOFFS LENGTH
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd
sd

vgp1417_04-01
vgp1417_04-02
vgp1417_04-03
vgp1417_04-04
vgp1417_05-01
vgp1417_05-02
vgp1417_05-03
vgp1417_05-04
vgp1417_06-01
vgp1417_06-02
vgp1417_06-03
vgp1417_07-01
vgp1417_07-02
vgp1417_08-01
vgp1417_08-02
vgp1417_08-03
vgp1417_08-04
vgp1417_09-01

lvol1-01
lvol9-01
lvol10-01
lvol6-01
lvol5-01
lvol7-01
lvol11-01
lvol7-01
lvol2-01
lvol11-01
lvol8-01
lvol8-01
lvol8-01
lvol3-01
lvol4-01
lvol6-01
lvol8-01
lvol8-01

vgp1417_04
vgp1417_04
vgp1417_04
vgp1417_04
vgp1417_05
vgp1417_05
vgp1417_05
vgp1417_05
vgp1417_06
vgp1417_06
vgp1417_06
vgp1417_07
vgp1417_07
vgp1417_08
vgp1417_08
vgp1417_08
vgp1417_08
vgp1417_09

[COL/]OFF DEVICE
[COL/]OFF AM/NM
[COL/]OFF DEVICE

MODE
MODE
MODE

0
6291456 0
emc0_084b ENA
6291456 12582912 0
emc0_084b ENA
18874368 27262976 0
emc0_084b ENA
46137344 10485760 4194304 emc0_084b ENA
0
4194304 0
emc0_084f ENA
4194304 31457280 0
emc0_084f ENA
35651584 16155104 0
emc0_084f ENA
51806688 12582912 31457280 emc0_084f ENA
0
20971520 0
emc0_0853 ENA
20971520 25787936 16155104 emc0_0853 ENA
46759456 23780416 114269008 emc0_0853 ENA
0
56312048 0
emc0_0857 ENA
56312048 14227824 138049424 emc0_0857 ENA
0
2097152 0
emc0_085b ENA
2097152 6291456 0
emc0_085b ENA
8388608 4194304 0
emc0_085b ENA
12582912 57956960 56312048 emc0_085b ENA
0
17592064 152277248 emc0_526f ENA

More information of a subdisk:

[root@server1 ~]# vxprint -g vgp1417 -l vgp1417_04
Disk:
vgp1417_04
info:
diskid=1325653859.61.server1
assoc:
device=emc0_084b type=auto
flags:
autoconfig
device:
path=/dev/vx/dmp/emc0_084bs3
devinfo: publen=70539872 privlen=65536
mediatype: hdd

Performance monitoring
When having I/O issues the standard command you would use could be iostat:
[root@server1 ~]# iostat -mdx
Linux 2.6.18-274.3.1.el5 (server1)
Device:
cciss/c0d0
cciss/c0d0p1
cciss/c0d0p2
sda
sdb
sdc
sdd
sde
sdf
sdg
.
.

rrqm/s
6.45
0.00
6.45
0.00
0.00
0.00
0.00
0.00
0.00
0.00

wrqm/s
40.26
0.00
40.26
0.00
0.00
0.00
0.00
0.00
0.00
0.00

06/07/2012

r/s
w/s
3.54 25.84
0.00 0.00
3.54 25.84
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00

rMB/s
0.05
0.00
0.05
0.00
0.00
0.00
0.00
0.00
0.00
0.00

wMB/s avgrq-sz avgqu-sz

0.17
15.73
0.82
0.00
31.97
0.00
0.17
15.73
0.82
0.00
3.98
0.00
0.00
9.71
0.00
0.00
9.71
0.00
0.00
9.71
0.00
0.00
9.71
0.00
0.00
9.71
0.00
0.00
9.71
0.00

Remark:
As stated in man pages if you do not specify interval and count parameter displayed
statistics are since server boot time so not really relevant
[root@server1 ~]# iostat -md -p sda 1 2
Linux 2.6.18-274.3.1.el5 (server1)
06/07/2012
Device:
sda

tps
0.00

MB_read/s
0.00

MB_wrtn/s
0.00

MB_read
7

MB_wrtn
0

Device:
sda

tps
0.00

MB_read/s
0.00

MB_wrtn/s
0.00

MB_read
0

MB_wrtn
0

You can also use nmon.

With the command we have seen in previous chapter you are now able to identify to
which disk group/volume/VXVM disks it is associated
But even more interesting you can use VxVM command to get detailed information
of disk using Operation System disk names:
[root@server1 ~]# vxdmpadm iostat show pathname=sdax
cpu usage = 374381us
per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
PATHNAME
READS
WRITES
READS
WRITES
READS
WRITES
sdax
1182902
4756775 328093320 294705430
5.10
20.82
[root@server1 ~]# vxdmpadm iostat show dmpnodename=sdax
cpu usage = 375891us
per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
PATHNAME
READS
WRITES
READS
WRITES
READS
WRITES
sdax
1183412
4762187 328098810 295062539
5.10
20.82
sdh
1133886
4773058 324904565 294251856
5.35
20.75
[root@server1 ~]# vxdmpadm iostat show pathname=sdax interval=2 count=2

await
28.01
21.14
28.01
1.87
4.51
1.83
1.66
2.36
2.51
2.34

svctm
5.13
14.42
5.13
1.87
4.51
1.83
1.66
2.36
2.51
2.34

%util
15.07
0.00
15.07
0.00
0.00
0.00
0.00
0.00
0.00
0.00

PATHNAME
sdax
sdax

cpu usage = 374396us

per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
READS
WRITES
READS
WRITES
READS
WRITES
1182902
4756777 328093320 294705543
5.10
20.82
0

0.00

0.00

[root@server1 ~]# vxdmpadm -e iostat show pathname=sdax

cpu usage = 374509us
per cpu memory = 1114112b
ERROR I/Os
PATHNAME
READS
WRITES
sdax
0
0
As well as using VxVM disk names:
[root@server1 ~]# vxdmpadm -q iostat show dmpnodename=emc0_084b
cpu usage = 374484us
per cpu memory = 1114112b
QUEUED I/Os
PENDING I/Os
DMPNODENAME
READS
WRITES
emc0_084b
0
0
0
[root@server1 ~]# vxdmpadm -z iostat show dmpnodename=emc0_084b
cpu usage = 374558us
per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
PATHNAME
READS
WRITES
READS
WRITES
READS
WRITES
sdax
1182910
4756814 328093384 294707962
5.10
20.82
sdh
1133430
4767643 324899821 293884532
5.35
20.76
[root@server1 ~]# vxdmpadm -u bytes -z iostat show dmpnodename=emc0_084b
cpu usage = 374888us
per cpu memory = 1114112b
OPERATIONS
BYTES
AVG TIME(ms)
PATHNAME
READS
WRITES
READS
WRITES
READS
WRITES
sdax
1183012
4760612 480699392 707400192
5.10
20.82
sdh
1133494
4771329 3140286976 287247360
5.35
20.76
You can display information of controllers to see if your multipathing policy is
working well:
[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c0
cpu usage = 374524us
per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
CTLRNAME
READS
WRITES
READS
WRITES
READS
WRITES
c0
135753949 218251362 3572140375 2841606614
7.62
7.10
[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c2
cpu usage = 374541us
per cpu memory = 1114112b
OPERATIONS
BLOCKS
AVG TIME(ms)
CTLRNAME
READS
WRITES
READS
WRITES
READS
WRITES
c2
135349009 215663711 3357968941 2737547261
7.96
6.92
Those interesting information can also be display at disk group level:
[root@server1 ~]# vxstat -u m -g vgp1417
OPERATIONS
BLOCKS
TYP NAME
READ
WRITE
READ
vol lvol1
827072
413478
17059m
vol lvol2
761459
357294
3465m
vol lvol3
9518191
7008657
175801m
vol lvol4
610864 15518552
259528m
vol lvol5
170534
303104
886m
vol lvol6
2646674
2379866
69607m
vol lvol7
16854588 20012885
206030m
vol lvol8
79042007 128500851
899927m
vol lvol9
301556
4788315
2090m
vol lvol10
980251
4487076
287228m
vol lvol11
82761
10693
338m

WRITE
29032m
2781m
56452m
176116m
998m
16601m
119118m
689835m
74546m
193539m
82m

AVG TIME(ms)
READ WRITE
1.51
7.34
4.46 10.80
0.26
0.02
1.86
0.19
4.50
8.68
1.19
1.66
0.10
0.16
0.04
0.02
3.00
0.83
3.20
0.62
2.00 11.79

Remark
-d, -p, -s and -u m options are interesting to display information at different
granularity level. -r option can also be used to reset statistics and focus on current
period only (to remove boot time statistics or reorganization operations)
There is also vxtrace command to trace operations on volumes but I have, most
probably, not really understood its added value as the output is too much verbose
and does not contains any synthetic information:
[root@server1 ~]# vxtrace -g vgp1417 -o disk lvol8
57 START write disk emc0_0857 op 0 block 50782432 len 16
57 END write disk emc0_0857 op 0 block 50782432 len 16 time 3
58 START write disk emc0_0857 op 0 block 195779 len 1
58 END write disk emc0_0857 op 0 block 195779 len 1 time 1
59 START write disk emc0_0857 op 0 block 1246640 len 16
59 END write disk emc0_0857 op 0 block 1246640 len 16 time 2
.
.
Remark:
I/O statistics are gathered by default but when you system is stable or when you
have high I/O demand it may be interesting to deactivate it to maximize throughput.
To check if statistics are gathered:
[root@server1 ~]# vxtune vol_stats_enable
1
Then to de-activate them, as well as for DMP part (not possible to check if there are
activated or not):
[root@server1 ~]# vxtune vol_stats_enable 0
[root@server1 ~]# vxdmpadm iostat stop
If you have slow response from a device group it may be also interesting to check if
you have running operation in background:
[root@server1 ~]# vxtask -h list
TASKID PTID TYPE/STATE
PCT
PROGRESS
Even if default VxVM parameters suit most workloads you can change them using
vxdmpadm command. To display tunable VxDMP parameters use:
[root@server1 ~]# vxdmpadm gettune all
Tunable
Current Value Default Value
------------------------------------------ ------------dmp_cache_open
on
on
dmp_daemon_count
10
10
dmp_delayq_interval
15
15
dmp_enable_restore
on
on
dmp_fast_recovery
on
on
dmp_health_time
60
60
dmp_log_level
1
1
dmp_low_impact_probe
on
on
dmp_lun_retry_timeout
0
0
dmp_path_age
300
300
dmp_pathswitch_blks_shift
9
9
dmp_probe_idle_lun
on
on
dmp_probe_threshold
5
5
dmp_queue_depth
32
32
dmp_restore_cycles
10
10
dmp_restore_interval
300
300
dmp_restore_policy
check_disabled
check_disabled
dmp_retry_count
5
5
dmp_scsi_timeout
20
20
dmp_sfg_threshold
1
1
dmp_stat_interval
1
1
dmp_monitor_fabric
on
on
dmp_monitor_osevent
on
on
dmp_native_support
off
off
Remark:
You may also want to use vxbench available in VRTSspt package

References
Veritas Volume Manager documentation

Veritas Volume Manager

Veritas VXDMP
Basic I/O Monitoring on Linux
Understanding Linux Load Average Part 1
Understanding Linux Load Average Part 2
Understanding Linux Load Average Part 3