SAN Add on Solaris with Veritas file system

More disk space is always needed, and SAN add is usual activity for system admin. And after receiving request from customer, or business owner of file system, this has to be designed properly before implementation. Here is the example when using Veritas file system.

It's good idea to have standard form to be filled by requestor, like hostname, mount point, ownership, permissions, size (current, additional), dedicated/shared/replicated LUN.
Once you have this info, you can design the work. Adding new LUN (file system) is simplier then increasing existing one, which has to be planned in more details.
In most cases, you end up adding bigger LUN, mirroring data (logical volume, actually plex within it), then removing plex from smaller LUN, removing LUN from disk group and finally a Storage team can reclaim LUN.

Discover newly allocated LUNs

Once new LUNs are allocated, run commands so system can recognize them.

# devfsadm
# vxdctl enable

Then use script san_sol_info.sh to list SAN LUNs or use one line command (note about ignoring channels 5/6)

# luxadm probe | grep Logical | grep -v /dev/rdsk/c6 | grep -v /dev/rdsk/c5  |awk -F: '{print $2}'

/dev/rdsk/c4t20110002AC0086D2d2s2
....
/dev/rdsk/c4t20110002AC0086D2d29s2

Or the example in the sh shell

# for i in `luxadm probe | grep Logical | grep -v /dev/rdsk/c6 | grep -v /dev/rdsk/c5 | awk -F: '{print $2}'`
> do
> echo $i ": \c" ; luxadm -v display $i | grep capacity | awk -F: '{print $2}'
> done
/dev/rdsk/c4t20110002AC0086D2d2s2 :     16384.000 MBytes
....
/dev/rdsk/c4t20110002AC0086D2d28s2 :    524288.000 MBytes

Prepare LUNs for Veritas use

#  /etc/vx/bin/vxdisksetup -f -i c4t20110002AC0086D2d29 
VxVM vxdisksetup ERROR V-5-2-43 c4t20110002AC0086D2d29: Invalid disk device for vxdisksetup

If you see above error, format/label disk with Solaris command "format" (this may not be necessary all the time). If vxdisksetup still doesn't work, try below command:

# cat format.cmd
label
quit

# foreach i (`cat newluns`)
 foreach? echo --- formating $i
 foreach? format -f format.cmd $i
 foreach? end

#  vxdisk -f init c4t20110002AC0086D2d29

So basically, if you have all new LUNs in file "newluns" you can prepare them for Veritas use with:

# foreach i (`cat newluns`)
 foreach? echo --- Preparing $i for Veritas use -----
 foreach? /etc/vx/bin/vxdisksetup -f -i $i
 foreach? vxdisk -f init $i
 foreach? end

Create Veritas Disk Group

#  vxdg init dgtest dgtest_disk001=c4t20110002AC0086D2d29 

#  vxdg list 
NAME         STATE           ID
dgtest       enabled,cds          1389999403.91.sap05.austin.hp.com

To add additional disk to DG:

#  vxdg -g <dg> adddisk <dg>_disk002=c4t201100xxxxxxxxxxxdx

To remove disk from DG (example from sap05):

#  vxdg -g dg10cl1 rmdisk dg10cl1_disk001 
VxVM vxdg ERROR V-5-1-16414 Disk dg10cl1_disk001 is used by one or more subdisks which are pending to be reclaimed.
 Use "vxdisk reclaim 3pardata0_0_6" to reclaim space used by these subdisks, and retry "vxdg rmdisk" command.
 Note: reclamation is irreversible.

Okay, this is something not seen on HP-UX, but this is Solaris.

# vxdisk reclaim 3pardata0_0_6 
Disk 3pardata0_0_6 : Done.

Now remove disk command works:

#  vxdg -g dg10cl1 rmdisk dg10cl1_disk001

Create Veritas logical volume

#  vxassist -g dgtest make lvol1 16000m 

#  vxprint -hrt 
Disk group: dgtest

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

dg dgtest       default      default  28000    1389999403.91.sap05.austin.hp.com

dm dgtest_disk001 3pardata0_0_27 auto 65536    33483648 -

v  lvol1        -            ENABLED  ACTIVE   16384    SELECT    -        fsgen
pl lvol1-01     lvol1        ENABLED  ACTIVE   16384    CONCAT    -        RW
sd dgtest_disk001-01 lvol1-01 dgtest_disk001 0 16384    0         3pardata0_0_27 ENA

Create Veritas file system

#   mkfs -F vxfs -o bsize=8192,largefiles /dev/vx/rdsk/dgtest/lvol1 
    version 9 layout
    32768000 sectors, 2048000 blocks of size 8192, log size 2048 blocks
    rcq size 128 blocks
    largefiles supported

Add line to /etc/vfstab file and mount file system.

# dgtest
/dev/vx/dsk/dgtest/lvol1 /dev/vx/rdsk/dgtest/lvol1 /dgtest-growing vxfs 2 yes largefiles

Resizing (mostly growing) existing LVOL

For example, to add 50G to lvol, execute (the file system grows automatically)

#  /opt/VRTS/bin/vxresize -g <dg_name> <lvol_name> +50G

Or to resize to desired size execute:

#  /opt/VRTS/bin/vxresize -g <dg_name> <lvol_name> <desired_size><M|G>

The command to show max size of new volume to be created in a DG

#  vxassist -g dg# maxsize

The command to show max grow of lvol in DG

#  vxassist -g dg# maxgrow lvol#

Mirroring LVOL to new LUNs and releasing old LUNs

Sometimes it's needed to mirror LVOLs to new LUNs, and release old ones. The most common situations is because some old stripes are created with too many number of columns, I've seen 18 or 33. Better practice is to have stripe with ncol=2 or 4 (number of new LUNs is the one that can be divided by 4)
Once new LUNs are added to a DG, start mirroring (in background with -b)

#  vxassist -b -g <dg_name> mirror <lvol_name> layout=stripe stripeunit=128k nstripe=4 alloc=<disk1>,<disk2>,<etc>

Use vxtask to monitor progress of mirroring:

# vxtask list

Also use " vxprint -g <dg> -hrt " to make sure mirroring is done when you see all plexes are active.
Once mirroring is done on all LVOLs, dissociate (dis) and remove (rm) old plexes:

#  vxplex -g <dg_name> -o rm dis <plex_name>

Then remove disks from DG

#  vxdg -g <dg_name> rmdisk <disk_name>

Initialize disk again and it's now ready to be returned to Storage Team:

#  /opt/VRTS/bin/vxdisksetup -i <disk_name>

Adding SWAP

Create a Veritas disk group

#  vxdg init dgswap dgswap_disk001=c4t20110002AC0086D2d4 

# vxdg list
NAME         STATE           ID
dgtest       enabled,cds          1390002309.94.sap05.austin.hp.com
dgswap       enabled,cds          1390003875.96.sap05.austin.hp.com

# swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 256,1      16 67108848 67108848

# swap -s
total: 135976k bytes allocated + 25776k reserved = 161752k used, 88976288k available

Create logical volume

#  vxassist -g dgswap make lvol1 32000m 
#  vxprint -g dgswap -hrt

Add swap

# swap -a /dev/vx/dsk/dgswap/lvol1

# swap -s
total: 136816k bytes allocated + 25736k reserved = 162552k used, 121742560k available

# swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 256,1      16 67108848 67108848
/dev/vx/dsk/dgswap/lvol1 349,25000     16 65535984 65535984

Add to /etc/vfstab

# dgswap
/dev/vx/dsk/dgswap/lvol1  -   -    swap    -    no      -

Back to the main page