Back to the main page

RedHat Device Mapper Multipath (DM-Multipath)

Here is some examples working on RHEL6.3 and testing how DM-Multipathing works. It allows configuration of multiple I/O paths from server SunFire X4200 to StorEdge 6120 (50G LUN). The picture shows their connection through FC switch.




Verify connections from FC switch:
# show topology
  Port   Local Local                   Remote Remote                  Unique
  Number Type  PortWWN                 Type   NodeWWN                 ID
  ------ ----- -------                 ------ -------                 ------
  2      F     20:02:00:c0:dd:01:b0:65 N      10:00:00:03:ba:cc:c7:50 020200   P # storage 6120
  6      F     20:06:00:c0:dd:01:b0:65 N      10:00:00:03:ba:cc:c7:38 020600   P # storage 6120
  11     F     20:0b:00:c0:dd:01:b0:65 N      20:00:00:00:c9:4a:bc:b2 020b00   P # rhel
  15     F     20:0f:00:c0:dd:01:b0:65 N      20:00:00:00:c9:48:bf:b7 020f00   P # rhel

Verify LUN permissions on the 6120:
# lun perm list
lun     slice   WWN                     Group Name      Group Perm      WWN Perm        Effective Perm
--------------------------------------------------------------------------------------------------------
0       0       default                 --              --              rw              rw
1       1       default                 --              --              none            none
--------------------------------------------------------------------------------------------------------

Check corresponding "fc" files in directory /sys/class on RHEL box.
Names are: 
Transport: /sys/class/fc_transport/targetH:B:T/
Remote Port: /sys/class/fc_remote_ports/rport-H:B-R/
Host: /sys/class/fc_host/hostH/

host numbers are designated by H
bus numbers are B
targets are T
logical unit numbers (LUNs) are L
remote port numbers are R
# foreach i ( `ls | grep fc` )
foreach? echo ----------- directory `pwd`/$i ------------
foreach? ls -la $i
foreach? end
----------- directory /sys/class/fc_host ------------
lrwxrwxrwx  1 root root host5 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/fc_host/host5
lrwxrwxrwx  1 root root host6 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/fc_host/host6
----------- directory /sys/class/fc_remote_ports ------------
lrwxrwxrwx  1 root root rport-5:0-0 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-0/fc_remote_ports/rport-5:0-0
lrwxrwxrwx  1 root root rport-5:0-1 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-1/fc_remote_ports/rport-5:0-1
lrwxrwxrwx  1 root root rport-5:0-2 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-2/fc_remote_ports/rport-5:0-2
lrwxrwxrwx  1 root root rport-5:0-3 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-3/fc_remote_ports/rport-5:0-3
lrwxrwxrwx  1 root root rport-5:0-5 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-5/fc_remote_ports/rport-5:0-5
lrwxrwxrwx  1 root root rport-6:0-0 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-0/fc_remote_ports/rport-6:0-0
lrwxrwxrwx  1 root root rport-6:0-1 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-1/fc_remote_ports/rport-6:0-1
lrwxrwxrwx  1 root root rport-6:0-2 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-2/fc_remote_ports/rport-6:0-2
lrwxrwxrwx  1 root root rport-6:0-3 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-3/fc_remote_ports/rport-6:0-3
lrwxrwxrwx  1 root root rport-6:0-4 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-4/fc_remote_ports/rport-6:0-4
----------- directory /sys/class/fc_transport ------------
lrwxrwxrwx  1 root root target5:0:0 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-2/target5:0:0/fc_transport/target5:0:0
lrwxrwxrwx  1 root root target5:0:1 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/rport-5:0-3/target5:0:1/fc_transport/target5:0:1
lrwxrwxrwx  1 root root target6:0:0 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-3/target6:0:0/fc_transport/target6:0:0
lrwxrwxrwx  1 root root target6:0:1 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/rport-6:0-4/target6:0:1/fc_transport/target6:0:1
----------- directory /sys/class/fc_vports ------------
total 0
drwxr-xr-x  2 root root 0 Jan 23 00:15 .
drwxr-xr-x 47 root root 0 Jan 23 08:15 ..

For example, to find the first WWN of the 6120 (compare with "show topology" from FC switch):
# cat /sys/class/fc_transport/target5:0:0/node_name
0x10000003baccc750
# cat /sys/class/fc_transport/target6:0:0/node_name
0x10000003baccc750

And to find the second WWN of the 6120:
# cat /sys/class/fc_transport/target5:0:1/node_name
0x10000003baccc738
# cat /sys/class/fc_transport/target6:0:1/node_name
0x10000003baccc738

To get some more info about FC HBA on RHEL host:
# ls -la /sys/class/fc_host/
lrwxrwxrwx  host5 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:01.0/host5/fc_host/host5
lrwxrwxrwx  host6 -> ../../devices/pci0000:04/0000:04:01.0/0000:05:02.0/host6/fc_host/host6

# ls -la /sys/class/fc_host/host5
-r--r--r-- 1 root root 4096 Jan 23 10:23 active_fc4s
lrwxrwxrwx 1 root root    0 Jan 23 10:23 device -> ../../../host5
-rw-r--r-- 1 root root 4096 Jan 23 10:23 dev_loss_tmo
-r--r--r-- 1 root root 4096 Jan 23 10:23 fabric_name
--w------- 1 root root 4096 Jan 23 10:23 issue_lip
-r--r--r-- 1 root root 4096 Jan 23 10:23 maxframe_size
-r--r--r-- 1 root root 4096 Jan 23 10:23 max_npiv_vports
-r--r--r-- 1 root root 4096 Jan 23 10:23 node_name
-r--r--r-- 1 root root 4096 Jan 23 10:23 npiv_vports_inuse
-r--r--r-- 1 root root 4096 Jan 23 10:23 port_id
-r--r--r-- 1 root root 4096 Jan 23 10:23 port_name
-r--r--r-- 1 root root 4096 Jan 23 10:23 port_state
-r--r--r-- 1 root root 4096 Jan 23 10:23 port_type
drwxr-xr-x 2 root root    0 Jan 23 10:23 power
-r--r--r-- 1 root root 4096 Jan 23 10:23 speed
drwxr-xr-x 2 root root    0 Jan 23 10:23 statistics
lrwxrwxrwx 1 root root    0 Jan 23 08:16 subsystem -> ../../../../../../../class/fc_host
-r--r--r-- 1 root root 4096 Jan 23 10:23 supported_classes
-r--r--r-- 1 root root 4096 Jan 23 10:23 supported_fc4s
-r--r--r-- 1 root root 4096 Jan 23 10:23 supported_speeds
-r--r--r-- 1 root root 4096 Jan 23 10:23 symbolic_name
-rw-r--r-- 1 root root 4096 Jan 23 10:23 tgtid_bind_type
-rw-r--r-- 1 root root 4096 Jan 23 00:15 uevent
--w------- 1 root root 4096 Jan 23 10:23 vport_create
--w------- 1 root root 4096 Jan 23 10:23 vport_delete

[root@unixlab-7 host5]# cat fabric_name
0x100000c0dd01b065
[root@unixlab-7 host5]# cat node_name
0x20000000c948bfb7
[root@unixlab-7 host5]# cat port_name
0x10000000c948bfb7
[root@unixlab-7 host5]# cat port_state
Online
[root@unixlab-7 host5]# cat port_type
NPort (fabric via point-to-point)    
[root@unixlab-7 host5]# cat speed
2 Gbit
[root@unixlab-7 host5]# cat port_id
0x020f00

Also for example more info about specific FC port on FC Switch can be found in directory /sys/class/fc_remote_ports/rport-5:0-0/
After all connections are verified, it's needed to have multipath s/w installed. The yum local repository was created earlier.
#  yum install device-mapper-multipath.i686
Loaded plugins: product-id, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-multipath.i686 0:0.4.9-56.el6 will be installed
--> Processing Dependency: device-mapper-multipath-libs = 0.4.9-56.el6 for package: device-mapper-multipath-0.4.9-56.el6.i686
--> Processing Dependency: libmultipath.so for package: device-mapper-multipath-0.4.9-56.el6.i686
--> Running transaction check
---> Package device-mapper-multipath-libs.i686 0:0.4.9-56.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
Installing:
 device-mapper-multipath       i686    0.4.9-56.el6       local-rhel-6.3          96 k
Installing for dependencies:
 device-mapper-multipath-libs  i686    0.4.9-56.el6       local-rhel-6.3          159 k
Transaction Summary
Install       2 Package(s)
Total download size: 255 k
Installed size: 537 k
Is this ok [y/N]: y
Downloading Packages:
Total                                            5.2 MB/s | 255 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : device-mapper-multipath-libs-0.4.9-56.el6.i686              1/2
  Installing : device-mapper-multipath-0.4.9-56.el6.i686                   2/2
Installed products updated.
  Verifying  : device-mapper-multipath-libs-0.4.9-56.el6.i686              1/2
  Verifying  : device-mapper-multipath-0.4.9-56.el6.i686                   2/2
Installed:
  device-mapper-multipath.i686 0:0.4.9-56.el6
Dependency Installed:
  device-mapper-multipath-libs.i686 0:0.4.9-56.el6
Complete!

Now configure/enable multipath, this creates file /etc/multipath.conf and also starts multipathd daemon.
# /sbin/mpathconf --enable --with_multipathd y
Starting multipathd daemon:                [  OK  ]

# service multipathd status
multipathd (pid  2850) is running...

To flush all unused multipath device maps:
# multipath -F
Jan 23 15:18:42 | mpathe: map in use

To see used multipath device maps (the LUN 50G is here):
# multipath -ll
mpathe (360003baccc75000050e20927000734d9) dm-2 SUN,T4
size=50G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 5:0:0:0 sde 8:64  active ready  running
  |- 5:0:1:0 sdf 8:80  active ready  running
  |- 6:0:0:0 sdg 8:96  active ready  running
  `- 6:0:1:0 sdh 8:112 active ready  running

Another command to be used:
# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: FUJITSU  Model: MAY2073RCSUN72G  Rev: 0501
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: SEAGATE  Model: ST914602SSUN146G Rev: 0400
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: HITACHI  Model: H101414SCSUN146G Rev: SA25
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: TEAC     Model: DV-28SL          Rev: 1.0A
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: SUN      Model: T4               Rev: 0302
  Type:   Direct-Access                    ANSI  SCSI revision: 03
Host: scsi5 Channel: 00 Id: 01 Lun: 00
  Vendor: SUN      Model: T4               Rev: 0302
  Type:   Direct-Access                    ANSI  SCSI revision: 03
Host: scsi6 Channel: 00 Id: 00 Lun: 00
  Vendor: SUN      Model: T4               Rev: 0302
  Type:   Direct-Access                    ANSI  SCSI revision: 03
Host: scsi6 Channel: 00 Id: 01 Lun: 00
  Vendor: SUN      Model: T4               Rev: 0302
  Type:   Direct-Access                    ANSI  SCSI revision: 03

Find block device for the LUN:
# ls -la /dev/mapper/
crw-rw----  1 root root 10, 58 Jan 23 07:22 control
lrwxrwxrwx  1 root root      7 Jan 23 15:22 mpathc -> ../dm-0
lrwxrwxrwx  1 root root      7 Jan 23 15:22 mpathd -> ../dm-1
lrwxrwxrwx  1 root root      7 Jan 23 15:45 mpathe -> ../dm-2
lrwxrwxrwx  1 root root      7 Jan 23 15:45 mpathep1 -> ../dm-3

Create ext3 file system, add needed line in /etc/fstab file and mount it:
# mkfs.ext3 -v /dev/mapper/mpathep1
mke2fs 1.41.12 (17-May-2010)
fs_types for mke2fs.conf resolution: 'ext3', 'default'
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13106008 blocks
655300 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             4.4G  884M  3.3G  22% /
tmpfs                 7.9G     0  7.9G   0% /dev/shm
/dev/sda2             4.4G  3.1G  1.1G  74% /.0
/dev/sda3             4.4G  215M  3.9G   6% /var
/dev/sda6              50G  2.9G   45G   7% /.1
/dev/mapper/mpathep1   50G  180M   47G   1% /.2

# df -h /.2
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/mpathep1   50G  180M   47G   1% /.2

And let's now remove one FC cable from FC switch port 11, only "host 5" is active.
{fc_switch}# show topology
  Port   Local Local                   Remote Remote                  Unique
  Number Type  PortWWN                 Type   NodeWWN                 ID
  ------ ----- -------                 ------ -------                 ------
  2      F     20:02:00:c0:dd:01:b0:65 N      10:00:00:03:ba:cc:c7:50 020200   P
  6      F     20:06:00:c0:dd:01:b0:65 N      10:00:00:03:ba:cc:c7:38 020600   P
  15     F     20:0f:00:c0:dd:01:b0:65 N      20:00:00:00:c9:48:bf:b7 020f00   P

{rhel_host}# multipath -ll
mpathe (360003baccc75000050e20927000734d9) dm-2 SUN,T4
size=50G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 5:0:0:0 sde 8:64 active ready  running
  `- 5:0:1:0 sdf 8:80 active ready  running

And return FC cable to port 11 and remove another FC cable from FC switch port 15, now only "host 6" is active.
{rhel_host}# multipath -ll
mpathe (360003baccc75000050e20927000734d9) dm-2 SUN,T4
size=50G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:0:0 sdg 8:96  active ready  running
  `- 6:0:1:0 sdh 8:112 active ready  running

Now the "fdisk" command shows /dev/sdg1 but before when server was connected to port 11, the LUN was /dev/sde1
Disk /dev/sdg: 53.7 GB, 53687091200 bytes
256 heads, 63 sectors/track, 6501 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x45bd0000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1        6501    52424032+  83  Linux

Back to the main page