Some Linux InfiniBand Diag tools

This page for now is just quick list of some InfiniBand commands. InfiniBand is a high-performance, multi-purpose network architecture based on a switch design, called "switched fabric.

# rpm -qi infiniband-diags 
Name        : infiniband-diags             Relocations: (not relocatable)
Version     : 1.6.4                             Vendor: Oracle America
Release     : 1.el6                         Build Date: Sat 16 Aug 2014 08:28:07 PM PDT
Install Date: Tue 25 Nov 2014 12:42:23 AM PST      Build Host: moj-host.jasam.ca
Group       : System Environment/Libraries   Source RPM: infiniband-diags-1.6.4-1.el6.src.rpm
Size        : 1050955                          License: GPLv2 or BSD
Signature   : RSA/8, Mon 25 Aug 2014 05:54:22 PM PDT, Key ID 72f97b74ec551f03
URL         : http://openfabrics.org/
Summary     : OpenFabrics Alliance InfiniBand Diagnostic Tools
Description :
This package provides IB diagnostic programs and scripts needed to diagnose an IB subnet.


# lsmod | grep ib 
ib_umad                12018  0
ib_mad                 40449  1 ib_umad
ib_core                75061  2 ib_umad,ib_mad
libahci                23557  1 ahci

# ibaddr 
GID fe80::10:e000:14b:ff2 LID start 0x32 end 0x32

# ibnodes 
Ca      : 0x0010e000014b0810 ports 2 "host1 mlx4_0"
Ca      : 0x0021280001fc613e ports 2 "host2 mlx4_0"
Ca      : 0x0010e000013438f0 ports 2 "host38 mlx4_0"
Ca      : 0x0010e00001343a30 ports 2 "host39 mlx4_0"
Ca      : 0x0010e000012964a0 ports 2 "sparc32 mlx4_0"
Ca      : 0x0010e000014b0760 ports 2 "sparc2 mlx4_1"
Ca      : 0x0010e00001341e50 ports 2 "MT25408 ConnectX Mellanox Technologies"
Ca      : 0x0010e000014b1820 ports 2 "MT25408 ConnectX Mellanox Technologies"
Ca      : 0x0010e000014b0ff0 ports 2 "qasparc2 mlx4_0"
Switch  : 0x002128f56901a0a0 ports 36 "SUN DCS 36P QDR ibsw 10.x.x.x" enhanced port 0 lid 1 lmc 0


#  ibhosts 
Ca      : 0x0010e000014b0810 ports 2 "host1 mlx4_0"
Ca      : 0x0021280001fc613e ports 2 "host2 mlx4_0"
Ca      : 0x0010e000013438f0 ports 2 "host38 mlx4_0"
Ca      : 0x0010e00001343a30 ports 2 "host39 mlx4_0"
Ca      : 0x0010e000012964a0 ports 2 "sparc32 mlx4_0"
Ca      : 0x0010e000014b0760 ports 2 "sparc2 mlx4_1"
Ca      : 0x0010e00001341e50 ports 2 "MT25408 ConnectX Mellanox Technologies"
Ca      : 0x0010e000014b1820 ports 2 "MT25408 ConnectX Mellanox Technologies"
Ca      : 0x0010e000014b0ff0 ports 2 "qasparc2 mlx4_0"


# iblinkinfo 
CA: host1 mlx4_0:
      0x0010e000014b0811  52 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   32[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
      0x0010e000014b0812  57 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   31[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: host2 mlx4_0
      0x0021280001fc613f  39 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   30[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
      0x0021280001fc6140  42 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   35[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: sparc38 mlx4_0:
      0x0010e000013438f1  35 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   22[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
      0x0010e000013438f2  36 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   21[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: sparc39 mlx4_0:
      0x0010e00001343a31  33 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   20[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
      0x0010e00001343a32  34 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   19[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: sparc32 mlx4_0:
      0x0010e000012964a1  41 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   11[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: qasparc-2 mlx4_1:
      0x0010e000014b0762  49 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1    9[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: MT25408 ConnectX Mellanox Technologies:
      0x0010e00001341e51  47 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1    7[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
      0x0010e00001341e52  48 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1    8[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
CA: MT25408 ConnectX Mellanox Technologies:
      0x0010e000014b1821  51 1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1    1[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )
Switch: 0x002128f56901a0a0 SUN DCS 36P QDR ibsw 10.x.x.x:
           1    1[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      51  1[  ] "MT25408 ConnectX Mellanox Technologies" ( )
           1    2[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1    3[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    4[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1    5[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    6[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1    7[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      47    1[  ] "MT25408 ConnectX Mellanox Technologies" ( )
           1    8[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      48    2[  ] "MT25408 ConnectX Mellanox Technologies" ( )
           1    9[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      49    2[  ] "qasparc-2 mlx4_1" ( )
           1   10[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      50    2[  ] "qasparc-2 mlx4_0" ( )
           1   11[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      41    1[  ] "sparc32 mlx4_0" ( )
           1   12[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   13[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   14[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   15[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1   16[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   17[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1   18[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1   19[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      34    2[  ] "sparc39 mlx4_0" ( )
           1   20[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      33    1[  ] "sparc39 mlx4_0" ( )
           1   21[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      36    2[  ] "sparc38 mlx4_0" ( )
           1   22[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      35    1[  ] "sparc38 mlx4_0" ( )
           1   23[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   24[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   25[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   26[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   27[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   28[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   29[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   30[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      39    1[  ] "host2 mlx4_0" ( )
           1   31[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      57    2[  ] "host1 mlx4_0" ( )
           1   32[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      52    1[  ] "host1 mlx4_0" ( )
           1   33[  ] ==(                Down/ Polling)==>             [  ] "" ( )
           1   34[  ] ==(                Down/Disabled)==>             [  ] "" ( )
           1   35[  ] ==( 4X10.0 Gbps Active/  LinkUp)==>      42    2[  ] "host2 mlx4_0" ( )
           1   36[  ] ==(                Down/ Polling)==>             [  ] "" ( )
CA: qasparc-2 mlx4_0:
      0x0010e000014b0ff2 50 2[  ] ==( 4X10.0 Gbps Active/  LinkUp)==> 1   10[  ] "SUN DCS 36P QDR ibsw 10.x.x.x" ( )

# ibnetdiscover -l 
Ca       : 0x0010e000014b0810 ports 2 devid 0x1003 vendid 0x2c9 "host1 mlx4_0"
Ca       : 0x0021280001fc613e ports 2 devid 0x673c vendid 0x2c9 "host2 mlx4_0"
Ca       : 0x0010e000013438f0 ports 2 devid 0x1003 vendid 0x2c9 "sparc38 mlx4_0"
Ca       : 0x0010e00001343a30 ports 2 devid 0x1003 vendid 0x2c9 "sparc39 mlx4_0"
Ca       : 0x0010e000012964a0 ports 2 devid 0x1003 vendid 0x2c9 "sparc32 mlx4_0"
Ca       : 0x0010e000014b0760 ports 2 devid 0x1003 vendid 0x2c9 "qasparc-2 mlx4_1"
Ca       : 0x0010e00001341e50 ports 2 devid 0x1003 vendid 0x2c9 "MT25408 ConnectX Mellanox Technologies"
Ca       : 0x0010e000014b1820 ports 2 devid 0x1003 vendid 0x2c9 "MT25408 ConnectX Mellanox Technologies"
Ca       : 0x0010e000014b0ff0 ports 2 devid 0x1003 vendid 0x2c9 "qasparc-2 mlx4_0"
Switch   : 0x002128f56901a0a0 ports 36 devid 0xbd36 vendid 0x2c9 "SUN DCS 36P QDR ibsw 10.1.3.6"

We see here there are 2 IB cards, each has 2 ports

#  ibstat -s 
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.11.1280
        Hardware version: 0
        Node GUID: 0x0010e000014b0ff0
        System image GUID: 0x0010e000014b0ff3
CA 'mlx4_1'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.11.1280
        Hardware version: 0
        Node GUID: 0x0010e000014b0760
        System image GUID: 0x0010e000014b0763

We see that 2 ports are down

# ibstatus 
Infiniband device 'mlx4_0' port 1 status:
        default gid:     fe80:0000:0000:0000:0010:e000:014b:0ff1
        base lid:        0x0
        sm lid:          0x0
        state:           1: DOWN
        phys state:      3: Disabled
        rate:            40 Gb/sec (4X QDR)
        link_layer:      IB

Infiniband device 'mlx4_0' port 2 status:
        default gid:     fe80:0000:0000:0000:0010:e000:014b:0ff2
        base lid:        0x32
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      IB

Infiniband device 'mlx4_1' port 1 status:
        default gid:     fe80:0000:0000:0000:0010:e000:014b:0761
        base lid:        0x0
        sm lid:          0x0
        state:           1: DOWN
        phys state:      3: Disabled
        rate:            40 Gb/sec (4X QDR)
        link_layer:      IB

Infiniband device 'mlx4_1' port 2 status:
        default gid:     fe80:0000:0000:0000:0010:e000:014b:0762
        base lid:        0x31
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      IB

Back to the main page