IB HCA Firmware

From UF HPC Wiki

Mellanox firmware for our IB cards resides in /dist/hpc/mellanox.

We have a few different flavors of IB HCAs on the cluster. Most HCAs are of the "Lion Cub" variety. We have two different kinds of these, corresponding to two different board revisions.

If the card is a "Topspin card" (purchased with PhaseIIa), use firmware file fw-25208-4_8_200-MHEL-CF128-T.bin.

If the card is a "Mellanox card" (purchased with PhaseIIb), use firmware file fw-25208-4_8_200-MHEA28-1TC_A1-A3.bin.

You can tell if a node has a "Topspin card" via tvflash:

[root@r1a-s42 ~]# tvflash -i
HCA #0: MT25208 Tavor Compat, Lion Cub, revision A0
  Primary image is v4.7.600 build 3.2.0.110, with label 'HCA.LionCub.A0'
  Secondary image is v4.7.600 build 3.2.0.82, with label 'HCA.LionCub.A0'

  Vital Product Data
    Product Name: Lion cub
    P/N: 99-00026-01
    E/C: Rev: B04
    S/N: TS0533X00231            
    Freq/Power: PW=10W;PCIe 8X
    Date Code: 0533
    Checksum: Ok

Note that the P/N and S/N are Cisco/Topspin part and serial numbers.

Here is tvflash output for a "Mellanox card":

[root@submit ~]# tvflash -i
HCA #0: MT25208 Tavor Compat, Lion Cub, revision A0
  Primary image is v4.7.600 build 3.2.0.110, with label 'HCA.LionCub.RevC.A0'
  Secondary image is valid, unknown source

  Vital Product Data
    Product Name: Lion cub
    P/N: MHEA28-1TC           
    E/C: A1
    S/N: MT0614X00681            
    Freq/Power: PCIe x8         
    Checksum: Ok
    Date Code: N/A

OFED's ibv_devinfo command can also be used to see firmware-related information. Here is typical output of ibv_devinfo:

[root@r3a-s20 ~]# ibv_devinfo 
hca_id: mthca0
        fw_ver:                         4.8.200
        node_guid:                      0005:ad00:0004:d1f8
        sys_image_guid:                 0002:c900:0100:d050
        vendor_id:                      0x02c9
        vendor_part_id:                 25208
        hw_ver:                         0xA0
        board_id:                       MT_00A0000001
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 2
                        port_lid:               79
                        port_lmc:               0x00

                port:   2
                        state:                  PORT_DOWN (1)
                        max_mtu:                2048 (4)
                        active_mtu:             512 (2)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00