为vSAN启用RDMA

交换机上启用LLDP、PFC和DCBx,PFC优先级为3(默认值),DCBx的标准为IEEE(默认值)。

dcb pfc
#
interface 25GE1/0/10
 description Host-01
 port link-type trunk
 undo port trunk allow-pass vlan 1
 port trunk allow-pass vlan 125
 stp edged-port enable
 lldp tlv-enable dcbx
 dcb pfc enable mode manual
#
interface 25GE1/0/12
 description Host-02
 port link-type trunk
 undo port trunk allow-pass vlan 1
 port trunk allow-pass vlan 125
 stp edged-port enable
 lldp tlv-enable dcbx
 dcb pfc enable mode manual
#
lldp enable
#

通过主机-配置-物理适配器确认用于vSAN网卡的设备位置和端口,以下示例中是0000:8a:00的端口2(vmnic3)和0000:8b:00的端口1(vmnic4),首先查看一下LLDP和DCBx的相关配置。需要先安装MFT才能使用相关命令。

[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8a:00.1 query|grep -iE "dcb|lldp"
        LLDP_NB_DCBX_P1                             False(0)            
        LLDP_NB_RX_MODE_P1                          OFF(0)              
        LLDP_NB_TX_MODE_P1                          OFF(0)              
        LLDP_NB_DCBX_P2                             False(0)            
        LLDP_NB_RX_MODE_P2                          OFF(0)              
        LLDP_NB_TX_MODE_P2                          OFF(0)              
        DCBX_IEEE_P1                                True(1)             
        DCBX_CEE_P1                                 True(1)             
        DCBX_WILLING_P1                             True(1)             
        DCBX_IEEE_P2                                True(1)             
        DCBX_CEE_P2                                 True(1)             
        DCBX_WILLING_P2                             True(1)             

[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8b:00.0 query|grep -iE "dcb|lldp"
        LLDP_NB_DCBX_P1                             False(0)            
        LLDP_NB_RX_MODE_P1                          OFF(0)              
        LLDP_NB_TX_MODE_P1                          OFF(0)              
        LLDP_NB_DCBX_P2                             False(0)            
        LLDP_NB_RX_MODE_P2                          OFF(0)              
        LLDP_NB_TX_MODE_P2                          OFF(0)              
        DCBX_IEEE_P1                                True(1)             
        DCBX_CEE_P1                                 True(1)             
        DCBX_WILLING_P1                             True(1)             
        DCBX_IEEE_P2                                True(1)             
        DCBX_CEE_P2                                 True(1)             
        DCBX_WILLING_P2                             True(1)       

发现LLDP没有启动,启用LLDP和DCBx,DCBx IEEE已启用。

[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8a:00.1 set LLDP_NB_DCBX_P2=1 LLDP_NB_RX_MODE_P2=2 LLDP_NB_TX_MODE_P2=2 DCBX_WILLING_P2=1 DCBX_IEEE_P2=1 

Device #1:
----------

Device type:        ConnectX5           
Name:               MCX512A-ACU_Ax_Bx   
Description:        ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
Device:             0000:8a:00.1        

Configurations:                                          Next Boot       New
        LLDP_NB_DCBX_P2                             False(0)             True(1)             
        LLDP_NB_RX_MODE_P2                          OFF(0)               ALL(2)              
        LLDP_NB_TX_MODE_P2                          OFF(0)               ALL(2)              
        DCBX_WILLING_P2                             True(1)              True(1)             
        DCBX_IEEE_P2                                True(1)              True(1)             

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8b:00.0 set LLDP_NB_DCBX_P1=1 LLDP_NB_RX_MODE_P1=2 LLDP_NB_TX_MODE_P1=2 DCBX_WILLING_P1=1 DCBX_IEEE_P1=1 

Device #1:
----------

Device type:        ConnectX5           
Name:               MCX512A-ACU_Ax_Bx   
Description:        ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
Device:             0000:8b:00.0        

Configurations:                                          Next Boot       New
        LLDP_NB_DCBX_P1                             False(0)             True(1)             
        LLDP_NB_RX_MODE_P1                          OFF(0)               ALL(2)              
        LLDP_NB_TX_MODE_P1                          OFF(0)               ALL(2)              
        DCBX_WILLING_P1                             True(1)              True(1)             
        DCBX_IEEE_P1                                True(1)              True(1)             

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

重启后查看网卡当前配置,确认LLDP和DCBx均已启用

[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8a:00.1 query|grep -iE "dcb|lldp"
        LLDP_NB_DCBX_P1                             False(0)            
        LLDP_NB_RX_MODE_P1                          OFF(0)              
        LLDP_NB_TX_MODE_P1                          OFF(0)              
        LLDP_NB_DCBX_P2                             True(1)             
        LLDP_NB_RX_MODE_P2                          ALL(2)              
        LLDP_NB_TX_MODE_P2                          ALL(2)              
        DCBX_IEEE_P1                                True(1)             
        DCBX_CEE_P1                                 True(1)             
        DCBX_WILLING_P1                             True(1)             
        DCBX_IEEE_P2                                True(1)             
        DCBX_CEE_P2                                 True(1)             
        DCBX_WILLING_P2                             True(1)             
[root@yaoge123:~] /opt/mellanox/bin/mlxconfig -d 0000:8b:00.0 query|grep -iE "dcb|lldp"
        LLDP_NB_DCBX_P1                             True(1)             
        LLDP_NB_RX_MODE_P1                          ALL(2)              
        LLDP_NB_TX_MODE_P1                          ALL(2)              
        LLDP_NB_DCBX_P2                             False(0)            
        LLDP_NB_RX_MODE_P2                          OFF(0)              
        LLDP_NB_TX_MODE_P2                          OFF(0)              
        DCBX_IEEE_P1                                True(1)             
        DCBX_CEE_P1                                 True(1)             
        DCBX_WILLING_P1                             True(1)             
        DCBX_IEEE_P2                                True(1)             
        DCBX_CEE_P2                                 True(1)             
        DCBX_WILLING_P2                             True(1)             

查看网卡的DCB状态,模式为IEEE,PFC已启用且优先级为3(0 0 0 1 0 0 0 0)

[root@2288Hv6-05:~] esxcli network nic dcb status get -n vmnic3
   Nic Name: vmnic3
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities: 
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 8
         PFC Traffic Classes: 8
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration: 
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 8
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 13 13 13 13 12 12 12 12
         RX Bandwidth Per TC: 13 13 13 13 12 12 12 12
         TSA Assignment Table Per TC: 2 2 2 2 2 2 2 2
         Priority Assignment Per TC: 1 0 2 3 4 5 6 7
         Recommended TC Bandwidth Per TC: 13 13 13 13 12 12 12 12
         Recommended TSA Assignment Per TC: 2 2 2 2 2 2 2 2
         Recommended Priority Assignment Per TC: 1 0 2 3 4 5 6 7
   IEEE PFC Configuration: 
         Number Of Traffic Classes: 8
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 0 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps: 
[root@2288Hv6-05:~] esxcli network nic dcb status get -n vmnic4
   Nic Name: vmnic4
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities: 
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 8
         PFC Traffic Classes: 8
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration: 
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 8
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 13 13 13 13 12 12 12 12
         RX Bandwidth Per TC: 13 13 13 13 12 12 12 12
         TSA Assignment Table Per TC: 2 2 2 2 2 2 2 2
         Priority Assignment Per TC: 1 0 2 3 4 5 6 7
         Recommended TC Bandwidth Per TC: 13 13 13 13 12 12 12 12
         Recommended TSA Assignment Per TC: 2 2 2 2 2 2 2 2
         Recommended Priority Assignment Per TC: 1 0 2 3 4 5 6 7
   IEEE PFC Configuration: 
         Number Of Traffic Classes: 8
         PFC Configuration: 0 0 0 1 0 0 0 0
         Macsec Bypass Capability Is Enabled: 0
         Round Trip Propagation Delay Of Link: 0
         Sent PFC Frames: 0 0 0 0 0 0 0 0
         Received PFC Frames: 0 0 0 0 0 0 0 0
   DCB Apps: 

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据