NVMe over RoCE 对RoCE网卡的固件兼容性有较高要求
- 打开ESXi的SSH并进入维护模式，远程登录ESXi主机
- 用 esxcfg-nics -l 查看网卡列表，用 esxcli network nic get -n 查看网卡驱动和固件版本
[root@yaoge123:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:19:00.0 igbn Up 1000Mbps Full **:**:**:**:**:** 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:19:00.1 igbn Up 1000Mbps Full **:**:**:**:**:** 1500 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:8a:00.0 nmlx5_core Up 25000Mbps Full **:**:**:**:**:** 1500 Mellanox Technologies ConnectX-5 EN NIC; 10/25GbE; dual-port SFP28; PCIe3.0 x8; (MCX512A-ACU)
vmnic3 0000:8a:00.1 nmlx5_core Up 25000Mbps Full **:**:**:**:**:** 1500 Mellanox Technologies ConnectX-5 EN NIC; 10/25GbE; dual-port SFP28; PCIe3.0 x8; (MCX512A-ACU)
vmnic4 0000:8b:00.0 nmlx5_core Up 25000Mbps Full **:**:**:**:**:** 1500 Mellanox Technologies ConnectX-5 EN NIC; 10/25GbE; dual-port SFP28; PCIe3.0 x8; (MCX512A-ACU)
vmnic5 0000:8b:00.1 nmlx5_core Up 25000Mbps Full **:**:**:**:**:** 1500 Mellanox Technologies ConnectX-5 EN NIC; 10/25GbE; dual-port SFP28; PCIe3.0 x8; (MCX512A-ACU)
[root@yaoge123:~] esxcli network nic get -n vmnic2
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 1000BaseCX-SGMII/Full, 10000BaseKR/Full, 25000BaseTwinax/Full
Auto Negotiation: true
Backing DPUId: N/A
Cable Type: FIBRE
Current Message Level: -1
Driver Info:
Bus Info: 0000:8a:00:0
Driver: nmlx5_core
Firmware Version: 16.32.1010
Version: 4.23.0.36
Link Detected: true
Link Status: Up
Name: vmnic2
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: true
Pause TX: true
Supported Ports: FIBRE, DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver: internal
Virtual Address: **:**:**:**:**:**
Wakeon: None
- 在 VMware兼容性 搜索该网卡型号查询对应的驱动和固件版本
- 下载 MFT，选择和ESXi版本匹配的安装包
- 下载的Mellanox-MFT-Tools_*-package.zip解压，将解压出的.zip文件传至ESXi的/tmp，确认上传的.zip文件内根目录就有index.xml
- 下载的Mellanox-NATIVE-NMST_*-package.zip解压，将解压出的.zip文件传至ESXi的/tmp，确认上传的.zip文件内根目录就有index.xml
- 用 esxcli software component apply -d 安装上传的两个.zip，注意要写全路径
[root@yaoge123:/tmp] esxcli software component apply -d /tmp/Mellanox-MFT-Tools_4.26.1.101-1OEM.801.0.0.21495797_22944840.zip
Installation Result
Message: Operation finished successfully.
Components Installed: Mellanox-MFT-Tools_4.26.1.101-1OEM.801.0.0.21495797
Components Removed:
Components Skipped:
Reboot Required: false
DPU Results:
[root@yaoge123:/tmp] esxcli software component apply -d /tmp/Mellanox-NATIVE-NMST_4.26.1.101-1OEM.801.0.0.21495797_22944879.zip
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Components Installed: Mellanox-NATIVE-NMST_4.26.1.101-1OEM.801.0.0.21495797
Components Removed:
Components Skipped:
Reboot Required: true
DPU Results:
- 重启ESXi再打开SSH
- 下载 Firmware ，查找对应网卡的兼容版本固件下载
- 下载的fw-*.bin.zip解压，将解压出的.bin文件上传至ESXi的/tmp
- 在/tmp下执行/opt/mellanox/bin/mlxfwmanager，会自动发现新固件文件，然后添加-u升级，如果已有的版本新要降级再加-f强制刷
[root@yaoge123:/tmp] /opt/mellanox/bin/mlxfwmanager
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX5
Part Number: MCX512A-ACU_Ax_Bx
Description: ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
PSID: MT_0000000425
PCI Device Name: mt4119_pciconf0
Base GUID: ***
Base MAC: ***
Versions: Current Available
FW 16.32.1010 16.34.1002
PXE 3.6.0502 3.6.0700
UEFI 14.25.0017 14.27.0014
Status: Update required
Device #2:
----------
Device Type: ConnectX5
Part Number: MCX512A-ACU_Ax_Bx
Description: ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
PSID: MT_0000000425
PCI Device Name: mt4119_pciconf1
Base GUID: ***
Base MAC: ***
Versions: Current Available
FW 16.32.1010 16.34.1002
PXE 3.6.0502 3.6.0700
UEFI 14.25.0017 14.27.0014
Status: Update required
---------
Found 2 device(s) requiring firmware update. Please use -u flag to perform the update.
[root@yaoge123:/tmp] /opt/mellanox/bin/mlxfwmanager -u
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX5
Part Number: MCX512A-ACU_Ax_Bx
Description: ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
PSID: MT_0000000425
PCI Device Name: mt4119_pciconf0
Base GUID: ***
Base MAC: ***
Versions: Current Available
FW 16.32.1010 16.34.1002
PXE 3.6.0502 3.6.0700
UEFI 14.25.0017 14.27.0014
Status: Update required
Device #2:
----------
Device Type: ConnectX5
Part Number: MCX512A-ACU_Ax_Bx
Description: ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; UEFI Enabled (x86/ARM)
PSID: MT_0000000425
PCI Device Name: mt4119_pciconf1
Base GUID: ***
Base MAC: ***
Versions: Current Available
FW 16.32.1010 16.34.1002
PXE 3.6.0502 3.6.0700
UEFI 14.25.0017 14.27.0014
Status: Update required
---------
Found 2 device(s) requiring firmware update...
Perform FW update? [y/N]: y
Device #1: Updating FW ...
FSMST_INITIALIZE - OK
Writing Boot image component - OK Done
Device #2: Updating FW ...
FSMST_INITIALIZE - OK
Writing Boot image component - OK Done
Restart needed for updates to take effect.
- 重启后再检查网卡的固件版本
[root@yaoge123:~] esxcli network nic get -n vmnic2
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 1000BaseCX-SGMII/Full, 10000BaseKR/Full, 25000BaseTwinax/Full
Auto Negotiation: true
Backing DPUId: N/A
Cable Type: FIBRE
Current Message Level: -1
Driver Info:
Bus Info: 0000:8a:00:0
Driver: nmlx5_core
Firmware Version: 16.34.1002
Version: 4.23.0.36
Link Detected: true
Link Status: Up
Name: vmnic2
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: true
Pause TX: true
Supported Ports: FIBRE, DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver: internal
Virtual Address: **:**:**:**:**:**
Wakeon: None
