{"id":3136,"date":"2024-02-08T15:23:03","date_gmt":"2024-02-08T07:23:03","guid":{"rendered":"https:\/\/www.yaoge123.com\/blog\/?p=3136"},"modified":"2024-05-23T07:07:07","modified_gmt":"2024-05-22T23:07:07","slug":"vmware-esxi-%e9%85%8d%e7%bd%ae-nvme-over-roce","status":"publish","type":"post","link":"https:\/\/www.yaoge123.com\/blog\/archives\/3136","title":{"rendered":"VMware ESXi \u914d\u7f6e NVMe over RoCE"},"content":{"rendered":"\n\n\n<p>NVMe over RoCE \u7684\u76f8\u5173\u914d\u7f6e\u9700\u8981\u5728 vCenter Server Client \u4e0b\u9762\u64cd\u4f5c\uff0cESXi Host Client \u4e0b\u662f\u6ca1\u6709\u76f8\u5173\u9009\u9879\u7684\uff0c\u81f3\u5c11\u56fe\u5f62\u754c\u9762\u662f\u8fd9\u6837\uff0c\u7528 esxicli \u4f30\u8ba1\u53ef\u4ee5\u76f4\u63a5\u5728 ESXi \u4e0b\u9762\u914d\u7f6e\u3002\u6240\u4ee5\u9700\u8981\u5148\u7528\u672c\u5730\u7684\u76d8\u90e8\u7f72\u4e00\u5957VCSA\uff0c\u7cbe\u7b80\u914d\u7f6e\u5927\u7ea6150GB\u5c31\u591f\u4e86\uff0c\u7b49NVMe\u914d\u7f6e\u597d\u518d\u8fc1\u79fb\u5230\u5b58\u50a8\u4e0a\u3002<\/p>\n<ul>\n<li>\u5728\u914d\u7f6e-\u7f51\u7edc-RDMA \u9002\u914d\u5668\u4e0b\u786e\u8ba4RDMA\u9002\u914d\u5668\uff08vmrdmaX\uff09\u5bf9\u5e94\u7684\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668\uff08vmnicX\uff09\uff0c\u53ef\u901a\u8fc7\u4ea4\u6362\u673a shutdown \u7b49\u65b9\u6cd5\u786e\u8ba4\u89c4\u5212\u7528\u4e8e NVMe \u7684\u7aef\u53e3\u5bf9\u5e94\u7684\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668\u540d\u79f0\u3002<br \/><em>\u5982 vmrdma0 \u5bf9\u5e94 vmnic2\u3001vmrdma3 \u5bf9\u5e94 vmnic5\u3002<\/em><\/li>\n<li>\u5728\u914d\u7f6e-\u7f51\u7edc-\u865a\u62df\u4ea4\u6362\u673a\u4e0b\u6dfb\u52a0\u7f51\u7edc\uff0c\u9009\u62e9 VMkernel \u7f51\u7edc\u9002\u914d\u5668\u3001\u9009\u62e9\u65b0\u5efa\u6807\u51c6\u4ea4\u6362\u673a\u3001\u8f93\u5165MTU\uff08Dorado \u63a8\u83505500\uff09\uff0c\u9009\u62e9\u4e00\u4e2a\u7528\u4e8eNVMe\u7684\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668\uff0cVLAN ID \u586b\u5199\u5bf9\u5e94\u4ea4\u6362\u673a\u7684 NVMe over RoCE \u7528 VLAN\uff0c\u53ef\u7528\u670d\u52a1\u4e2d\u4ec5\u52fe\u9009 NVMe over RDMA\uff0c\u914d\u7f6e\u9759\u6001IPv4\u5730\u5740\u3002<\/li>\n<li>\u91cd\u590d\u4e0a\u8ff0\u8fc7\u7a0b\uff0c\u4e3a\u6bcf\u4e2a NVMe \u7aef\u53e3\u5bf9\u5e94\u7684\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668\u521b\u5efa\u72ec\u7acb\u7684\u865a\u62df\u4ea4\u6362\u673a\u5e76\u914d\u7f6e\u72ec\u7acb\u7684 VLAN\u3001\u72ec\u7acb\u7684 VMkernel \u7aef\u53e3\uff0c\u5b9e\u73b0\u6bcf\u4e2a VMKernel \u548c\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668\u4e4b\u95f4\u4e00\u4e00\u6620\u5c04\u3002<br \/><em>\u5982 vSwitch1 \u7ed1\u5b9a\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668 vmnic2\u3001VMkernel \u7aef\u53e3 vmk1 192.168.123.11\u3001VLAN ID\u00a0 123\uff1bvSwitch2 \u7ed1\u5b9a\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668 vmnic5\u3001VMkernel \u7aef\u53e3 vmk2 192.168.124.11\u3001VLAN ID\u00a0 124\u3002<\/em><\/li>\n<li>\u5728\u914d\u7f6e-\u5b58\u50a8-\u5b58\u50a8\u9002\u914d\u5668-\u6dfb\u52a0\u8f6f\u4ef6\u9002\u914d\u5668\uff0c\u6dfb\u52a0 NVMe over RDMA \u9002\u914d\u5668\uff0c\u9009\u62e9\u5bf9\u5e94\u7684RDMA\u8bbe\u5907\u3002\u91cd\u590d\u8be5\u8fc7\u7a0b\u4e3a\u6bcf\u4e2a NVMe \u7684 RDMA\u9002\u914d\u5668 \u6dfb\u52a0 \u8f6f\u4ef6\u9002\u914d\u5668\u3002<br \/><em>\u5982\u4e3a vmrdma0 \u6dfb\u52a0 vmhba64\uff0c\u4e3a vmrdma3 \u6dfb\u52a0 vmhba65<\/em><\/li>\n<li>\u5728\u914d\u7f6e-\u5b58\u50a8-\u5b58\u50a8\u9002\u914d\u5668\u4e2d\uff0c\u9009\u4e2d\u521a\u521a\u6dfb\u52a0\u7684vmhbaXX\uff08VMware NVME over RDMA Storage Adapter\uff09\uff0c\u5728\u63a7\u5236\u5668\u4e2d\u6dfb\u52a0\u63a7\u5236\u5668\uff0c\u586b\u5199\u5b50\u7cfb\u7edfNQN\uff08\u5b58\u50a8\u7684NQN\uff09\u3001\u8be5\u7f51\u7edc\u5bf9\u5e94VLAN\u4e0b\u7684\u5b58\u50a8IP\uff0c\u4fdd\u6301\u6d3b\u52a8\u65f6\u95f4 Dorado \u63a8\u83505\u79d2\uff0c\u91cd\u590d\u6b64\u8fc7\u7a0b\u6dfb\u52a0\u5b58\u50a8\u5728\u8be5VLAN\u4e0b\u7684\u6240\u6709IP\u3002<br \/><em>\u5982 vmhba64 \u6dfb\u52a0\u6709 192.168.123.101-104 \u56db\u4e2a\u63a7\u5236\u5668<\/em><\/li>\n<li>\u91cd\u590d\u4e0a\u8ff0\u8fc7\u7a0b\u7ed9\u5269\u4e0b\u7684 vmhbaXX \u6dfb\u52a0\u63a7\u5236\u5668\u3002<br \/>\u5982 <em> vmhba65 \u6dfb\u52a0\u6709 192.168.124.101-104 \u56db\u4e2a\u63a7\u5236\u5668<\/em><\/li>\n<li>\u590d\u67e5\u6240\u6709\u8bbf\u95ee\u8def\u5f84 esxcli nvme fabrics connection list<\/li>\n<li>\u4e0a\u8ff0\u7684\u6bcf\u4e2a\u63a7\u5236\u5668\u5982\u679c\u90fd\u53ef\u4ee5\u8bbf\u95ee\u540c\u4e00\u4e2aLUN\uff0c\u90a3\u4e48\u6bcf\u4e00\u4e2a\u90fd\u662f\u4e00\u4e2a\u591a\u8def\u5f84\u30027.0\u6700\u591a4\u6761\uff0c8.0\u6700\u591a8\u6761\uff0c8.0U1\u6700\u591a32\u6761\u3002<\/li>\n<li>\u6700\u7ec8\u7ed3\u679c\u5982\u4e0b\u8868\u6240\u793a\uff0c8\u4e2a\u8def\u5f84 = 2\u4e2a\u7269\u7406\u7f51\u7edc\u9002\u914d\u5668 x 4\u4e2a\u63a7\u5236\u5668<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>RDMA\u9002\u914d\u5668<\/td><td>VMkernel\u9002\u914d\u5668<\/td><td>VMkernel \u9002\u914d\u5668 IP<\/td><td>\u7269\u7406\u9002\u914d\u5668<\/td><td>\u865a\u62df\u4ea4\u6362\u673a<\/td><td>VLAN<\/td><td>\u5b58\u50a8\u9002\u914d\u5668<\/td><td>\u5b58\u50a8\u63a7\u5236\u5668 IP<\/td><\/tr><tr><td>vmrdma0<\/td><td>vmk1<\/td><td>192.168.123.11<\/td><td>vmnic2<\/td><td>vSwitch1<\/td><td>123<\/td><td>vmhba64<\/td><td>192.168.123.101-104<\/td><\/tr><tr><td>vmrdma3<\/td><td>vmk2<\/td><td>192.168.124.11<\/td><td>vmnic5<\/td><td>vSwitch2<\/td><td>124<\/td><td>vmhba65<\/td><td>192.168.124.101-104<\/td><\/tr><\/tbody><\/table><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>NVMe over RoCE \u7684\u76f8\u5173\u914d\u7f6e\u9700\u8981\u5728 vCenter Server Client \u4e0b\u9762\u64cd\u4f5c\uff0cESXi [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[8],"tags":[],"class_list":["post-3136","post","type-post","status-publish","format-standard","hentry","category-network"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/paOwEq-OA","_links":{"self":[{"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/posts\/3136","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/comments?post=3136"}],"version-history":[{"count":4,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/posts\/3136\/revisions"}],"predecessor-version":[{"id":3196,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/posts\/3136\/revisions\/3196"}],"wp:attachment":[{"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/media?parent=3136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/categories?post=3136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yaoge123.com\/blog\/wp-json\/wp\/v2\/tags?post=3136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}