After a recent comment from @robertquast, it occurred to me that there’s quite a bit of confusion about the way that UCS Fabric Interconnects handle layer-2 forwarding in the default and recommended Ethernet Host Virtualizer mode.<\/p>\n
The basic idea behind EHV is that the Fabric Interconnect appears to the upstream network as if it were a host and not a layer-2 switch. \u00a0To accomplish this (and not require our old friend Spanning Tree Protocol), the Fabric Interconnect employs a number of L2 forwarding rules and deja vu checks to prevent the creation of L2 loops. \u00a0If you know how VMware vSwitches operate with multiple active uplinks, you’ve already got a pretty good understanding of EHV.<\/p>\n
EHV separates L2 links into two classes (at least in UCS) – border ports and server ports. \u00a0Border ports connect to outside L2 switching not under the control of UCS, while server ports connect to the chassis in the system (and in turn the blades).<\/p>\n
Due to the way that UCS provides connectivity to the blades, through the use of vNIC objects, the switching infrastructure already understands every “physical” NIC that exists “below” the Fabric Interconnect. \u00a0Therefore, it has no need to learn those addresses (or expire them), since it can track the physical connectivity and knows when a MAC address is or is not available. \u00a0 The exception to this is virtual MAC addresses that are not managed by the vNIC objects – specifically, those created or managed by a virtualization or clustering solution (i.e. VMware). \u00a0 These addresses are learned and then aged out by traditional L2 switch mechanisms, through a configurable timeout. \u00a0 See my other post regarding MAC address aging for more details.<\/p>\n
Uplinks are handled very differently. \u00a0 Most notably, in EHV mode, the Fabric Interconnects do not ever learn MAC addresses that exist on the uplink switches. \u00a0 Each physical blade port is pinned to a physical uplink port (or Port Channel). Unknown unicast MAC addresses received on border ports are dropped. \u00a0 Known unicast MAC addresses received on border ports are first subjected to a deja vu check to ensure that the frame arrives on the destination MAC address’ pinned uplink, and assuming a successful check, the frame is forwarded to the destination port.<\/p>\n
The problem arises when a silent VM’s MAC address is aged out of the MAC table in the Fabric Interconnect. \u00a0 If an outside L2 device generates a frame destined for the now-unknown unicast MAC, and the Fabric Interconnect receives the frame on a border port, the Fabric Interconnect will drop the frame. \u00a0 This essentially creates a black hole for that MAC address unless the VM generates some traffic *or* a device inside the Fabric Interconnect L2 domain generates a frame destined for the silent VM’s MAC address. \u00a0In this case, the Fabric Interconnect *will* broadcast the unknown unicast MAC frame to all server ports (in the same VLAN, of course) and that particular blade’s pinned uplink. \u00a0The VM should respond to the frame at which point the Fabric Interconnect will re-learn the address and traffic will flow normally.<\/p>\n
So the basic flow…<\/p>\n
Frames arriving on border ports:<\/p>\n
To known unicast -> forwarded to appropriate server port<\/p>\n
To unknown unicast -> dropped<\/p>\n
To broadcast -> forwarded to all server ports in VLAN<\/p>\n
Frames arriving on server ports:<\/p>\n
To known unicast -> forwarded to appropriate server port (remember we only learn addresses “south” of FI)<\/p>\n
To unknown unicast -> forwarded \u00a0only to pinned uplink port<\/p>\n
To broadcast -> forwarded to all server ports in VLAN and only pinned uplink port (same as unknown unicast)<\/p>\n
Questions? \u00a0Discussion?<\/p>\n
***NOTE*** Edited on June 7th, 2010 to correct an error in the description of behavior of unknown unicast messages received on server ports.<\/p>\n","protected":false},"excerpt":{"rendered":"
After a recent comment from @robertquast, it occurred to me that there’s quite a bit of confusion about the way that UCS Fabric Interconnects handle layer-2 forwarding in the default and recommended Ethernet Host Virtualizer mode. The basic idea behind EHV is that the Fabric Interconnect appears to the upstream network as if it were … Continue reading L2 Forwarding Rules in UCS EHV Mode<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[20,33],"class_list":["post-83","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-ehv","tag-l2"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts\/83","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=83"}],"version-history":[{"count":0,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts\/83\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=83"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=83"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=83"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}