Since my previous announcements about various books on UCS and related topics have gotten rolled off the main page, I thought it would be useful to collect them into a bookstore. I’ve added a link (see the navigation tabs at the top of the screen) to my UCS Bookstore. Feel free to have a look, and make any suggestions on books you think should be included.
The misunderstanding of Cisco’s enhanced network products for VMware environments has hit critical mass. At this point very few people know what does what, how, and when to use it. Hopefully this will demystify some of it.
Product name for a family of products, does not specifically refer to any one product so forget the idea of hardware vs. software implementation, etc. Think of the Nexus family of switches: 1000v, 2000, 4000, 5000, 7000. All different products solving different design goals but are components of the Data Center 3.0 portfolio. The separate products that fall under VN-Link are described below:
The Nexus 1000v is a Cisco software switch for VMware environments. It is comprised of two components: a Virtual Supervisor Module (VSM) which acts as the control plane, and a Virtual Ethernet Module (VEM) which acts as a data plane. 2 VSM modules operate in an active/standby fashion for HA and each VMware host gets a VEM. This switch is managed by a Cisco NXOS CLI and looks/smells/feels like a physical switch from a management perspective…that’s the whole point:
‘Network teams, here’s your network back, thanks for letting us borrow it.’ – The Server Team
The Nexus 1000v does not rely on any mystical magic such as VN-Tag (discussed shortly) to write frames. Standard Ethernet rules apply and MAC based forwarding stays the same. The software switch itself is proprietary (just like any hardware/software you buy from anyone) but the protocol used is standards based Ethernet.
Hypervisor Bypass/Direct path I/O:
Hypervisor bypass is the ability for a VM to access PCIe adapter hardware directly in order to reduce the overhead on a physical VMware’s hosts CPU. This functionality can be done with most any PCIe device using VMware’s Direct-Path I/O. The advantage here is less host CPU/memory overhead for I/O virtualization. The disadvantage is currently no support for vMotion and limits as to the number of Direct-Path I/O devices per host. This doesn’t require Cisco hardware or software to do, but Cisco does have a device that makes this more appealing in blade servers with limited PCIe devices (the VIC discussed later.)
Pass Through Switching (PTS):
PTS is a capability of the Cisco UCS blade system. It relies on management intelligence in the UCS Manager and switching intelligence on each host to pull management of the virtual network into the UCS manager. This allows a single point of management for the entire access layer including the virtual switching environment, hooray less management overhead more doing something that matters!
PTS directly maps a Virtual Machines virtual NIC to an individual physical NIC port across a virtualized pass-through switch. No internal switching is done in the VMware environment, instead switching and policy enforcement are handled by the upstream Fabric Interconnect. What makes this usable is the flexibility on number of interfaces provided by the VIC, discussed next.
Virtual Interface Card (VIC) the card formerly known as Palo:
The virtual interface card is an DCB and FCoE capable I/O card that is able to virtualize the PCIe bus to create multiple interfaces and present them to the operating system. Theoretically the card can create a mix of 128 virtual Ethernet and Fibre Channel interfaces, but the real usable number is 58. Don’t get upset about the numbers your operating system can’t even support 58 PCIe devices today ;-). Each virtual interface is known as a VIF and is presented to the operating system (any OS) as an individual PCIe device. The operating system can then do anything it chooses and is capable of with the interfaces. In the example of VMware the VMware OS (yes there is an actual OS installed there on the bare metal underneath the VMs) can then assign those virtual interfaces (VIF) to vSwitches, VM kernel ports, or Service Console ports, as it could with any other physical NIC. It can also assign them to the 1000v, to be used for Direct-Path I/O, or to use with Pass-Through Switching. Even more important is the flexibility to use separate VIFs for each of these purposes on the same host (read: none of these is mutually exclusive.) The VIC relies on VN-Tag for identification of individual VIFs, this is the only technology discussed in this post that uses VN-tag (although there are others.)
VN-Tag is a frame tagging method that Cisco has proposed to the IEEE and is used in several Cisco hardware products. VN-Tag serves two major purposes:
1) It provides individual identification for virtual interfaces (VIF.)
2) It allows a VN-Tag capable Ethernet switch to switch and forward frames for several VIFs sharing a set of uplinks. For example if VIF 1 and 2 are both using port 1 as an uplink to a VN-Tag capable switch device the VN-Tag allows the switch to forward the frame back down the same link because the destination VIF is different than the source VIF.
VN-Tag has been successfully used in production environments for over a year. If you’re using a Nexus 2000, you’re already using VN-Tag. VN-Tag is used by the: Nexus 2000 Series Switches, the UCS I/O Module (IOM), and the Cisco Virtual Interface Card (VIC.) The switching for these devices is handled by one of the two VN-Tag capable switches: Nexus 5000 or UCS 6100 Fabric interconnect. Currently all implementations of VN-Tag use hardware to write the tags.
– Joe Onisick (http://www.definethecloud.net)
Update 2011/01/31 – I’ve added new thoughts and comments on the subject here: http://www.unifiedcomputingblog.com/?p=234
Which is better? Which is faster?
I’ve been stuck on this one for a while. I’m traditionally a pure fibre channel kind of guy, so I’ve been pretty convinced that traditional FC was here to stay for a while, and that FCoE – as much as I believe in the technology – would probably be limited to the access and aggregation layers for the near term. That is, until it was pointed out to me the encoding mechanisms used by these two technologies and the effective data rates they allowed. I’m not sure why it never occurred to me before, but it hit me like the proverbial ton of bricks this week.
First off, a quick review of encoding mechanisms. Any time we’re transmitting or storing data, we encode it in some form or another. Generally, this is to include some type of a checksum to ensure that we can detect errors in reading or receiving the data. I remember the good old days when I discovered that RLL hard drive encoding was 50% more efficient than MFM encoding, and with just a new controller and a low level format, my old 10MB (yes, that’s ten whopping megabytes, kids. Ask your parents what a megabyte was.) suddenly became 15MB! Well, we’re about to embark on a similar discovery.
1, 2, 4, and 8 Gb Fibre Channel all use 8b/10b encoding. Meaning, 8 bits of data gets encoded into 10 bits of transmitted information – the two bits are used for data integrity. Well, if the link is 8Gb, how much do we actually get to use for data – given that 2 out of every 10 bits aren’t “user” data? FC link speeds are somewhat of an anomaly, given that they’re actually faster than the stated link speed would suggest. Original 1Gb FC is actually 1.0625Gb/s, and each generation has kept this standard and multiplied it. 8Gb FC would be 8×1.0625, or actual bandwidth of 8.5Gb/s. 8.5*.80 = 6.8. 6.8Gb of usable bandwidth on an 8Gb FC link.
10GE (and 10G FC, for that matter) uses 64b/66b encoding. For every 64 bits of data, only 2 bits are used for integrity checks. While theoretically this lowers the overall protection of the data, and increases the amount of data discarded in case of failure, that actual number of data units that are discarded due to failing serialization/deserialization is minuscule. For a 10Gb link using 64b/66b encoding, that leaves 96.96% of the bandwidth for user data, or 9.7Gb/s.
So 8Gb FC = 6.8Gb usable, while 10Gb Ethernet = 9.7Gb usable. Even if I was able to use all of the bandwidth available on an 8Gb FC port (which is very unlikely at the server access layer), with 10GE running FCoE, I’d still have room for 3 gigabit Ethernet-class “lanes”. How’s that for consolidation?
10Gb FC has the same usable bandwidth, and without the overhead (albeit a small 2% or so) of FCoE, but you don’t get the consolidation benefits of using the same physical link for your storage and traditional Ethernet traffic.
M. Sean McGee posted this great comparison of VirtualConnect and UCS. I’ve often struggled to give students a clear picture of the differences – HP will tell you that “VirtualConnect is just as good, and we’ve been doing it for years!”. Well, yes… it does some things similarly, and you can’t argue the timeframe. UCS does a lot more – and until now, I didn’t have a great source that directly compared them. From now on, all I have to do is send them to M. Sean McGee’s post!