Tag: UCS

Brocade’s Flawed FCoE “Study”

Disclosures
I do not work for Cisco, Brocade, or any of the companies mentioned here. I do work for a reseller that sells some of these products, but this post (as are all posts on this site) is my opinion only, and does not necessarily reflect the views of my employer or any of the manufacturers listed here. Evaluator Group Inc. did invite me to have a call with them to discuss the study via a tweet sent from the @evaluator_group Twitter account. Mr. Fellows also emailed me to offer a call. After my analysis, I deemed such a call unnecessary.

Ok, with that out of the way…

The “Study”

There were quite a few incredulous tweets floating around this week after Brocade publicized an “independent” study performed by Russ Fellows of Evaluator Group Inc. It was also reviewed by Chris Mellor of The Register, which is how I came to know about it. In the review, Mr. Mellor states that “Brocade’s money was well spent,” though I beg to differ.

As of this posting, the study is still available from the Evaluator Group Inc. website, though I would hope that after some measure of peer review, it will be removed given how deeply flawed it is. As I do not have permission to redistribute the study, I will instead suggest that you get a copy at the above link and follow along.

The stated purpose of the study was to compare traditional Fibre Channel (hereafter FC) against Fibre Channel over Ethernet (hereafter FCoE), specifically as a SCSI transport between blade servers and solid state storage. To reduce equipment requirements, only a single path was designed into the test, unlike a production environment that would have at a minimum two. The report further stated that an attempt would be made to keep the amount of bandwidth available to each scenario equal.

The Tech

The vendor of storage was not disclosed, though it should be fairly irrelevant (with one exception to be noted below). The storage was connected via two 16Gb FC links to a Brocade 6510 switch. The Brocade 6510 is a “top of rack” style traditional FC switch that is not capable of FCoE.

The chosen architecture for the FC test was an HP c7000 blade enclosure containing two blades, using a Brocade FC switch. The embedded Brocade switch is connected to the Brocade 6510 via a single 16Gb FC link.

The FCoE test was performed using a Cisco UCS architecture, consisting of a single Fabric Interconnect, connected via 4 10Gb converged Ethernet links to a single blade chassis containing two blades. The Fabric Interconnect is connected to the Brocade 6510 via two 8Gb FC links. As of this writing, the only FC connectivity supported by Cisco UCS is 10Gb FCoE or 1/2/4/8Gb FC.

So what’s the problem?

There are many, many fundamental flaws with the study. I eventually ran out of patience to catalog them individually, so I’m instead going to call out some of the most egregious transgressions.

To start, let’s consider testing methodology. The stated purpose of this test was to evaluate storage connectivity options, narrowed to FC and FCoE. It was not presented as a comparison of server vendors. As such, as many variables as possible should be eliminated to isolate the effects of the protocol and transport. This is the first place that this study breaks down. Why was Cisco UCS chosen? If the effects of protocol and transport are truly the goal of the test, why would the HP c7000 not also be the best choice? There are several ways to achieve FCoE in a c7000, both externally and internally.

The storage in use is connected via two 16Gb FC links. The stated reason for this is that the majority of storage deployments still use FC instead of FCoE, which is certainly true. The selection of the Brocade 6510 is interesting, however, in that Brocade has other switches that would have been capable of supporting FCoE and FC simultaneously. It’s clear that the choice of an FC only switch was designed to force the FCoE traffic to be de-encapsulated before going to the storage. Already we can see that we are not testing FC vs. FCoE, but rather FC natively end to end vs. one hop of FCoE. Even so, the latency and performance impact caused by the encapsulation of the FC protocol into Ethernet is negligible. The storage vendor was not disclosed, and as such, I do not know if it could have also supported FCoE, making for a true end-to-end FCoE test.  Despite the study’s claim, end-to-end FCoE is not immature and has been successfully deployed by many customers.

In UCS architecture, all traffic is converged between the blade chassis and the Fabric Interconnect. All switching, management, configuration, etc, occurs within the Fabric Interconnect. The use of four 10Gb Ethernet links between the chassis and Fabric Interconnect is significant overkill given the stated goal of maintaining similar bandwidth between the tests. At worst, two links would have been required to provide each blade with a dedicated 10Gb of bandwidth. Presumably, the decision to go with four was so that the claim could be made that more bandwidth was made available per blade than was available to the 16Gb-capable blades in the HP solution. The study did not disclose the logical configuration of the UCS blades, but the performance data suggests a configuration of a single vHBA per blade. In this configuration, the vHBA would follow a single 10Gb path from the blade to the Fabric Interconnect (via the IO Module), and would in turn be pinned to a single 8Gb FC uplink. Already you can see that regardless of the number of links provided from chassis to Fabric Interconnect, the bottleneck will be the 8Gb FC uplink. The second blade’s vHBA would be pinned (automatically, mind you) to the second 8Gb FC uplink. Essentially in this configuration, each blade has 8Gb of FC bandwith to the Brocade 6510 switch. The VIC 1240 converged network adapter (CNA) on the blade is capable of 20Gb of bandwidth to each fabric. The creation of a second vHBA and allowing the operating system to load balance across them would have provided more bandwidth. The study mentions the use of a software FCoE initiator as being part of the reason for increased CPU utilization.

We didn’t understand the technology, but…

In the “ease of use” comparison, it was noted that the HP environment was configured in three hours, whereas it took eight hours to configure UCS. The study makes it clear that they did not have the requisite skill to configure UCS and required the support of an outside VAR (who was not named) to complete the configuration. The study also states that the HP was configured without assistance. Clearly the engineering team involved here was skilled in HP and not UCS. How this reflects poorly on the product (and especially FC vs. FCoE – that’s the point, right?) is beyond me. I can personally (and have) configure a UCS environment like this in well under an hour. It would probably take me eight hours to perform similar configuration on an HP system, given my lack of hands-on experience in configuring them. This is not a flaw of the HP product, and I wouldn’t penalize it as such. (There are lots of reasons I like UCS over HP c7000, but that’s significantly beyond the scope of this post)

Many of the “ease of use” characteristics cited reflected an all Brocade environment – similar efficiencies would have existed in an all Cisco environment as well, which the study neglected to test.

A software what?

The study observes a spike in CPU utilization with increased link utilization, which is (incorrectly) attributed to the use of a software FCoE initiator. This one point threw me (and others) off quite a bit, as it is extremely rare to use a software FCoE initiator, and non-existent when FCoE capable hardware is present (such as the VIC 1240 in use here). After a number of confusing tweets from the @evaluator_group twitter account, it became clear that while they say they were using a software initiator, it was a misunderstanding of the Cisco VIC 1240 – again pointing to a lack of skill and experience with the product. My suspicion is that the spike in CPU utilization (and latency, and corresponding increase in response times) occurred not due to the FCoE protocol, but rather the queuing that was required when the two 8Gb FC links (total of 13.6Gb/s total bandwidth available, though not aggregated – each vHBA will be pinned to one uplink) became saturated. This is entirely consistent with observed application/storage performance when the links are saturated. This is entirely speculation, however, as the logical configuration of the UCS was not provided.  Despite there being similar total bandwidth available, neither server would have been able to burst above 6.8Gb/s, leading to queuing (and the accompanying latency/response impact).

Is that all?

I could go on and on with individual points that were wrong, misleading, or poorly designed, but I don’t actually think it’s necessary. Once the real purpose of the test (Brocade vs. Cisco) became clear, every conclusion reached in the FC vs. FCoE discussion (however incorrect) is moot.

If Brocade really wants to fund an FC vs. FCoE study that will stand up to scrutiny, it needs to use the same servers (no details were provided on specific CPUs in use – they could have been wildly different for all we know), the same chassis, and really isolate the protocol as they claimed to do. Here’s the really sad part – Brocade could have proven what they wanted to (that 16Gb FC is faster than 10Gb FCoE) in a fair fight. Take the same HP chassis used for the FC test, and put in an FCoE module (with CNAs on the servers) instead. Connect via FCoE to a Brocade FCoE capable switch, and use FCoE capable storage. Despite the test’s claim, there’s a lot of FCoE storage out there in production – just ask NetApp and EMC. At comparable cable counts, 16Gb FC will be faster than 10Gb FCoE. What a shock, huh? Instead, this extraordinarily flawed “study” has cost Brocade and unfortunately Evaluator Group Inc. a lot of credibility.

I’m not anti-Brocade (though I do prefer MDS for FC switching, which is not news to anyone who knows me), I’m not anti-FC (I still like it a lot, though I think pure FC networks’ days are numbered), I’m just really, really anti-FUD. Compete on tech, compete on features, compete on value, compete on price, compete on whatever it is that makes you different. Just don’t do it in a misleading, dishonest way. Respect your customers enough to know they’ll see through blatant misrepresentations, and respect your products enough to let them compete fairly.
—
Updated: Check out Tony Bourke’s great response here.

Why doesn’t Cisco…?

I get asked a lot why Cisco doesn’t have feature X, or support hardware Y in their UCS product line.   A recent discussion with a coworker reminded me that lots of those questions are out there, so I might as well give my opinion on them.

Disclaimer : I don’t work for Cisco, I don’t speak for Cisco, these are just my random musings about the various questions I hear.

Why doesn’t Cisco have non-Intel blades, like AMD or RISC-type architectures?  Are they going to in the future?

As of today, Intel processors (the Xeon 5500/5600, 6500/7500 families) represent the core (pun intended) of the x86 processor market.  Sure, even Intel has other lines (Atom, for one), and AMD still makes competitive processors, but most benchmarks and analysts (except for those employed by other vendors) agree that Intel is the current king.   AMD has leapfrogged Intel in the past, and may do so again in the future, but for right now – Intel is where it’s at.

If you look at this from a cost-to-engineer perspective, it starts to make sense.   It will cost Cisco just as much to develop an AMD-based blade as it does for the more popular and common Intel processors.   Cisco may be losing business to customers that prefer AMD, but until they’ve run out of customers on the Intel side of things, it just doesn’t make financial sense to attack the AMD space as well.

As for RISC/Unix type architectures (really, any non-x86 platform), who’s chip would they use?  HP?  Not likely.  IBM?  Again, why support a competitor’s architecture – especially one as proprietary as IBM.  (Side note – I’m a really big fan of IBM AIX systems, just not in the “blade” market)   Roll their own?  Why bother?  It’s still a question of return on investment.   Even if Cisco could convince customers to abandon their existing proprietary architectures for a Cisco proprietary processor, how much business do you really think they’d do?   Nowhere near enough to justify the development cost.

Why doesn’t Cisco have Infiniband adapters for their blades?  What about the rack-mount servers?

One of the key concepts in UCS is the unified fabric, using only Ethernet as the chassis-to-Fabric Interconnect topology.  By eliminating protocol-specific cabling (Fibre Channel, Infiniband, etc), the overall complexity of the environment is reduced and the bandwidth is flexibly allocated between upper (above Ethernet) layer protocols.   Instead of having separate cabling and modules for different protocols (a la legacy blade architectures), any protocol needed is encapsulated over Ethernet.   Fibre Channel over Ethernet (FCoE) is the first such implemenatation in UCS, but certainly won’t be the last.

Infiniband as a protocol has a number of compelling features for certain applications, so I’d definitely see Cisco supporting RDMA over Converged Ethernet (RoCE) in the future.  RoCE does for Infiniband what FCoE does for Fibre Channel.  The underlying transport is replaced with Ethernet, while keeping the protocol intact.  Proponents of Infiniband will point to the transport’s legendary latency characteristics, specifically low and predictable.   The UCS unified fabric architecture provides just such an environment – low, predictable latency that’s consistent in both inter- and intra-chassis applications.

As for the rack-mount servers, there’s nothing stopping customers from purchasing and installing their own PCI Infiniband adapters.   Cisco isn’t producing one, and won’t directly support it – but rather treats it as a 3rd party device to be supported by that manufacturer.

What about embedded hypervisors?

Another key feature of UCS is that the blades themselves are stateless, at least in theory.  No identity (MACs, WWNs, UUIDs, etc), no personality (boot order, BIOS configuration) until one is assigned by the management architecture.    Were the blades to have an embedded hypervisor, that statelessness is lost.  Even though it’s potentially a very small amount of stateful data (IP address, etc), it’s still there.   This is probably the most-likely to be supported question in my list.  My expectation is that at some point in the future, the UCS Manager will be able to “push” an embedded hypervisor, along with its configuration, to the blade along with the service profile.   By making UCS Manager the true stateful owner of the configuration data, having a “working copy” on the blade becomes less of an issue.

Final thoughts…

I’ve used this analogy in the past, so I’ll repeat it here.   I look at UCS as sort of the Macintosh of the server world.   It’s a closely controlled set of hardware in order to provide the best possible user experience, at the cost of not supporting some edge-case configurations or feature sets.   No, you can’t have Infiniband, or GPUs on the blade, or embedded hypervisors.   The fact is that the majority of data center workloads don’t need these features.   If you need those features, there are plenty of vendors that provide them.  If you want a single vendor for all your servers – regardless of edge-case requirements – there are certainly vendors that provide that (HP, IBM, etc).   In my opinion, though, it’s that breadth of those product offering that makes those solutions less attractive.   In accommodating for every possible use case, you end up with a very complex architecture.   Cisco UCS is streamlined to provide the best possible experience for the bulk of data center workloads.   Cisco doesn’t need to be, or want to be as near as I can tell, an “everything to everybody” solution.  Pick something you can do really, really well and do it better than anyone else.   Let the “other guys” work on the edge cases.  Yes – that will cost Cisco some business.   Believe it or not, despite what the rhetoric on Twitter would have you believe, there’s enough business out there for all of these server vendors.   Cisco, even if they’re wildly successful in replacing legacy servers with UCS, isn’t going to run HP or IBM or Dell out of business.   They don’t need to.   They can make a lot of money, and make a lot of customers very happy, co-existing in the marketplace with these vendors.   Cisco provides yet another choice.   If it doesn’t meet your needs, don’t buy it.   🙂

No offense or disrespect is intended to my HP and IBM colleagues.   You guys make cool gear too, you’re just solving the problems in a different way.   Which way is “best”?  Well, now, that really comes down to the specific customer doesn’t it?

Great UCS write-up by Joe Onisick

If you’re not currently following Joe’s blog over at definethecloud.wordpress.com, you should start.

He just posted another great article on why UCS is his server platform of choice.   Before you write him off as just another Cisco fan-boy, definitely take a look at his logic.   Even if you have another vendor preference, he presents some excellent points to consider.

Take a look : http://definethecloud.wordpress.com/2010/05/23/why-cisco-ucs-is-my-a-game-server-architecture/

Tolly Report

A lot of people have been asking me what I think of the recently released Tolly report comparing the bandwidth of the HP and Cisco blade solutions.

The short answer is, I don’t think much of it.   It’s technically sound, and the the conclusions it reaches are perfectly reasonable – for the conditions of the tests they performed.   In keeping with Tolly’s charter, the tests were repeatable, documented, and indisputable.  The problem is, the results of the testing only tell half the story.   The *conclusions* they reach, on the other hand, aren’t as defensible.

It’s really not necessary to get into a point by point rebuttal.   At least not for me.   I’m sure Cisco will be along any minute to do just that.

The facts that Tolly established during the report aren’t groundbreaking or surprising.   Essentially, the tests were built to demonstrate the oversubscription of links between UCS chassis and Fabric Interconnects, which Cisco is quite willing to disclose at 2:1.  These tests were paired with HP comparisons of blade-to-blade traffic on the same VLAN, which in HP architectures keeps the traffic local to the chassis.  The interesting thing there is that if the traffic were between the same blades but in different VLANs, the Cisco and HP solutions would have performed identically (assuming the same aggregation-layer architecture).   What makes that interesting is that the Tolly report’s figures depend on a specific configuration and test scenario – the Cisco values won’t change (or get any worse) no matter how you change the variables.  The HP values will vary widely.

And that, my friends, is where I see the true benefit of Cisco’s architecture.   Change the conditions of the test repeatedly, and you’ll get the same results.

I’m not faulting Tolly in this case.  Not at all.  They were asked to test a certain set of conditions, and did so thoroughly and presumably accurately.  It’s just that you can’t take that set of data and use it to make any kind of useful comparison between the platforms.   The real world is much more complicated than a strictly controlled set of test objectives.   Do we really think that HP went to Tolly and asked “We want a fair fight?”   Of course not.

What’s the difference between VN-Link and VN-Tag?

This is a question that constantly comes up in the classes and discussions that I’m involved in.

Part of the problem is that Cisco’s own documentation tends towards the marketing and less towards the technology.  Because of that, even a lot of Cisco folks are confused on exactly what is meant by each term.  Hopefully, this short summary will help.

VN-Link is a marketing umbrella term used by Cisco to describe any number of approaches to providing physical network type visibility to non-physical or non-directly attached devices.  This could mean virtual machines, could mean virtual interfaces on a remote interface card (a la Cisco’s Virtual Interface Card – aka Palo), or could mean physical interfaces on a non-switching remote device such as the Nexus 2000-series devices.

Probably the easiest way to group these is by the existence of a “Virtual Ethernet Port” on a switching device that is controlled like a physical port, but doesn’t directly map to a local physical switch port.

There are two approaches currently in use that fall under the VN-Link umbrella.

The Cisco Nexus 1000V switch, which is a software-only Cisco-branded switch that rides on top of VMware’s vNetwork Distributed Switch (DVS), is considered VN-Link because it provides virtual machine level visibility and granularity in network configuration and control.  Each virtual machine receives a “virtual Ethernet port” on the 1000V, which can be configured and controlled just like a physical Ethernet port would on a standard switch.

The Nexus 5000/2000 combination and the Cisco UCS Fabric Interconnect/IO Module combination both use an additional header in Ethernet frames called VN-Tag, which uniquely identifies some remote port which will receive a virtual Ethernet port on the local switch (50xx or 61xx).  This causes the Nexus 2000 or UCS IO Module to act as a remote line card to the host device, and doesn’t have to be managed individually.   All switching happens in the host device (50xx or 61xx).  The same VN-Tag technology is used by the Cisco Virtual Interface Card (VIC, or Palo) to identify the virtual interfaces being supported by the card.  With this tag added into the Ethernet frame, the host device (50xx or 61xx) can uniquely identify the source port (virtual or physical) and apply policy or configuration to it.

Note that the Nexus 1000V does not perform VN-Tag’ing – it simply doesn’t need to in order to meet it’s objective of providing VM-level visibility and control.

So both of these approaches meet the same architectural goal, while doing so with very different technologies.  Even so, they both fall under the same VN-Link umbrella.  Don’t confuse VN-Link, the “goal”, with the implementation.

A real deployment’s experience with adding hardware to UCS

One of the things we talk about in Cisco UCS is how easy is it is to expand the architecture.   Each time you add a chassis in competing solutions, you have lots of management points to configure – the chassis itself, the Ethernet and potentially Fibre Channel switching/connectivity, etc.   With UCS, it’s simply a matter of “telling” the Fabric Interconnects that the chassis is there and the rest happens auto-magically.  At HealthITGuy’s Blog, Michael Heil posts his experience in adding a new chassis to a running UCS deployment.  The rest of his posts on UCS, from a customer’s perspective, are also excellent.

Excellent discussion and comparison of UCS and HP blade offerings

Over at By the Bell, there’s a great post and commentary on differences between HP’s offerings and Cisco UCS.   As many of the comments point out, the post itself is a bit biased towards UCS, but the comments are an excellent set of discussions about the various pros and cons of the solutions.  As good as the post is, the comments (I think) are even better.   Check it out!