The Problem With Vendor Sponsored Testing

I suppose this post has been a long time coming.

It was spurred into reality by an exchange with @bladeguy who pointed out that Cisco, too, sponsors tests of their equipment – just like HP and the Tolly reports.  At first, I’d intended to do a comparison of the Tolly reports and the Principled Technologies reports, looking for obvious (or not so obvious) bias.   Once I started down that path, however, I realized it really wasn’t necessary.   Sponsored tests (from any organization) will always be biased, and therefore unreliable from a technical perspective.   There are always tuning parameters that the “loser” will insist were wrong which skewed the results, there are always different ways to architect the test that would have given the “loser” an edge.   That’s why they’re sponsored tests.

I commented briefly before on Tolly’s first HP vs. Cisco UCS report, so I won’t repeat it again here.  Suffice it to say, the bloggers and such did a pretty good job of chopping it up.

My issue with the Tolly reports I’ve seen thus far, the bandwidth debacle and the airflow test, simply don’t pass the smell test.   Yes, they’re repeatable.   Yes, the testing results are defensible.   But the conclusions and declarations of a “winner”?  Not so much.   I’m not faulting Tolly here.   They’re doing their job, producing the product that their customer has asked them for.  These tests aren’t sponsored by the HP engineering groups (for whom I have a lot of respect) looking to validate their technological prowess – they’re sponsored by the marketing departments to provide ammunition for the sales process.   As such, do you really think they’re going into this looking for a fair fight?   Of course not.   They’re going to stack the deck in their favor as much as they think they can get away with (and knowing marketing departments, more than they can get away with).   That’s what marketing groups (from every vendor) do.

@bladeguy pointed out that Cisco has engaged Principled Technologies to do some testing of UCS equipment versus both legacy and current HP equipment.  At first glance, I didn’t detect any significant bias – especially in the tests comparing legacy equipment to current UCS gear.   I’m not sure how any bias could be construed, since really they’re just showing the advantage and consolidation ratios capable when moving from old gear to new gear.   Obviously you can’t compare against Cisco’s legacy servers (since there aren’t any), and HP servers are the logical choice since they have a huge server market share.   I would suspect that similar results would have been achieved when comparing legacy HP equipment against current HP equipment as well.   HP can (and I’m sure has) perform similar tests if they’d like to demonstrate that.

The more troublesome tests are those comparing current generations of equipment from two manufacturers.   The sponsor of the test will always win, or that report will never see the light of day.  That’s simply how it works.   Companies like Tolly, Principled Technologies, etc aren’t going to bite the hand that feeds them.   As such, they’re very careful to construct the tests such that the sponsor will prevail.   This is no secret in the industry.   It’s been discussed many times before.

Even the Principled Technologies tests that compared current generations of hardware looked like pretty fair fights to me.  If you look closely at the specifications of the tested systems, they really tend to reveal the benefits of more memory, or other such considerations, as opposed to the hardware itself.   @bladeguy pointed out several items in the Principled Technologies tests that, in his opinion, skewed the results towards Cisco.   I’m not in any position to refute his claims – but the items he mentioned really come down to tuning.   So essentially he’s saying that the HP equipment in the test wasn’t tuned properly, and I’m certainly not going to argue that point.   As a sponsored test, the sponsor will be victorious.

And therein lies the problem.   Sponsored tests are meaningless, from any vendor.   I simply don’t believe that sponsored tests provide value to the technical community.  But that’s ok – they’re not targeted at the technical community.   They’re marketing tools, used by sales and marketing teams to sway the opinions of management decision makers with lots of “independent” results.    If I want to know which server platform is better for my environment, I’m going to do my own research, and if necessary invite the vendors in for a bake-off.   Prove it to me, with your tuning as necessary, and I’ll have the other vendors do the same.

My real problem with these tests, understanding that they’re not aimed at the technical community, is the that many in the technical community use them to “prove” that their platform is superior to whoever their competing against at the moment.   Like politics, these kinds of arguments just make me tired.   Anyone coming into the argument already has their side picked – no amount of discussion is going to change their mind.   My point for blogging about UCS is not to sell it – I don’t sell gear.   It’s because I believe in the platform and enjoy educating about it.

I happen to prefer Cisco UCS, yes.   If you’ve ever been in one of my UCS classes, you’ll have also heard me say that HP and IBM – Cisco’s chief rivals in this space – also make excellent equipment with some very compelling technologies.   The eventual “best” solution simply comes down to what’s right for your organization.   I understand why these sponsored tests exist, but to me, they actually lessen the position of sponsor.   They make me wonder, “if your product is so good, why stack the deck in the test?”   The answer to that, of course, is that the engineers aren’t the ones requesting or sponsoring the test.

As came up in my UCS class today, for the vast majority of data center workloads, the small differences in performance that you might be able to get out of Vendor X or Vendor Y’s server is completely meaningless.   When I used to sell/install storage, I used to get asked the question as to which company’s storage to buy if the customer wanted maximum performance.  My answer, every single time was, “HP, or IBM, or HDS, or EMC, or…”   Why?  Because technology companies are always leapfrogging each other with IOPS and MB/s and any other metric you can think of.   What’s the fastest today in a particular set of circumstances will get replaced tomorrow by someone else.

So what’s the solution?  Well, true independent testing, of course.   How do you do true independent testing?   You get a mediator (Tolly, Principled Technologies, etc are fine), representatives from both manufacturers to agree on the testing criteria, and allow each manufacturer to submit their own architecture and tuning to meet the testing criteria.   The mediator then performs the tests.    The results are published with the opportunity for both manufacturers to respond to the results.   Think any marketing organization from any company would ever allow this to happen?   The standard line in the testing industry is “Vendor X was invited to participate, but declined.”  Of course they declined.   They’ve already lost before the first test was run.   I wouldn’t expect Cisco to participate in a HP-sponsored Tolly test any more than I’d expect HP to participate in a Cisco-sponsored Principled Technologies test.

Don’t chase that 1% performance delta.   You’ll just waste time and money.   Find the solution that meets your organizational needs, provides you the best management and support options, and buy it.   Let some other chump chase that magical “fastest” unicorn.   It doesn’t exist.  As in all things IT, “it depends.”

All comments are welcome, including those from testing organizations!

Private Isolated VSANs?

Ok, so this isn’t really UCS related.   Just a random thought I had today while working on a lab project… why don’t we have Private VSANs?   As in, the same type of technology as Private VLANs (PVLANs)?

First, some background.   Standard SAN best practice for access control is to use single-initiator/single-target zoning.   This means that there’s one zone for each combination of host and storage, tape, virtualization platform, etc port.    Some administrators think this is overkill, and create just a few zones of lots of initiators to single targets, but this is overall a bad idea.   The purpose of this post is not to argue for single-initiator zoning, since it’s accepted recommended practice.

Private VLANs provide a method for simplifying access control within a L2 Ethernet domain, restricting access between nodes.   Community PVLANs allow communication only between members of the same community, and the promiscuous port(s).   This is actually fairly close to the idea of a fibre channel zone, with the distinction that fibre channel doesn’t have promiscuous ports.   Isolated PVLANs allow communication only between each individual node and the promiscuous port(s).   In a way, you could compare this to having a lot of nodes (initiators) zoned only to a single target node (target) in fibre channel – but without the administrative overhead of zoning.

So, why not combine these approaches?   Having the concept of an Isolated Private VSAN would simplify some types of fibre channel deployments, by enforcing recommended practices around access control without the administrative overhead.  In a smaller environment, you could simply create an Isolated Private VSAN to contain the ports for a given fabric – set the storage ports as promiscuous, and all node ports would be restricted to connecting only to the storage ports – and prevented from communicating with each other.   In fact, I’d imagine that this would be enforced with standard FC zoning (since that’s the hosts are expecting when they query the name server anyway) – really we’d just be automating the creation of the zones.   Cisco already does something similar by automatically creating zones when doing Inter-VSAN Routing (IVR).

For slightly larger environments, I could even see adding in the idea of Community Private VSANs – whereby you group initiators and specify specific target (promiscuous) ports per community – without having to add additional VSANs.

Now that I’m thinking out-loud, why not have isolated zones instead?   Mark a zone as “isolated”, and tag any necessary WWNs/ports/etc as promiscuous, and enforce the traditional zoning behind the scenes.

True, this approach wouldn’t accomplish anything that traditional VSANs and zoning do not.  The implementation would likely have to use traditional zoning behind the scenes.   Just as PVLANs aren’t used in every situation, nor would PVSANs, but I could definitely see some use cases here.  So what do you think?   Am I completely insane?   Thoughts, comments, rebukes are all welcome.  🙂

UCSM 1.3(1c) Released!

Cisco has released UCS Manager version 1.3(1c).   This is the first public release in the 1.3 line, also known as “Aptos+”.

Release notes are here: http://www.cisco.com/en/US/docs/unified_computing/ucs/release/notes/ucs_22863.html

Haven’t gotten a chance to play with the new version yet, but there are some significant enhancements.    Among them…

  • 1 GE support on UCS6120 and UCS6140 Fabric Interconnects
    • On the 6120, you can now use 1GE transceivers in the first 8 physical ports.
    • On the 6140, you can now use 1GE transceivers in the first 16 physical ports.
    • Watch for a post soon on why I think this is a bad idea.  🙂
  • Support for the new, 2nd generation mezzanine cards
    • Both Emulex and Qlogic have produced a 2nd generation mezzanine card, using a single-chip design which should lower power consumption
      • Be warned that these new mezzanine cards won’t support the “Fabric Failover” feature as supported by the first generation CNAs, or by the VIC (Palo) adapter
      • These aren’t shipping quite yet, but will be soon
    • A Broadcom BCM57711 mezzanine adapter
      • This will compete with the Intel based, 10GE mezzanine adapters that UCS has had until now
      • The Broadcom card supports TOE (TCP Offload Engine) and iSCSI Offload, but not iSCSI boot
    • An updated Intel mezzanine adapter, based on the Niantic chipset
  • Support for the B440-M1 blade
    • The B440 blade will be available in a 2 or 4 processor configuration, using the Intel Xeon 7500 processors
    • Up to 4 SFFP hard drives
    • 32 DIMM slots, for up to 256GB of memory
    • 2 Mezzanine slots
    • Full-width form factor
  • SSD hard drive support in B200-M2, B250-M2, and B440-M1 blades
    • First drive available is a Samsung 100GB SSD
  • Improved SNMP support
  • Ability to configure more BIOS options, such as virtualization options, through the service profiles
    • This is a big step towards making UCS blades honestly and truly stateless
    • Previously, I’d recommended that UCS customers configure each blade’s BIOS options to support virtualization when they received them, whether or not they were going to use ESX/etc on all of the blades.  This way they didn’t have to worry about setting them again when moving service profiles
  • Support for heterogeneous mezzanine adapters in full-width blades
  • Increased the supported limit of chassis to 14.
  • Increased the limit of VLANs in UCSM to 512
    • There’s been some discussion around this lately, particular in the service provider space.   Many service providers need many more VLANs than this for their architectures.
    • I’ve seen reference to a workaround using ESX, Nexus 1000V, private VLANs, and a promiscuous VLAN through the Fabric Interconnect into an upstream switch, but I’m still trying to get my head around that one.  🙂
  • Ability to cap power levels per blade
    • Will have to wait until I get a chance to test out the code level to see what kinds of options are available here

Looking forward to seeing customer reaction to the new features.

Correction to L2 Forwarding Rules post

I posted here about the L2 forwarding rules when UCS is in EHV mode.   Several readers have pointed out a flaw in the logic I posted, which was taken from Cisco’s DCUCI course.   In Cisco’s defense, I did write that course.   🙂

At issue is how UCS deals with unknown unicast frames.   The other post incorrectly states than an unknown unicast frame received from a server port would be flooded out all other server ports participating in the same VLAN.   This is not the case.

The logic behind EHV mode is that it is impossible to have an unknown unicast address “behind” or “south” of the Fabric Interconnect.   All adapter MAC addresses are known, either because they were assigned in the service profile or inventoried (if using “derived” values).    For MAC addresses that are generated within a host, say for a virtual machine, the assumption is that at creation (or arrival through vMotion, etc) the MAC address will be announced using gratuitous ARP or other traffic generation techniques and the Fabric Interconnect can learn the address through normal L2 methods.

So to clarify, an unknown unicast frame received from a server port will be flooded out ONLY that interface’s pinned uplink.   Otherwise, all traffic destined for MAC addresses outside of UCS (such as the MAC address of a default gateway, for example) would also get flooded internally – which would not be a good thing.

Great UCS write-up by Joe Onisick

If you’re not currently following Joe’s blog over at definethecloud.wordpress.com, you should start.

He just posted another great article on why UCS is his server platform of choice.   Before you write him off as just another Cisco fan-boy, definitely take a look at his logic.   Even if you have another vendor preference, he presents some excellent points to consider.

Take a look : http://definethecloud.wordpress.com/2010/05/23/why-cisco-ucs-is-my-a-game-server-architecture/

UCS with disjointed L2 Domains

How do we deal with disjointed L2 domains in UCS?

To start, what’s a disjointed L2 domain?  This is where you have two Ethernet “clouds” that never connect, but must be accessed by the same UCS Fabric Interconnect.   Take, for example, a multi-tenant scenario where we have multiple customer’s servers within the same UCS cluster that must access different L2 domains.

How do we ensure that all traffic from Customer A’s blade only goes to their cloud, while Customer B’s blades only connect to their cloud?

The immediately obvious answer is to use UCS pin groups to tie each customers interfaces (through their vNIC configuration) to the uplinks that go to their cloud.   Unfortunately, this only solves half of the problem.

In the default operational mode of the Fabric Interconnects (called Ethernet Host Virtualizer, sometimes called End Host Virtualizer), only one uplink is used to receive multicast or broadcast traffic.   EHV mode assumes a single L2 fabric on the uplinks (VLAN considerations notwithstanding).  So in this example, only broadcasts or multicasts from one of the two fabrics would be accepted.   Obviously, this is a problem.

The only way to get around this is to put the Fabric Interconnects into Ethernet Switching mode.   This causes the Fabric Interconnect to behave as a standard L2 switch, including spanning tree considerations.  Now uplinks can receive broadcasts and multicasts regardless of the fabrics they are connected to.   This does, however, increase the administrative overhead of the Fabric Interconnects and reduces your flexibility in uplink configuration since now we must channel all ports going into the same L2 domain in order to use the bandwidth.

To me, a more ideal situation would be to leave the Fabric Interconnects in EHV mode, and use another L2 switch to perform the split between fabrics, such as the following:

This configuration allows the Fabric Interconnect to remain in EHV mode and has the upstream L2 switches performing the split between the L2 domains.  ACLs can be configured on the L2 switches as necessary to isolate the networks, something that cannot be done on the Fabric Interconnect regardless of mode.

Both of these scenarios assume that each of the two customer L2 clouds are using different VLAN numbering, since there’s no capacity in UCS to distinguish between the same VLAN numbers on either Fabric.   There are certainly L3 and other translation tricks that you could use to accomodate this, but that’s an entirely different post.  🙂

UCS Bookstore

Since my previous announcements about various books on UCS and related topics have gotten rolled off the main page, I thought it would be useful to collect them into a bookstore.   I’ve added a link (see the navigation tabs at the top of the screen) to my UCS Bookstore.    Feel free to have a look, and make any suggestions on books you think should be included.

8Gb Fibre Channel or 10Gb Ethernet w/ FCoE?

Update 2011/01/31 – I’ve added new thoughts and comments on the subject here: http://www.unifiedcomputingblog.com/?p=234

Which is better?   Which is faster?

I’ve been stuck on this one for a while.   I’m traditionally a pure fibre channel kind of guy, so I’ve been pretty convinced that traditional FC was here to stay for a while, and that FCoE – as much as I believe in the technology – would probably be limited to the access and aggregation layers for the near term.   That is, until it was pointed out to me the encoding mechanisms used by these two technologies and the effective data rates they allowed.   I’m not sure why it never occurred to me before, but it hit me like the proverbial ton of bricks this week.

First off, a quick review of encoding mechanisms.   Any time we’re transmitting or storing data, we encode it in some form or another.   Generally, this is to include some type of a checksum to ensure that we can detect errors in reading or receiving the data.   I remember the good old days when I discovered that RLL hard drive encoding was 50% more efficient than MFM encoding, and with just a new controller and a low level format, my old 10MB (yes, that’s ten whopping megabytes, kids.   Ask your parents what a megabyte was.)  suddenly became 15MB!   Well, we’re about to embark on a similar discovery.

1, 2, 4, and 8 Gb Fibre Channel all use 8b/10b encoding.   Meaning, 8 bits of data gets encoded into 10 bits of transmitted information – the two bits are used for data integrity.   Well, if the link is 8Gb, how much do we actually get to use for data – given that 2 out of every 10 bits aren’t “user” data?   FC link speeds are somewhat of an anomaly, given that they’re actually faster than the stated link speed would suggest.   Original 1Gb FC is actually 1.0625Gb/s, and each generation has kept this standard and multiplied it.  8Gb FC would be 8×1.0625, or actual bandwidth of 8.5Gb/s.   8.5*.80 = 6.8.   6.8Gb of usable bandwidth on an 8Gb FC link.

10GE (and 10G FC, for that matter) uses 64b/66b encoding.   For every 64 bits of data, only 2 bits are used for integrity checks.   While theoretically this lowers the overall protection of the data, and increases the amount of data discarded in case of failure, that actual number of data units that are discarded due to failing serialization/deserialization is minuscule.    For a 10Gb link using 64b/66b encoding, that leaves 96.96% of the bandwidth for user data, or 9.7Gb/s.

So 8Gb FC = 6.8Gb usable, while 10Gb Ethernet = 9.7Gb usable.   Even if I was able to use all of the bandwidth available on an 8Gb FC port (which is very unlikely at the server access layer), with 10GE running FCoE, I’d still have room for 3 gigabit Ethernet-class “lanes”.   How’s that for consolidation?

10Gb FC has the same usable bandwidth, and without the overhead (albeit a small 2% or so) of FCoE, but you don’t get the consolidation benefits of using the same physical link for your storage and traditional Ethernet traffic.

I’m sold.