The Problem With Vendor Sponsored Testing

I suppose this post has been a long time coming.

It was spurred into reality by an exchange with @bladeguy who pointed out that Cisco, too, sponsors tests of their equipment – just like HP and the Tolly reports.  At first, I’d intended to do a comparison of the Tolly reports and the Principled Technologies reports, looking for obvious (or not so obvious) bias.   Once I started down that path, however, I realized it really wasn’t necessary.   Sponsored tests (from any organization) will always be biased, and therefore unreliable from a technical perspective.   There are always tuning parameters that the “loser” will insist were wrong which skewed the results, there are always different ways to architect the test that would have given the “loser” an edge.   That’s why they’re sponsored tests.

I commented briefly before on Tolly’s first HP vs. Cisco UCS report, so I won’t repeat it again here.  Suffice it to say, the bloggers and such did a pretty good job of chopping it up.

My issue with the Tolly reports I’ve seen thus far, the bandwidth debacle and the airflow test, simply don’t pass the smell test.   Yes, they’re repeatable.   Yes, the testing results are defensible.   But the conclusions and declarations of a “winner”?  Not so much.   I’m not faulting Tolly here.   They’re doing their job, producing the product that their customer has asked them for.  These tests aren’t sponsored by the HP engineering groups (for whom I have a lot of respect) looking to validate their technological prowess – they’re sponsored by the marketing departments to provide ammunition for the sales process.   As such, do you really think they’re going into this looking for a fair fight?   Of course not.   They’re going to stack the deck in their favor as much as they think they can get away with (and knowing marketing departments, more than they can get away with).   That’s what marketing groups (from every vendor) do.

@bladeguy pointed out that Cisco has engaged Principled Technologies to do some testing of UCS equipment versus both legacy and current HP equipment.  At first glance, I didn’t detect any significant bias – especially in the tests comparing legacy equipment to current UCS gear.   I’m not sure how any bias could be construed, since really they’re just showing the advantage and consolidation ratios capable when moving from old gear to new gear.   Obviously you can’t compare against Cisco’s legacy servers (since there aren’t any), and HP servers are the logical choice since they have a huge server market share.   I would suspect that similar results would have been achieved when comparing legacy HP equipment against current HP equipment as well.   HP can (and I’m sure has) perform similar tests if they’d like to demonstrate that.

The more troublesome tests are those comparing current generations of equipment from two manufacturers.   The sponsor of the test will always win, or that report will never see the light of day.  That’s simply how it works.   Companies like Tolly, Principled Technologies, etc aren’t going to bite the hand that feeds them.   As such, they’re very careful to construct the tests such that the sponsor will prevail.   This is no secret in the industry.   It’s been discussed many times before.

Even the Principled Technologies tests that compared current generations of hardware looked like pretty fair fights to me.  If you look closely at the specifications of the tested systems, they really tend to reveal the benefits of more memory, or other such considerations, as opposed to the hardware itself.   @bladeguy pointed out several items in the Principled Technologies tests that, in his opinion, skewed the results towards Cisco.   I’m not in any position to refute his claims – but the items he mentioned really come down to tuning.   So essentially he’s saying that the HP equipment in the test wasn’t tuned properly, and I’m certainly not going to argue that point.   As a sponsored test, the sponsor will be victorious.

And therein lies the problem.   Sponsored tests are meaningless, from any vendor.   I simply don’t believe that sponsored tests provide value to the technical community.  But that’s ok – they’re not targeted at the technical community.   They’re marketing tools, used by sales and marketing teams to sway the opinions of management decision makers with lots of “independent” results.    If I want to know which server platform is better for my environment, I’m going to do my own research, and if necessary invite the vendors in for a bake-off.   Prove it to me, with your tuning as necessary, and I’ll have the other vendors do the same.

My real problem with these tests, understanding that they’re not aimed at the technical community, is the that many in the technical community use them to “prove” that their platform is superior to whoever their competing against at the moment.   Like politics, these kinds of arguments just make me tired.   Anyone coming into the argument already has their side picked – no amount of discussion is going to change their mind.   My point for blogging about UCS is not to sell it – I don’t sell gear.   It’s because I believe in the platform and enjoy educating about it.

I happen to prefer Cisco UCS, yes.   If you’ve ever been in one of my UCS classes, you’ll have also heard me say that HP and IBM – Cisco’s chief rivals in this space – also make excellent equipment with some very compelling technologies.   The eventual “best” solution simply comes down to what’s right for your organization.   I understand why these sponsored tests exist, but to me, they actually lessen the position of sponsor.   They make me wonder, “if your product is so good, why stack the deck in the test?”   The answer to that, of course, is that the engineers aren’t the ones requesting or sponsoring the test.

As came up in my UCS class today, for the vast majority of data center workloads, the small differences in performance that you might be able to get out of Vendor X or Vendor Y’s server is completely meaningless.   When I used to sell/install storage, I used to get asked the question as to which company’s storage to buy if the customer wanted maximum performance.  My answer, every single time was, “HP, or IBM, or HDS, or EMC, or…”   Why?  Because technology companies are always leapfrogging each other with IOPS and MB/s and any other metric you can think of.   What’s the fastest today in a particular set of circumstances will get replaced tomorrow by someone else.

So what’s the solution?  Well, true independent testing, of course.   How do you do true independent testing?   You get a mediator (Tolly, Principled Technologies, etc are fine), representatives from both manufacturers to agree on the testing criteria, and allow each manufacturer to submit their own architecture and tuning to meet the testing criteria.   The mediator then performs the tests.    The results are published with the opportunity for both manufacturers to respond to the results.   Think any marketing organization from any company would ever allow this to happen?   The standard line in the testing industry is “Vendor X was invited to participate, but declined.”  Of course they declined.   They’ve already lost before the first test was run.   I wouldn’t expect Cisco to participate in a HP-sponsored Tolly test any more than I’d expect HP to participate in a Cisco-sponsored Principled Technologies test.

Don’t chase that 1% performance delta.   You’ll just waste time and money.   Find the solution that meets your organizational needs, provides you the best management and support options, and buy it.   Let some other chump chase that magical “fastest” unicorn.   It doesn’t exist.  As in all things IT, “it depends.”

All comments are welcome, including those from testing organizations!

Fantastic post on statelessness : HP VirtualConnect vs. Cisco UCS

M. Sean McGee posted this great comparison of VirtualConnect and UCS.   I’ve often struggled to give students a clear picture of the differences – HP will tell you that “VirtualConnect is just as good, and we’ve been doing it for years!”.  Well, yes… it does some things similarly, and you can’t argue the timeframe.   UCS does a lot more – and until now, I didn’t have a great source that directly compared them.   From now on, all I have to do is send them to M. Sean McGee’s post!

The State of Statelessness

Tolly Report

A lot of people have been asking me what I think of the recently released Tolly report comparing the bandwidth of the HP and Cisco blade solutions.

The short answer is, I don’t think much of it.   It’s technically sound, and the the conclusions it reaches are perfectly reasonable – for the conditions of the tests they performed.   In keeping with Tolly’s charter, the tests were repeatable, documented, and indisputable.  The problem is, the results of the testing only tell half the story.   The *conclusions* they reach, on the other hand, aren’t as defensible.

It’s really not necessary to get into a point by point rebuttal.   At least not for me.   I’m sure Cisco will be along any minute to do just that.

The facts that Tolly established during the report aren’t groundbreaking or surprising.   Essentially, the tests were built to demonstrate the oversubscription of links between UCS chassis and Fabric Interconnects, which Cisco is quite willing to disclose at 2:1.  These tests were paired with HP comparisons of blade-to-blade traffic on the same VLAN, which in HP architectures keeps the traffic local to the chassis.  The interesting thing there is that if the traffic were between the same blades but in different VLANs, the Cisco and HP solutions would have performed identically (assuming the same aggregation-layer architecture).   What makes that interesting is that the Tolly report’s figures depend on a specific configuration and test scenario – the Cisco values won’t change (or get any worse) no matter how you change the variables.  The HP values will vary widely.

And that, my friends, is where I see the true benefit of Cisco’s architecture.   Change the conditions of the test repeatedly, and you’ll get the same results.

I’m not faulting Tolly in this case.  Not at all.  They were asked to test a certain set of conditions, and did so thoroughly and presumably accurately.  It’s just that you can’t take that set of data and use it to make any kind of useful comparison between the platforms.   The real world is much more complicated than a strictly controlled set of test objectives.   Do we really think that HP went to Tolly and asked “We want a fair fight?”   Of course not.

Cisco Kicks HP to the Curb

Well, it’s official.   HP is not going to be a Cisco gold partner any longer.

Given HP and Cisco’s very public competition, I can’t say this is any surprise.  While HP certainly has contributed significant sales to Cisco in the past in the form of routing and switching equipment, HP has aggressively moved to position their own products in front of Cisco’s recently (and why wouldn’t they?).

Does anyone think this actually means much for the sales of either company, or more of a “yawn” type move?

Full Story Here

Excellent discussion and comparison of UCS and HP blade offerings

Over at By the Bell, there’s a great post and commentary on differences between HP’s offerings and Cisco UCS.   As many of the comments point out, the post itself is a bit biased towards UCS, but the comments are an excellent set of discussions about the various pros and cons of the solutions.  As good as the post is, the comments (I think) are even better.   Check it out!