I suppose this post has been a long time coming.<\/p>\n
It was spurred into reality by an exchange with @bladeguy who pointed out that Cisco, too, sponsors tests of their equipment – just like HP and the Tolly reports.\u00a0 At first, I’d intended to do a comparison of the Tolly reports and the Principled Technologies reports, looking for obvious (or not so obvious) bias.\u00a0\u00a0 Once I started down that path, however, I realized it really wasn’t necessary.\u00a0\u00a0 Sponsored tests (from any organization) will always be biased, and therefore unreliable from a technical perspective.\u00a0\u00a0 There are always tuning parameters that the “loser” will insist were wrong which skewed the results, there are always different ways to architect the test that would have given the “loser” an edge.\u00a0\u00a0 That’s why they’re sponsored tests.<\/p>\n
I commented briefly before on Tolly’s first HP vs. Cisco UCS report, so I won’t repeat it again here.\u00a0 Suffice it to say, the bloggers and such did a pretty good job of chopping it up.<\/p>\n
My issue with the Tolly reports I’ve seen thus far, the bandwidth debacle and the airflow test, simply don’t pass the smell test.\u00a0\u00a0 Yes, they’re repeatable.\u00a0\u00a0 Yes, the testing results are defensible.\u00a0\u00a0 But the conclusions and declarations of a “winner”?\u00a0 Not so much.\u00a0\u00a0 I’m not faulting Tolly here.\u00a0\u00a0 They’re doing their job, producing the product that their customer has asked them for.\u00a0 These tests aren’t sponsored by the HP engineering groups (for whom I have a lot of respect) looking to validate their technological prowess – they’re sponsored by the marketing departments to provide ammunition for the sales process.\u00a0\u00a0 As such, do you really think they’re going into this looking for a fair fight?\u00a0\u00a0 Of course not.\u00a0\u00a0 They’re going to stack the deck in their favor as much as they think they can get away with (and knowing marketing departments, more than they can get away with).\u00a0\u00a0 That’s what marketing groups (from every vendor) do.<\/p>\n
@bladeguy pointed out that Cisco has engaged Principled Technologies to do some testing of UCS equipment versus both legacy and current HP equipment.\u00a0 At first glance, I didn’t detect any significant bias – especially in the tests comparing legacy equipment to current UCS gear.\u00a0\u00a0 I’m not sure how any bias could be construed, since really they’re just showing the advantage and consolidation ratios capable when moving from old gear to new gear.\u00a0\u00a0 Obviously you can’t compare against Cisco’s legacy servers (since there aren’t any), and HP servers are the logical choice since they have a huge server market share.\u00a0\u00a0 I would suspect that similar results would have been achieved when comparing legacy HP equipment against current HP equipment as well.\u00a0\u00a0 HP can (and I’m sure has) perform similar tests if they’d like to demonstrate that.<\/p>\n
The more troublesome tests are those comparing current generations of equipment from two manufacturers.\u00a0\u00a0 The sponsor of the test will always win, or that report will never see the light of day.\u00a0 That’s simply how it works.\u00a0\u00a0 Companies like Tolly, Principled Technologies, etc aren’t going to bite the hand that feeds them.\u00a0\u00a0 As such, they’re very careful to construct the tests such that the sponsor will prevail.\u00a0\u00a0 This is no secret in the industry.\u00a0\u00a0 It’s been discussed many times before.<\/p>\n
Even the Principled Technologies tests that compared current generations of hardware looked like pretty fair fights to me.\u00a0 If you look closely at the specifications of the tested systems, they really tend to reveal the benefits of more memory, or other such considerations, as opposed to the hardware itself.\u00a0\u00a0 @bladeguy pointed out several items in the Principled Technologies tests that, in his opinion, skewed the results towards Cisco.\u00a0\u00a0 I’m not in any position to refute his claims – but the items he mentioned really come down to tuning.\u00a0\u00a0 So essentially he’s saying that the HP equipment in the test wasn’t tuned properly, and I’m certainly not going to argue that point.\u00a0\u00a0 As a sponsored test, the sponsor will be victorious.<\/p>\n
And therein lies the problem.\u00a0\u00a0 Sponsored tests are meaningless, from any vendor.\u00a0\u00a0 I simply don’t believe that sponsored tests provide value to the technical community.\u00a0 But that’s ok – they’re not targeted at the technical community.\u00a0\u00a0 They’re marketing tools, used by sales and marketing teams to sway the opinions of management decision makers with lots of “independent” results.\u00a0\u00a0\u00a0 If I want to know which server platform is better for my environment, I’m going to do my own research, and if necessary invite the vendors in for a bake-off.\u00a0\u00a0 Prove it to me, with your tuning as necessary, and I’ll have the other vendors do the same.<\/p>\n
My real problem with these tests, understanding that they’re not aimed at the technical community, is the that many in the technical community use them to “prove” that their platform is superior to whoever their competing against at the moment.\u00a0\u00a0 Like politics, these kinds of arguments just make me tired.\u00a0\u00a0 Anyone coming into the argument already has their side picked – no amount of discussion is going to change their mind.\u00a0\u00a0 My point for blogging about UCS is not to sell it – I don’t sell gear.\u00a0\u00a0 It’s because I believe in the platform and enjoy educating about it.<\/p>\n
I happen to prefer Cisco UCS, yes.\u00a0\u00a0 If you’ve ever been in one of my UCS classes, you’ll have also heard me say that HP and IBM – Cisco’s chief rivals in this space – also make excellent equipment with some very compelling technologies.\u00a0\u00a0 The eventual “best” solution simply comes down to what’s right for your organization.\u00a0\u00a0 I understand why these sponsored tests exist, but to me, they actually lessen the position of sponsor.\u00a0\u00a0 They make me wonder, “if your product is so good, why stack the deck in the test?”\u00a0\u00a0 The answer to that, of course, is that the engineers aren’t the ones requesting or sponsoring the test.<\/p>\n
As came up in my UCS class today, for the vast majority of data center workloads, the small differences in performance that you might be able to get out of Vendor X or Vendor Y’s server is completely meaningless.\u00a0\u00a0 When I used to sell\/install storage, I used to get asked the question as to which company’s storage to buy if the customer wanted maximum performance.\u00a0 My answer, every single time was, “HP, or IBM, or HDS, or EMC, or…”\u00a0\u00a0 Why?\u00a0 Because technology companies are always leapfrogging each other with IOPS and MB\/s and any other metric you can think of.\u00a0\u00a0 What’s the fastest today in a particular set of circumstances will get replaced tomorrow by someone else.<\/p>\n
So what’s the solution?\u00a0 Well, true independent testing, of course.\u00a0\u00a0 How do you do true independent testing?\u00a0\u00a0 You get a mediator (Tolly, Principled Technologies, etc are fine), representatives from both manufacturers to agree on the testing criteria, and allow each manufacturer to submit their own architecture and tuning to meet the testing criteria.\u00a0\u00a0 The mediator then performs the tests.\u00a0\u00a0\u00a0 The results are published with the opportunity for both manufacturers to respond to the results.\u00a0\u00a0 Think any marketing organization from any company would ever allow this to happen?\u00a0\u00a0 The standard line in the testing industry is “Vendor X was invited to participate, but declined.”\u00a0 Of course they declined.\u00a0\u00a0 They’ve already lost before the first test was run.\u00a0\u00a0 I wouldn’t expect Cisco to participate in a HP-sponsored Tolly test any more than I’d expect HP to participate in a Cisco-sponsored Principled Technologies test.<\/p>\n
Don’t chase that 1% performance delta.\u00a0\u00a0 You’ll just waste time and money.\u00a0\u00a0 Find the solution that meets your organizational needs, provides you the best management and support options, and buy it.\u00a0\u00a0 Let some other chump chase that magical “fastest” unicorn.\u00a0\u00a0 It doesn’t exist.\u00a0 As in all things IT, “it depends.”<\/p>\n
All comments are welcome, including those from testing organizations!<\/p>\n","protected":false},"excerpt":{"rendered":"
I suppose this post has been a long time coming. It was spurred into reality by an exchange with @bladeguy who pointed out that Cisco, too, sponsors tests of their equipment – just like HP and the Tolly reports.\u00a0 At first, I’d intended to do a comparison of the Tolly reports and the Principled Technologies … Continue reading The Problem With Vendor Sponsored Testing<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[28,51],"class_list":["post-161","post","type-post","status-publish","format-standard","hentry","category-competitive","tag-hp","tag-tolly"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts\/161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=161"}],"version-history":[{"count":0,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=\/wp\/v2\/posts\/161\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.unifiedcomputingblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}