Great post, and I commend you on taking an objective stance. The moral is always the test will always reflect the funding and therefore these tests must be taken with a grain of salt or thrown out completely.

My major complaint with tests like the ones your describing is the summaries of results and assumptions obtained from the data. Some of the recent reports in question start with a summary that is far from an independant assesment.

If as Kevin states:

“In any given test project, we state what we are trying to prove, how we are going about proving same and what we found. We don’t try to make decisions for users, It is up to them to decide whether a given test is relevant to their needs.”

Then they should be very wary of making emphatic statements about the results if the results were based on testing designed to highlight strengths compared to weaknesses.

Joe

“In any given test project, we state what we are trying to prove, how we are going about proving same and what we found. We don’t try to make decisions for users, It is up to them to decide whether a given test is relevant to their needs.

We clearly state the level of vendor involvement when running a competitive test. Yes, there are times when Cisco (and other vendors) get involved in tests sponsored by competitors. In fact, it is more often than not that they DO get involved. And, yes, we do conduct tests where HP (and other Tolly test sponsors) is the competitor. In our most recent LAN switch test, sponsored by LG-Nortel and involving Cisco, Cisco notified us that they had a new model switch that would be more appropriate for the test. Cisco shipped us their preferred switch, it was tested in place of the original and those results were published. Furthermore, that test also compared LG-Nortel vs two HP LAN Switches. HP participated in the test and also confirmed the accuracy of their results. Please see: http://www.tolly.com/DocDetail.aspx?DocNumber=210125.

I don’t disagree with you that, in a perfect world, each user should run their own bake-off. While that may be possible with certain technologies and products, there are many tests that we run that require million-dollar-plus test rigs, a lot of test tool expertise and sometimes weeks of lab time. Most prospective buyers just can’t run them. In fact, many publications that feature a “test lab” cannot even test gear properly as they, too, do not have the requisite time, gear or expertise.

Whenever possible, we run our tests according to “common test plans” that we have developed with input from interested technologists from the vendor and analyst/consultant community. They do not yet include server tests but do cover LAN Switch and VoIP “green” tests, Data Loss Prevention, App Switching (L4-7) and other areas. We maintain a Common Test Plan site (http://www.commontestplan.org/) and are always open to suggestions for further test plans.

Thanks for your interest in the matter. Please feel free to post and/or share with your readers as you see fit.

Cordially,
Kevin Tolly
Founder
The Tolly Group”