The Pleasure and Peril of Comparison Shopping

Making decisions about IT infrastructure purchases is hard. There are many interlocked decisions to be made, each of which has multiple requirements. Often, a collection of compromises must occur. For the business project owner with little IT knowledge, it may seem far too complicated. It would be great to have tools that simplify these decisions and reduce the work required to make complicated decisions. It would also be useful to have an independent analysis that supports the decisions made.

One way to simplify these decisions is to rely on the expertise of others. Before the internet and blogging, a major source of expertise was print magazines. A chunk of revenue from a magazine’s advertisers was fed into a test lab. Vendors might submit products for evaluation, or the magazine might buy products. After weeks of testing, the magazine could publish reviews that summarized its findings and analysis. Even better were bake-offs, in which several products were tested against the same criteria. The magazine’s readers could then make informed decisions about which product to buy for their needs. Alas, the days of thick, advertising-heavy magazines have gone. Those advertisers now run their own websites or simply list on Amazon. With no advertising revenue, there is no lab budget. Most of the magazines have gone online now, too.

Where does an organization go these days to find a bake-off of infrastructure products? If you are Megacorp, then you run your own: a series of proof of concept (PoC) projects, one with each of the possible solutions. The reason that only Megacorp can do this is that PoCs cost a lot of money, mostly in staff salaries. Wouldn’t it be great if Megacorp published its results? Alas, the right choice produces market differentiation for Megacorp, so the results are very secret. For smaller organizations, the cost of running multiple PoCs exceeds the cost of choosing the wrong product. It would be nice if there was a place we could go to get comparison information to help us make our decisions.

One useful site for crowdsourced product comparisons is now available. WhatMatrix provides community-driven comparisons of IT infrastructure products. It covers a range of common product categories and a pretty comprehensive set of comparison criteria. Much of the analytical work of designing comparisons and evaluating products has been completed by independent consulting organizations. I have to expect that some customers will approach these organizations for assistance with projects as a result. The website allows for comparisons of up to three products, within one category, at a time. It has nice red/yellow/green color coding for some of the comparison lines, helping you zero in on areas of difference. Overall, it’s a pretty cool site, and the results are available for free. Another nice thing about it is that you can use the site to decide what other products in the category you should evaluate, as well as possibly rule some products out before spending more time or money on further evaluation.

Now let’s consider the risks of this sort of comparison shopping. The first thing to remember is that there is (alas) no one solution that is right for every situation. Even though one product may have the highest score in a category, that doesn’t mean it is always the best choice in that category. One thing you will need to do is look at the evaluation criteria and work out whether they map to your own needs. For example, a company may have a corporate standard for a specific hypervisor. WhatMatrix will score a solution that supports multiple hypervisors higher than one that only supports a single hypervisor. This is not part of the companys criteria, however; the business only cares that the solution supports the one hypervisor that it will be using.

Another factor to be aware of is a bias among the independent consultants. I don’t mean bias in a bad way: everyone has a bias based on their own knowledge and experience. Personally, I have done a lot of work with hyperconverged, recently with SimpliVity and in the past with Nutanix. Because I’m familiar with these products, I’m inclined to think they are good solutions for customers. I’m less familiar with using a NetApp/Cisco FlexPod, so I am less likely to see it as a solution for a problem. Everyone has a bias, and it is worth considering the bias of any analysis you use.

IT infrastructure decision making is complex, and there are usually multiple right answers. Relying on some external expertise is a reality with the decision process. WhatMatrix makes some great analysis available for free. Just make sure the analysis fits your situation.

Posted in SDDC & Hybrid CloudTagged , ,

Leave a Reply

2 Comments on "The Pleasure and Peril of Comparison Shopping"

Sort by:   newest | oldest | most voted
Thanks for your insight and comments Alastair. We’d like to address your concerns in our comments below. “One thing you will need to do is look at the evaluation criteria and work out whether they map to your own needs” Yes, totally concur. Therefore, in the next major release we will provide a “use case” filter and customization option. In simple terms this will allow the user to prioritize or exclude features according to THEIR deployment scenario and preferences (e.g. “integration with hypervisor xyz” might be very important to you if that is what you use – then again it… Read more »