About SPECsfs, benchmarks and benchmarketing

Bryan Cantrill wrote an excellent article about the SPECsfs: Eulogy for a benchmark. This article describes the shortcomings of the SPECsfs and the consequences to benchmarketing configurations (like a 228 short-stroking disk configuration for a benchmark with NetApp):

Be it due to incompetence or malice, SPEC's descent into a disk benchmark while masquerading as a system benchmark does worse than simply mislead the customer, it actively encourages the wrong engineering decisions. In particular, as long as SPEC SFS is thought to be the canonical metric of NFS performance, there is little incentive to add cache to NAS heads. (If SPEC SFS isn't going to use it, why bother?) The engineering decisions made by the NAS market leaders reflect this thinking, as they continue to peddle grossly undersized DRAM configurations -- like NetApp's top-of-the-line FAS6080 and its meager maximum of 32GB of DRAM per head! (By contrast, our Sun Storage 7410 has up to 128GB of DRAM -- and for a fraction of the price, I hasten to add.) And it is of no surprise that none of the entrenched players conceived of the hybrid storage pool; SPEC SFS does little to reward cache, so why focus on it? (Aside from the fact that it delivers much faster systems, of course!)

A must-read article! His introductory sentence “I come to bury SPEC SFS, not to praise it.” has to be taken seriously.