Some thoughts about SPECpower_ssj2008

SPEC announced a new benchmark: SPECpower_ssj2008. After reading the documentation, i started to dislike this new benchmark. Not because it´s badly designed, but i can imagine the benchmarketing tricks. It´s about the devil in the details. Power measurement is not an exact science. This begins with the variances between components of the same type. When you have a CPU rated as a 95 Watts CPU it´s not unusual that it needs something in the range between 70 and 95 Watts. The rules of the benchmark doesn´t mandate you to use a part with exactly the rated TDP. So it would be feasible to buy a large heap of a CPU and test for the CPU with the lowest power consumption. The problem: The benchmarked system and the system in your server rack may be the same from the rating of their components but you won´t be able to reach the same levels of power consumption. As the benchmarks doesn´t specify much about the configuration, it´s save to assume, that we will see many “power benchmark only”-specials : For example single power supply configurations. You won´t use such a server in your datacenter, but using 300 watts from a single 1200 watts power supply would give you a more efficient operation than using 150 Watts each from two 1200 watts power supplies. Configurations will be “reduced to the max”: Disabling NICs (the benchmark only mandates one interface), single harddisks, special Windows and BIOS settings. Additionally this benchmark will favour systems with complex power saving mechanisms. This is somewhat flawed out of two reasons. At first this benchmark shows something important. Look for example at the results from HP for their DL160: An idling system doesn´t need 0 watts, it needs more than the half of the power of an fully loaded system. Power saving may be a good thing for desktops being to fast for writing mail and doing some text processing. But you don´t want an idling server. You want to load it. At the end, this is the reason for virtualisation like VMware, vXM or Solaris Containers. Running under full load is the most efficient operational environment for any server (as the benchmark disclosures will show you) and thus any architecture should be designed for this operation point. But the SPECpower_ssj2008 doesn´t reflect this. I would prefer a different benchmark: Make the measurement and reporting of power consumption mandatory for every benchmark. Consumed kilowatt hours per benchmark run. This would prevent power saving configurations as well as 3500 hard disk performance specials for certain benchmarks.