Lies, fucking lies and Benchmarks

This blog contains only two entries until today but both have a very interesting view to different parts of the x86-world: Scientias Blog. The better article is The Dishonesty of Overclocking. I want to cite an entire part of the text, as it describes the problem of benchmarks and Woodcrest perfectly:

Dual core processors are becoming more common now. AMD had designed the Athlon 64 for dual core from the beginning. Intel's first attempts at dual core by simply sticking two chips inside the same package gave very poor results. Intel is doing much better now with the Core 2 Duo chips. However, there are some differnces in architecture and these differences are used to give Intel an advantage during testing. Conroe has a 4MB L2 cache that is shared between the two cores whereas K8 has a separate 1MB L2 cache for each core. One of the latest common ways to skew results is to only run a single benchmark so that the Intel core can use the entire L2 cache. Since AMD's cores have separate caches this only gives an advantage to Intel. This is very dishonest. The whole point of having dual core is to use both. The proper way to test would be to run a program on the second core to load it while the first core is tested. Running the same benchmark on both cores is not a proper test as this allows the Intel processor to share the benchmark code betwen the cores and save some space in the cache. In the real world, it is incredibly unlikely that anyone would have the same code in use by both cores at the same time. The program that loads the second core should be different from what is used on the first core to have a proper test. Yet, this proper way of tesing is still being avoided by review sites because the Core 2 Duo chips would not only have to share the oversized cache between the two cores but would also have to share meory accesses using the slower FSB. This kind of proper testing could end up erasing all of Conroe's apparent advantage and no review site has risked this publicly. If they have done this kind of tesing privately they have not published the results.

I strongly recommend this blog. I hope it stays at this high level …