IBM's reaction to Sun/Oracle TPC-C announcement
I expected a reaction from IBM in regard of Sun/Oracle. But not such a lame one as provided by Mrs. Stahl of IBM. Let’s forget for a minute that TPC-C is a horrible benchmark. What is a benchmark? A benchmark is a measure how fast a system can do things. Measuring how fast a system can count to one million is as well a benchmark as doing the general ledger for a company. When it’s your job to count to one million you would use the fastest system in that benchmark. If it’s your job to do a general ledger, you would use a benchmark testing that task. TPC-C is now a benchmark testing the capabilities of the sytem to do OLTP loads. Of course this benchmarks reached numbers beyond any reasoning in the real world. But for a long time IBM saw in this benchmark a good representation of OLTP loads. So let’s just assume you have a load that you think is reasonably near to the TPC-C load. What are the interesting numbers for you: Well …. of course performance of the complete architecture. But there are several important other metrics: Performance per $ for example (as your budget has limits), or performance per square meter (as your datacenter space has limits). Given that you have a load like the one specified by TPC-C it’s irrelevant how you reached the performance, just that you reach the performance. The Sun benchmarking team has demonstrated, that their results were better as well in performance per dollar ( Sun $2.34 vs. IBM $2.81 per tpmC) as well as in performance per rack (Sun 9 Racks vs. IBM 76 Racks). IBM tries now to tell the public, that this benchmark shows that Power is still superior, because they have more tpmC per Core. 4.7 times more tpmC per core - to be exact. I could calculate another derivate out of this - Price per Core in the configuration used for the benchmark: $18,051,719 / 384 cores = $47,009/core at Sun and $17,111,788 / 64 cores = $267,371/core for IBM. In the TCO calculation the single core at IBM is 5.6 times more expensive. This is the context you have to know, when Mrs. Stahl writes that a single core delivers more performance per core. But both is totally irrelevant for this benchmark. What counts performancewise is the number of TPC-C transactions per minute. Period. When you want to proof the impact of a better single-core performance, use a different benchmark. The business of a customer is doing transactions in his architecture on his complete conglomerate of servers, storage and networking, not doing transactions on a single core. And the customer buy a certain number of cores to do the job, not a single core. For the customer the way to yield a certain level of performance is irrelevant. Cluster or single system? Irrelevant! Fewer fast cores or more slower cores? Irrelevant! Of course the customer must be sure, that he has chosen the right benchmark … that TPC-C is representative for his or her workload, but that’s the real problem of TPC-C. IBM has three options now: Admitting that Sun/Oracle has beaten IBM in TPC-C or admitting that TPC-C isn’t a good benchmark. Both wont happen from my perspective: The first option would be really bad PR-wise. The second one would be bad, as TPC-C is the spine of many relative performance numbers, that IBM is using in competitive sizings as far as i know. Even the independent RPE benchmark from IDEAS contained TPC-C. As IBM mastered the art of optimizing for TPC-C and the TPC-Values for Sun were estimated, those relative numbers were somethat skewed. However, both outcomes would be advantageous for Sun. But i assume, they will just opt for the third option and will spindoctoring the results like in this article of Mrs. Stahl at developerworks. Must be bad, when the student is beaten by a horde of toddlers in squirrel costumes. PS: I’ve read the FDR for the Sun result the second time now … i have somehow the inkling, that this system use by Sun fought with one arm bound to the back. But i think i have to read it a third time to substantiate my thought. Disclosure Statement: TPC Benchmark C, tpmC, and TPC-C are trademarks of the Transaction Performance Processing Council (TPC). 12-node Sun SPARC Enterprise T5440 Cluster (1.6GHz UltraSPARC T2 Plus, 4 processor) with Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning, 7,717,510.6 tpmC, $2.34/tpmC. Available 12/14/09. IBM Power 595 (5GHz Power6, 32 chips, 64 cores, 128 threads) with IBM DB2 9.5, 6,085,166 tpmC, $2.81/tpmC, available 12/10/08. Source: http://www.tpc.org, results as of 10/11/09.