|HOME||PUBLIC LIBRARY||ANOTHER PERSPECTIVE||INFOPERSPECTIVES||CONTACT|
There's nothing quite as exciting as a battle between giants, particularly if observed from a safe distance. And there's nothing quite so scary as a battle between giants, particularly if you happen to be close enough to wonder whether you might suffer collateral damage. That's why lots of people flock to sumo wrestling matches but stay clear of the ring. And that's why lots of computer users pay attention to benchmark contests but are wary of justifying strategic decisions on the basis of these tests.
In Japan, the home of sumo, there is one set of rules used to rank the wrestlers, called rikishi, and one association that manages the major contests, Nihon Sumo Kyokai. In computer benchmarking, there seems to be no limit to the number of ways systems can be tested. Some tests are created and run by independent organizations like the Transaction Processing Performance Council and the Standard Performance Evaluation Corp., some by software vendors like SAP, and some by server vendors like IBM.
The various test results are not consistent, except, at best, with themselves. But they do produce winners and losers, leaders and followers, so they can attract considerable attention, particularly when they yield unexpected results. This element of surprise is what makes benchmarking at its most interesting resemble a sports contest. There is one other aspect of benchmarking that can cause a commotion, and that is when a participant is accused of bending or breaking the rules. This, too, is the case in sports.
Sumo has very few rules. To win, one rikishi has to force the other outside of the ring, which is about 15 feet in diameter, or get him to touch the clay floor with anything but the soles of his feet. How a fighter, who might weigh 350 or 400 pounds, actually does this is pretty much an open question. Only a few things are prohibited. A sumo wrestler must not hit with a closed fist, pull hair, gouge an eye, choke, kick the chest or stomach (which, many years ago, in a famous sumo contest, proved to be fatal), or grab the part of the other wrestler's costume that covers his vitals. That leaves each contestant with quite a few choices in the way of slapping, pushing, kicking, grunting, head-butting, and shoving, as well as all kinds of yanking of the big waistband each rikishi wears.
Still, in what sumo aficionados consider a great breach of etiquette, two Mongolians who rank pretty high in big-league sumo, which is of course dominated by the Japanese, have been behaving even more badly than the rules allow. Asashoryu, characterized in an Associated Press dispatch as "a fiery, heavily muscled 22-year-old from Mongolia who is arguably the best wrestler in the ring today," pulled the hair of fellow countryman Kyokushuzan.
This foul was so egregious a violation of sumo tradition that it was reported as far away as Concord, New Hampshire, where only the most dramatic stories of sumo get press attention, particularly in the summer, when local attention turns to such topics as how Vermont got lucky with good revenue producers like Ben & Jerry's when all they seem to get are tax refugees from Massachusetts.
Hair-pulling is pretty rare in benchmarking, except when losers pull out their own, but some experts say very unsportsmanlike things do happen.
In June, Tom's Hardware, a Web site respected by people who are interested in PC technology, accused a couple of motherboard makers of using tricks to make their products run faster on various benchmark tests.
Impressive benchmark results directly impact sales in the computer hotrod market, where users build their own machines, and indirectly affect the big vendors that buy motherboards by the millions for machines aimed at ordinary business and home users. Hewlett-Packard, for instance, uses motherboards from Asus and MSI, among others, in its popular home computers, and one consideration in HP's decision-making process is just how peppy a particular machine will be.
Efforts to gain an edge in benchmarks are not at all confined to the PC market. Vendors of servers and workstations also put a lot of work into improving benchmark results. Their efforts are usually subject to scrutiny, and arguments among performance specialists about some of the techniques used to goose performance are generally more complicated and less definitive than those that have cropped up in the PC game. In the workstation and server markets, SPEC (Standard Performance Evaluation Corp.) benchmarks are often cited as indicators of performance. Critics say the SPEC tests are sensitive to compiler tricks, but whenever there is carping the SPEC people seem to tighten up their rules. Still, the best efforts of the SPEC folk were not good enough to prevent controversy surrounding the performance claims made by Apple when it introduced its G5 boxes, machines that compete with Intel-based workstations.
But controversy isn't necessarily a bad thing. In 1993, American-born Chad Rowan, wresting under the name Akebono, became the number-one sumo wrestler in Japan. He was only the 64th rikishi to hold the top rank in the 1,500 year history of sumo, and was the first who was not Japanese. Perhaps Akebono's achievement should not have been a surprise. After all, if a young man who was six foot eight and weighed more than 500 pounds wanted to be a sumo wrestler, who could stop him? Not even another giant, it turns out, and quite a few tried.
Even where there is no question that the benchmark participants are within the rules, there are absurdities behind some of the results that vendors tout in their promotions. On August 5, IBM, claiming with some justification that its DB2 is a top-notch product when it comes to the TPC-C transaction processing benchmark, noted that the test results it got using a pSeries 690 reflected a simulated company with more than 60,000 warehouses.
And then there are claims based on estimates, claims about systems that are not yet in place and actually running. These claims crop up in supercomputing, most recently in late July, when IBM said it was building the world's third-fastest clustered computer system, a 1318-server cluster, for the Japanese National Institute of Advanced Industrial Science and Technology. A week later Dell promised to dethrone IBM with a 1280-server cluster, which it sold to the National Center for Supercomputing Applications at the University of Illinois. That system, not IBM's, would be the third-fastest computer on earth, based on projected performance, according to Dell.
But a lot of things could happen between now and the time Dell's biggest cluster gets installed. Like Asashoryu, Dell, taunted by adversaries, and new to the game, could manage to get disqualified or upstaged by another system going to another location. If something like that happened, we would not expect Dell to do anything embarrassing, as Asashoryu did. Not long after he lost a battle he would easily have won had it not been for the hair-pulling incident, Asashoryu walked past a Mercedes that was waiting for his nemesis Kyokushuzan. Depending on which account you believe, Asashoryu either accidentally or deliberately bumped into the car, causing considerable damage.
Memo to Michael Dell: Be careful where you park your clusters.
— Hesh Wiener August 2003