The saga of Mindcraft and its Windows NT vs. Linux shootout continues. On May 4, Bruce Weiner, president of the Mindcraft performance testing lab, posted an "Open Benchmark Invitation" on his Web site inviting the Linux community to participate in yet another comparison between Windows NT and Linux.
"I thought it only fair to do an Open Benchmark giving them everything they ask for," says Weiner -- whose company has come under sustained and virulent criticism from the Linux community in recent weeks. "At that point, they cannot reasonably claim that the benchmark was biased."
But the immediate response from leaders in the Linux world can be summed up succinctly: No way in hell. As far as Linux creator Linus Torvalds, leading Linux hacker Alan Cox and Red Hat CEO Bob Young are concerned, there's no advantage to be gained from participating in what they consider to be a game that's still rigged in Microsoft's favor.
"The entire test is engineered to favor NT," says Alan Cox, who laid out his technical objections in a recent LinuxToday article. "His 'open test' is simply tying the open software community's hands while providing a facade of openness."
On April 13, Mindcraft announced the results of its first test comparing NT and Linux. NT, Mindcraft said, performed significantly better in key areas such as file- and Web-serving. The Linux community immediately erupted -- and with some justification. As was eventually discovered, the test had been commissioned by Microsoft and conducted in a Microsoft computer lab with the help of at least one Microsoft independent contractor.
Within weeks, Mindcraft announced that it was redoing the test, this time with the help of the "top minds" of the Linux community. But, as Salon reported on April 27, Torvalds and Cox both expressed extreme dissatisfaction with the amount of direct input they were allowed. No outside observers would be permitted at the test, and they said they weren't being given enough information to make a difference.
Microsoft, says Weiner, is withholding the results of the second test until a third test can be carried out. This time, he says, Linux representatives will be encouraged to be physically present at a neutral site.
"Mindcraft wants to show we really did run a fair and accurate benchmark the second time," says Weiner. "We wanted to do that the first time but we unfortunately did make some mistakes tuning Linux, Apache and Samba. Despite the flames from many Linux proponents, we really do want to show Linux, Apache and Samba at their best on an enterprise-class system."
But Torvalds and Cox say that there are still some serious problems with how the new test is set up. One of the conditions set by Mindcraft is that no new patches or fixes can be incorporated in Linux that weren't already publicly available on April 20. This isn't fair, says Torvalds: For one thing, one of the main strengths of Linux is the ability of its open-source community to respond with new patches and fixes for every problem that arises. Even worse, adds Torvalds, Microsoft had the opportunity to "tune" NT for the particular hardware configuration that will be used in the test "for years."
"It's simply the same thing all over," says Torvalds. "It's a benchmark that they [Microsoft] selected and have been tuning for for a long time, and one that has not been tuned at all for in Linux. I would not be in the least surprised if we wouldn't want to do some kernel tuning, but their 'open rules' forbid that."
Weiner says, however, that the main purpose of the new test "is to address their concerns about the first and second tests. We were getting blasted by them and others, so we want to demonstrate how the products perform in the environment they were tested [in] ... For them to offer to come in and want to do the tuning, and then to back out of this open invitation because it is specifically an environment to reproduce the tests they were complaining about -- that's very disingenuous of them."
Weiner also says that he is currently rewriting the invitation to take into account some of the objections that have already been raised by the Linux community. But at this point, judging by the reaction of Linux leaders, any further accommodation between Mindcraft and the Linux world seems unlikely.
"I'll leave it to Red Hat whether they really feel they have to respond" to Mindcraft's invitation, says Torvalds. "I'd much rather make sure that we have a more balanced view -- make sure that if we really want an independent and fair test, both parties can have their own sub-tests, and the test configuration won't be dictated by one party."
Red Hat, meanwhile, doesn't seem in a hurry to help out.
"If Microsoft wants to bring a real benchmarking organization to this enterprise, like Ziff Davis, or the Meta group up in Connecticut," says Red Hat's Young, "if they want to bring someone who has some industry recognition, then absolutely we would be thrilled to participate."
Weiner bristles at the implication that Mindcraft is not independent.
"I just don't see why this is seen as a Microsoft test," says Weiner. "That's sort of a funny response from Bob Young. By not participating, he's saying that [Red Hat is] not going to do well. We got excellent support from Red Hat on our second test ... I think his lack of participating is something one ought to question."
"Microsoft owns no stock in us," adds Weiner. "Microsoft is a client of ours, as are a bunch of other hardware and software vendors. We've done a lot of Unix work in the past. I personally received a Unix Evangelist award back in 1993, as one of the early promoters of Unix. I have very fond and warm feelings towards all of those Unix derivatives. I would think [Bob Young] would welcome this as an opportunity to see how these things work. And, by doing that, he would learn where his product's strengths and weaknesses are, and improve on the weaknesses and build on the strengths ... Hopefully he will change his mind."
That doesn't sound likely. "When you fundamentally don't trust the people you are working with, why would you work with them?" asks Young. "These people have not given us any reason to trust them. They are not credible."
Shares