On April 13, a test lab named Mindcraft released a report commissioned by Microsoft that declared Windows NT to be far superior in performance to Linux. NT, trumpeted the study from Mindcraft, an "independent testing lab," is "2.5 times
faster than Linux as a file server and 3.7 times faster as a Web server."
Numbers like that can get Linux lovers hopping mad. But what really has the Linux community incensed is the way Mindcraft apparently went about seeking information and help from Linux experts.
Sure, Mindcraft -- either itself or through intermediaries -- did ask for help
in an Internet-based discussion forum. But it did so under false pretences:
It didn't tell its would-be helpers that it needed the information because it was comparing the performance of Windows NT and Linux at the commercial request of Microsoft. And its request for advice originated from a computer with a
Microsoft address. It refused to
answer any requests for more information about its test. Finally, it apparently promised to redo the test with the input of Linus Torvalds himself -- but isn't giving him the info he needs to do the job right.
The original news of Mindcraft's test sent the Linux world into immediate convulsions -- the sound of
open-source geeks gnashing their teeth could be heard all over the Internet. Were all those claims for Linux's
technical superiority just so much overheated hooey?
Not so fast, said the Net. As demonstrated in a recent report by Linux front man Eric Raymond, if there is one thing that the
Internet is really, really good at, it is marshaling the combined forces of
thousands and thousands of watchful eyes.
In combination with some
aggressive trade press reporting, numerous details undermining the Mindcraft report soon emerged. First, the study had been commissioned by
Microsoft -- a fact not mentioned in the original press release, and buried deep within the Mindcraft Web site. Second, the test had been able to take advantage of extensive
"tuning" information from Microsoft -- details on how to make NT work
especially well on the particular hardware configuration used by
Mindcraft. And third, Mindcraft's attempts to obtain similar tuning
information from the Linux community were half-hearted, at best -- and here is where the story gets interesting.
The Linux community swears
by its accessibility, by its willingness to help answer questions in a
variety of forums. But it isn't perfect. The convoluted story of Mindcraft's
attempt to get information from one of the most popular such forums --
Usenet newsgroups -- illustrates a key Linux weakness, the lack of
centralized sources of information on the newest and most advanced technical issues.
But at the same time, the story
underlines the essential worthlessness of commercially sponsored
comparison tests. The purpose of these tests is to please the customer who
commissions them. Why expend too much energy attempting to find information
that your customer probably won't appreciate?
The trail starts with a post to Usenet on March 11 by a correspondent
known only as "Will." The post -- which was cross-posted to
the newsgroups comp.os.linux.setup, comp.os.linux.networking and
comp.infosystems.www.servers.unix -- asked for information on how to maximize
performance from a Linux-Apache system on an extremely high-performance, multiprocessor hardware setup. And it received one lengthy post offering technical advice and asking for more details -- which were not forthcoming.
Nearly a month later, a perceptive Usenetter noticed that the hardware of
that setup matched exactly what Mindcraft had been using for its tests.
And, most intriguingly to the conspiracy minded, the
"message header" for the Usenet post from "Will" included information
indicating that the message had been originally sent from a computer on
Microsoft's internal network.
I sent "Will" an e-mail asking if the situation he had described in
his Usenet post indeed referred to the Mindcraft test configuration. I
also asked him why he had chosen not to identify himself.
The correspondent acknowledged that he was acting as an "intermediary" between Mindcraft and Usenet. As to his reluctance to come clean, he wrote:
"Any form of post comparing NT vs Linux, or by @mindcraft.com or
@microsoft.com would have yielded nothing but flaming insults (I've seen it
all too often). I've personally been running Linux for some time as well as
NT, and regardless of my personal opinion, it is appallingly clear that
Linux fans are deeply attached to their OS on an emotional level akin to Mac
users. So much so that communications often degrade to mindless sarcasm and
insults."
"Will" then noted that Mindcraft was in the process of "rerunning these
tests (and others) with the direct aid and assistance (albeit remotely) of
the top minds in the Linux community."
By consensus, the two "top minds" of the Linux community are Linus Torvalds and Alan Cox. Cox and Torvalds acknowledged that
they had been in communication with Bruce Weiner, the president of
Mindcraft, but both expressed dismay at the extent to which they were being
allowed to participate.
"I've traded a couple of e-mails with Mindcraft people about this," says Alan
Cox. "They seem solely intent on trying to re-create their existing
pro-Microsoft results and hoping, by attaching some kind of 'Linux top mind'
credibility to it, they can do more damage."
"The whole thing has been fairly painful," says Torvalds. "Mostly because
these people don't actually let us know what the hell they are doing. We've
been offering to be on site to see what the hell is going on, but so far
they've refused."
"There isn't that much we can do -- they don't actually allow anybody access
to the dang thing, so while they have e-mailed us about their problems, we
can only tell them that the results they are seeing are extremely low, and
we can only guess at the reason why," continues Torvalds. "Experiments on
similar (but not the same) hardware has shown much better performance in
other places."
"Will" refused to provide his full name or employer, although he denied that he
is a Microsoft employee. He explained the issue of his mail originating
from a Microsoft mail server by saying that he has "friends at Microsoft"
who "owe me favors." One such favor, he said, was to host his machine --
which, he noted, runs Red Hat Linux.
Getting up-to-date information about Linux
configuration issues for cutting-edge hardware is a serious issue for the
Linux community. But even if Mindcraft had identified itself, right off the
bat, to Usenet, or Red Hat, or Torvalds himself, it's not entirely clear
that the community could have come up with answers that would have
significantly improved Linux's performance.
Mindcraft did not return repeated phone calls and e-mail queries,
but at least one source indicates that the company is tiny, perhaps no more
than two or three employees. Mindcraft's business depends on satisfying
customers like Microsoft who request specific numbers on specific hardware and software combinations. As one engineer familiar with the testing business notes,
"The choice of what to test and which results to publish are always
driven by the customer's interests ... There's plenty of money in performance
testing, as long as it's possible to produce credible numbers that are
favorable to the customer."
Credible or incredible? The numbers certainly are favorable. One wonders,
though, whether the fallout from this episode will be exactly the
opposite of what Microsoft might have wished. The Linux world is suddenly
highly motivated to accomplish two new challenges -- centralizing access to
"tuning information" and upgrading Linux performance on the kind of advanced
hardware that hasn't been Linux's priority. Stay tuned.
Shares