Catch up on stories from the past week (and beyond) at the Slashdot story archive


Forgot your password?
Microsoft Software Linux Apache

Red Hat/Apache Slower Than Windows Server 2003? 628

phantomfive writes "In a recent test by a company called Veritest, Windows 2003 web server performs up to 300% higher throughput than Red Hat Linux running with Apache. Veritest used webbench to do there testing. Since the test was commisioned by Microsoft, is this just more FUD from a company with a long history? Or are the results valid this time? The study can be found here."
This discussion has been archived. No new comments can be posted.

Red Hat/Apache Slower Than Windows Server 2003?

Comments Filter:
  • by PsychicX ( 866028 ) on Saturday May 07, 2005 @02:22AM (#12460472)
    10%? 15%? Those are numbers I'd believe. But THREE HUNDRED PERCENT? I like Microsoft, and I like when somebody defends them. But this is just bull.
  • Let's see. A test commissioned by Microsoft says IIS is faster than Apache. The link for more information goes to Is this really "news"? Seems more like a thinly-disguised press release...
  • by bloodbob ( 584601 ) on Saturday May 07, 2005 @02:27AM (#12460498)
    Notice the total lack of the CGI script?
  • by Umbral Blot ( 737704 ) on Saturday May 07, 2005 @02:31AM (#12460517) Homepage
    What possibly possessed them to publish these results. No one in their right mind is going to believe 300% is an accurate figure under fair testing conditions.
  • by PaulQuinn ( 171592 ) on Saturday May 07, 2005 @02:33AM (#12460530)
    Why couldn't IIS be faster than Apache?
    Is Apache/Linux the "end-all-be-all, there is nothing that can be better so let's stop trying" type of quality?
    Are the guys who work at Microsoft a bunch of idiots that anyone can out-program?

    I'm sure IIS is better at some things, maybe more things, maybe less.

    Who cares! I don't think stats like these are why anyone chooses Apache/Linux over IIS/Windows.
  • by PsychicX ( 866028 ) on Saturday May 07, 2005 @02:35AM (#12460540)

    I just wish, just ONCE that somebody would do a fair evaluation, without an agenda to forward. But I guess that'll never happen. We all have bias...but surely we could at least attempt to get above that?
  • Re:One question... (Score:2, Insightful)

    by Nos. ( 179609 ) <> on Saturday May 07, 2005 @02:37AM (#12460549) Homepage
    Of course not, but that's not the point. Typically the guys who make the money decisions will read the headlines, and glaze over the rest, probably missing details like this study was paid for by Microsoft. Ever have to hand in a project proposal or such? I've done many, rarely does anything other than the Executive Summary get read. The rest is just there to make the document look good. Microsoft has a very big marketing department. They know this kind of stuff. Do you really think Microsoft would pay for these "studies" if they didn't show a positive return on investment?
  • by august sun ( 799030 ) on Saturday May 07, 2005 @02:38AM (#12460554)
    Can we please for once be mature about it and look at their methodology objectively? I'll even grant that because it was commisioned by MS a little extra scrutiny is certainly due; but summarily discarding the study simply for this reason is the intellectual equivalent of sticking our fingers in our ears and screaming "lalalalalala" at the top of our lungs.
  • by rokzy ( 687636 ) on Saturday May 07, 2005 @02:38AM (#12460556)
    wrong, it does tell us which is faster - linux. if Windows was faster, why would they need to benchmark against a crippled system?

    sure there's a chance I'm wrong, but for me weighing the CHANCE of better performance from Windows against the CERTAINTY that they have lied about their product (or been completely incompetant) is a no-brainer.

    and that's not considering costs (remember guys, using linux always requires an old, slow mainframe to be factored into the TOC!)
  • by Coryoth ( 254751 ) on Saturday May 07, 2005 @02:38AM (#12460558) Homepage Journal
    Under fairer circumstances, who knows, IIS might have still won, but this rigged benchmark has nothing to offer us in deciding which server is faster.

    I've reached the point where I completely ignore all the studies and benchmarks like this, from both sides. It is, quite simply, far too easy to set the constraints and metrics up so as to make sure you come out ahead. What's worse, it has become absolutely standard practice to do so. Studies have become completely useless because you can guarantee that they've been cooked one way or another.

  • by HairyCanary ( 688865 ) on Saturday May 07, 2005 @02:41AM (#12460571)
    It very well could be. However, let's try 1) an indepedent test, paid for by neither competitor, and 2) the most recent version of IIS against the most recent version of Apache, and 3) the most recent version of Windows against the most recent version of Linux. I can guarantee a win in any test so long as I am allowed to dictate all of the conditions. I wonder how many combinations they tried before they found one that IIS6 could beat?
  • by team99parody ( 880782 ) on Saturday May 07, 2005 @02:47AM (#12460596) Homepage
    At least they're up-front about it these days.

    I think they realized that the CXOs and other execs who make the big decisions never read the fine print anyway; and such disclaimers will never make the headline or a large-font pull-quote in any such marketing literature, so there's no harm in being up-front about it.

    Even fake grass-roots efforts [] (astroturfing) can be done openly these days.

    Oh, and to get on the Team99-bloggers-good side, I just wanted to say that Longhorn is so stunningly awesomely good that Microsoft won't have to resort to this kind of silly FUD once longhorn is released.

  • What about Norton? (Score:3, Insightful)

    by qualico ( 731143 ) <worldcouchsurfer&gmail,com> on Saturday May 07, 2005 @02:51AM (#12460615) Journal
    Wonder what that benchmark would be if you installed the FULL Norton package on it?

    This bull reminds me of those advertisements for weight loss.

    Stick stomach out....Suck stomach in
    No cosmetics..........New facial
    Front shot...............Side shot
    Grubby clothes........New fashions

  • by Pinefresh ( 866806 ) <william DOT simpson AT gmail DOT com> on Saturday May 07, 2005 @02:55AM (#12460632)
    Well, if you really want to know, you could probably do one. It couldn't be too hard to put a simple one togeather, and it would solve the question for you.
  • by cranos ( 592602 ) on Saturday May 07, 2005 @03:06AM (#12460674) Homepage Journal
    Um okay I did the "mature" thing and checked out the report. The report is two years old and compares an RC version of w2k3/IIS6 against an old version of Redhat AS/Apache thus rendering it completely useless for doing an evaluation today. Not only that but it neglects to compare against other linux distributions such as SUSE or Mandrake thus rendering the "Windows better than Linux" claims deceptive at best.

  • by Anonymous Coward on Saturday May 07, 2005 @03:06AM (#12460681)
    I like Microsoft, and I like when somebody defends them.

    I've been in IT for about 17 years. I've seen MS destroy "the little guy" time and time again, with thier power and yet with all that power, money and developer base, deliver garbage year after year, to this day.

    Then I compare them with offerings like Mac OS X, the BSD's and Linux and wonder, how on Earth someone can say, "I like Microsoft".

    Seriously now, what is there to like about them?
  • by grcumb ( 781340 ) on Saturday May 07, 2005 @03:17AM (#12460730) Homepage Journal

    People keep saying, 'When are we going to get a real benchmark?" Well, why don't we roll our own? Seriously.

    Here's my idea:

    Slashdot has strong zealot^H^H^H^H^H^Hsupporters for both Microsoft and Linux. Let's have a contest to select the best qualified from each side, have them work in teams on identical hardware. Let them make any changes, tweaks or optimisations they can dream up. Then, let 'em rip.

    I'm dead serious about this, by the way. Let's get off this endless roundabout and for once make a clear comparison.

    For bonus points, once the first contest is finished, we should take the two servers, leave them exposed to the Internet and see which one gets 0wned first. 8^)

  • by jmv ( 93421 ) on Saturday May 07, 2005 @03:18AM (#12460734) Homepage
    The thing with benchmarks is that when they're made by an organisation you can trust, you don't really have to dig the details (and there are always some details you won't see). If I have to dig through everything, I might as well do the benchmark myself! Now, looking at a benchmark sponsored by Microsoft is like reading a study on climate written by an oil company, a study on health by a tobacco company... or even a Linux-Windows benchmark done by RedHat (although I trust RH a bit more than MS).

    The only benchmark by MS which I might trust is one saying Windows is slower and/or worse than Linux. Somehow, I never saw any of those.
  • So, for that piece of crap, you say "a *lot* of stuff he says is 'true'" (emphasis mine), and the best you offer is that people using appellations like "Micro$oft" burns your ass? Come on. Maybe it's the "penis envy" part? Or the part about "Setting up a server in Windows takes a couple of minutes"? Sorry, I see a bunch of wanking and sour-graping, combined with outright lies, and a few pointed pokes at the Linux Zealot community - which I don't credit with any 'hit points' because this is obviously from someone who would like to be a Windows Zealot, but isn't competent... or maybe it's a Linux Zealot making Windows Zealots look like idiots?
  • May not be FUD (Score:2, Insightful)

    by Sivar ( 316343 ) <`moc.liamg]' `ta' `[snrubnselrahc'> on Saturday May 07, 2005 @03:44AM (#12460811)
    I actually thought that this was common knowledge--that Windows Server 2003 with ASPX was faster than Linux/Apache with PHP, or that Server 2003 was generally faster with static content. (I admit, I only glanced over the article, and Adobe Acrobat's search tool is the worst of crap, so sue me if it didn't mention ASPX).

    1)ASP (not ASPX) are fairly flaky and recent versions are roughly comparable to, but slower than, PHP4 (not sure about 5), in general.

    2) Windows is not very good at creating new processes quickly. This is why CGI (not fastCGI) in the platform is so glacially slow.
    Let's have an example. Let's say that you make a dynamic webpage in which all content is generated by a C++ CGI program. Ignoring database access for the time being, since that dilutes the example, on Windows, the website would be MUCH slower than the same website written in ASPX, even though the actual execution time of the C++ program is shorter (assuming a competent C++ coder).
    This is because for each request, Windows must create a new process (the CGI program), and destroy the process when the request is complete.
    While the execution time is low, the process management overhead dwarfs the actual page runtime, because Windows doesn't do that sort of thing quickly. This is why CGI has long been blacklistedon Windows systems by good web devs, and this is one reason that Apache 1.x was such a dog on Windows. Apache 1.x creates a new Apache process for each request.

    Now Linux, on the other hand, creates processes about as fast as it creates threads, which is to say, really damn fast. Apache 1 has always worked just fine on Linux (and indeed most Unix systems) because the overhead of creating a process, while significant, isn't slower than a dead slug stuck in frozen molasses like it is on Windows.
    Apache 2.x allows requests to be served by a thread or a process, or a number of processes that each create several threads (any Apache gurus please correct me if any of this is off).
    It follows that this isn't a big deal on Linux (because process creation isn't really much slower than thread creation), but is a very big deal on Windows.

    Windows has ASPX, which is Microsoft's marketing term for the use of the .NET framework for web content delivery (get it--the 'X' makes it sound cool. Or something). .NET is compiled, and ASPX needs neither process nor thread creation. Like any .NET application, ASPX can run sort of close to native speeds (native + lots of wrapper overhead + generic memory management overhead and such.)

    Yet Apache is still back here creating a process or thread for each and every request (note that there are some ways to speed things up. FastCGI comes to mind, but I don't want to get into the gory details that I don't know enough about). This is not the brightest way to do it in terms of performance, but then, Apache appears to have been designed for universality and configurability over raw throughput.

    It is unwise to hold the attitude that Apache can't be beaten by IIS, especially when IIS is optimized for one platform--by the vendor of that platform. Apache isn't even the fastest on Linux. Take a look at Zeus [] webserver. It serves circles around Apache on any platform it supports--including Penguin land.
    In fact, Zeus uses a technique called SendFile() which, oddly enough [], is strikingly similar Microsoft's own TransmitFile() API. Hmm.

    Think of it this way: Apache is to IIS as GCC is to ICC, at least in terms of performance and generality.
    Intel's compiler (ICC) consistantly blows away GCC in terms of the performance and size of the compiled code, but GCC runs on just about anything with a CPU, can cross-compile, is free, doesn't pull any PHB evil tricks [], and actually compiles things like the Linux kernel without pat
  • by Fefe ( 6964 ) on Saturday May 07, 2005 @03:55AM (#12460841) Homepage
    It's ridiculous how the Slashdot crowd is falling victim to Pavlov again.

    If someone publishes a benchmark about your software, and finds out your software does not perform well, don't whine, don't behave like a child, don't start kicking and screaming, don't tear his hair out. Behave professionally.

    Good starting points:

    • Does their test setup matter?
    • Can their number possibly be true?
    • What weak spots about the competition does their test reveal?
    • What can we do to improve the results?

    Let me summarize what I think about their test. First of all, I believe their numbers. Apache sucks performance-wise, in particular if you run a busy site with dynamic content. That's why people are using squid in local accelerator mode before Apache. This is a good indication that some performance tuning is in order. But no, people rather wait for Microsoft to find out and then they start thinking about fixing it.

    If this test was meant to be unfair FUD, they would not have tested TUX, just Apache.

    But now to my questions above:

    Question 1: is their setup relevant?

    No. Sites who answer more than 5000 requests per second are not using a single web server, they are using a load balancer and a cluster.

    Question 2: Can their numbers possibly be true?

    The point I find least believable is that IIS had better CGI performance than Apache. Creating a process is really slow on Windows. Their result should be independently verified.

    Question 3: What weak spots about the competition does their test reveal?

    They did not test a single-CPU webserver (which is what almost everyone is using).

    They did not test FastCGI or APAPI dynamic web pages.

    So if we wanted to do a more balanced review, we would look at these.

    Question 4: What can we do to improve the results.

    Document APAPI better, I'd say. Almost nobody is writing their dynamic web page modules with APAPI.
    Everyone is using PHP or mod_perl. Benchmark Apache in real-world scenarios. Document best practices.
  • > 1. You rejuvenate and dance when you hear a windows flaw exposed, but you conveniently ignore the thousands of security flaws exposed in linux.

    "Rejuvenate" means "renew, appear to grow younger". Did you mean "become jubilant"?

    I don't become jubilant when anybody's security flaw is exposed. In the case of Open Source apps, patches are generally available in a couple of days.

    > 2. You yell loudly TROLL! at any person's post or at any person you see posting facts that you do not want to hear about your oh so cool linux.

    No, just the ones that misstate the facts or are attempts at FUD.

    > 3. You know it's a classic case of penis envy, you don't have all the support, software and hardware available for linux and you have to let that anger out somewhere, but you don't have the brains to admit it.

    Um, Linux supports all my hardware just great.

    > 4. You hate windows, hate Microsoft, but race to emulate windows, have programs to run office from within linux, and spend a $300 on a Windows emulator, only Windows fools.

    > I run Linux, Windows, and Solaris machines. I use and so have no need for Microsoft Office. But if I did, I could run it using WINE, which I can get for free. Unlike MS Office.

    > 5. You cannot admit that you don't have professional usage of Linux outside server markets.

    I use Linux *professionally* on the desktop.

    > 6. You cannot admit that most of the joe user out there when told that there is linux will respond, what is that?

    Sounds like there's a need for some consciousness-raising, then. Alothugh I've noticed that more and more people -- even Joe Sixpack types -- don't go glassy-eyed when Linux is mentioned these days.

    > 7. You cannot admit that there is no professional printing capabilities in linux.

    I don't have any problems printing from Linux.

    > 8. You cannot admit that you are a masochist (otherwise why would someone spend hours playing with scripts, and recompiling programs that are available for Windows?)

    Well, it did take me about 30 seconds to learn how to type "./configure - make - make install - make clean". Or if I'm feeling lazy, I can just double-click an RPM file icon in Konqueror.

    > 9. You cannot admit that there is no professional desktop publishing done on Linux.

    Sorry, mate, you're talking to someone who does just that for a living.

    > 10. You cannot admit that no one in their right mind would do professional video editing in Linux.

    I honestly don't know about that. But I do know that lots of movies' special effects are being generated these days using Linux-powered render farms.

    > 11. You cannot admit that linux sucks when it comes for gaming/home entertainment or education.

    There are tonnes of educational apps available for Linux -- many of them come with commercial distros. There are still more on the Net. As for games -- if I want to play games, I'll buy an X-Box.

    > 12. You have problems in understanding Windows, and you will blame your own incompetence on Microsoft.

    Over the years, I've used and administered Windows 3.1/95/98/Me/2000 and have no problems doing so. But after just 6 months, I can install, configure, and administer a Linux machine faster and more reliably.

    > 13. You have problems in pointing a clicking, but have no problems in wading through cryptic scripts written by lunatics.

    Pointing and clicking has its place. But there are lots of things that are actually easier via a command line. For instance, I'd much rather run a MySQL server that way than use the GUI tools. Nice thing about Linux and Open Source apps in general is that you've a choice in the matter. If you don't like the command line, don't use the bloody thing.

    > 14. Nothing will get past that shit that fills your head, you will not admit to any facts.

    Can't respond to an assertion that's semantically nil, sorry.

    > 15. You can't admit that naming of linux components, packages, and others are weird and fits profiles of troubled teenagers. gentoo, lgx, rpm ....

    See #14.

    > 16. You feel angered because you were left out by microsoft's Media technologies, they support Mac, Sun sparc, but not linux.

    I get along fine with Linux media technologies, thanks.

    > 17. You feel inferior deep inside but unable to admit it, you don't have a database as easy and powerful as Access.

    ROFLMAO. You shouldn't have said that within my hearing, mate. I work for a company that produces an Open Source database that's easier to use and more powerful than Access, and unlike Access it's (a) available for about 15 different platforms and (b) free as in speech and (c) free as in beer. (BTW, I am MS-certified on Access and so I *do* know what I'm talking about, thank you very much.)

    > 18. You cannot tell that not a single office package outside Microsoft's is worth looking at or bothering with.

    I use professionally every single day. I make my living with it. I much prefer being dependent on it than I ever was being dependent on MS Office.

    > 19. You don't know that your CD recorder software sucks.

    I can burn a CD or DVD about 3 times faster and 5 times more reliably with k3b on Linux than with Nero on Windows. (Exact same hardware -- I originally had Win2K Server on this box before switching to SuSE 9.2 about 6 months ago.)

    > 20. You don't have DVD-RAM, DVD-R, DVD-RW support in your pathetic OS.

    I beg your pardon? I have a DVD-R/W/R+W that's fully functional under a stock SuSE 9.2 installation. "It just works."

    > 21. While the rest of the world moves on, you're stuck in a stone age technology that needs third party software to boot into GUI.

    > 22. You act out of prejudice, you kill file domains and users of specific news readers while you ignore the bullshit that your fellow linux losers post.


    > 23. You don't know commercial support in Linux is almost non existent.

    Buy a commercial distro, get commercial support. If you need it. So far, I've not needed it.

    > 24. You miss the fact that companies are leaving linux because of the chaos, and the cheap linux losers who are unwilling to pay and support hard work, Corel, gaming companies,...etc.

    Companies *leaving* Linux? Which orifice did you pull that one from? And many Linux users contribute their own hard work in making improvements or helping other users rather than throw away money on MSCEs.

    > 25. You are unaware that linux has no terminal services (there is a lame one that no one uses), and commercial support for it is not happening.

    Let's see... there's telnet, ssh, rlogin, VNC... I use VNC between my Linux and Windows machines all the time (in both directions), wasn't a bit of trouble to set it up, either.

    > 26. You are unaware that setting up servers on Windows takes couple of minutes while on linux, good luck playing with configuration scripts.

    Time it took me to install Windows on this machine (2.4 GHz Celeron/1GB RAM) and configure it as a Web, DB, and file server: about 3 hours. Time to install Linux on this machine and configure it as Web, DB, and file server: about 90 minutes.

    > 27. You cannot admit that support for USB on linux is laughable at best.

    I have no trouble with USB devices on Linux. Plug it in, Linux recognises it, I use it. "It just works."

    > 28. You think that Linux is better because slashdot told you so.

    I think Linux is better because it's more reliable, more flexible, and cheaper.

    > 29. You spend countless hours flaming people because they post their opinions about your oh so cool linux and your attitude, instead of researching things for yourself and understanding fact in order not to look this stupid.

    I do my research on Linux by using it every day. How do you do yours?

    > 30. You think that anyone who uses linux has a clue.

    Not necessarily. Just like I don't automatically assume that someone who uses Windows hasn't any clue.

    > 31. You think that linux cannot crash.

    Hasn't happened to me yet. Worst I've had to put up with in 6 months is that a couple of times something's made KDE hang, and I've had to ssh from another box to kill the runaway process. (Usually, it's Acrobat that's the culprit.) Can you do that with a Windows machine? No, you can't -- when the Windows GUI hangs, your only alternative is the power button.

    > 32. You think that everyone is interested in your conspiracy theories about Microsoft (or should i say M$ in order for you, teenagers to understand?), and how they destroyed linux, ...etc.

    Does Microsoft attempt to shut out competition illegally? Definitely, that's been shown in court. Microsoft "destroy" Linux? Not happening so far as I can tell.

    > 33. You keep ignoring the fact that thousands of linux servers get hacked every year, but it takes one Windows server hacked to get you and your fellow linux idiots to dance and celebrate.

    It's not Linux servers getting trojaned and ruining the Net for the rest of us. It's Windows desktops.
  • by xiando ( 770382 ) on Saturday May 07, 2005 @04:15AM (#12460890) Homepage Journal
    I personally do not trust someone who claims to be "Veritest is an independent testing agnecy authorized by Microsoft to carry out the testing for applications developed on windows platform." to do a fair evalution of Linux vs Windows. If a company who makes a product gives you a huge pile of money at regular intervals and you are asked to compare that product to another product, who are you going to vote for? Who is your daddy? Sadly, money is everything.
  • Re:This is new? (Score:2, Insightful)

    by 51mon ( 566265 ) <> on Saturday May 07, 2005 @04:20AM (#12460899) Homepage
    The answer to "why now" is on the Veritest web page.

    The paper is hilarious if you actually read it.

    Key points...

    They looked for tips on optimising Apache on the Redhat website ?!(guys next time try

    They found that Apache 2 was 50% faster than Apache 1 (without any tuning, Apache 2 offers a selection of threading models so a fair comparion would have tried each in turn, with or without tuning), so presumably didn't do any further tests with that in case it made MS look bad.

    They tuned Windows for the server but effectively plead ignorance of how to tune Redhat for the server.

    My guess is even then, on this hardware, sensible tuning of the kind they did to Windows would have made Apache comparable or better.

    There were issues with this hardware selection at the time. Driving Gigabit ethernet is pretty demanding stuff, and you need drivers that can handle interrupt load.

    However anyone who actually needs an 8 CPU machine to serve gigabit websites will probably do their own benchmarking and tuning, in the unlikely event their application software gives them a free choice of platform.

    When my employers online business is big enough to need gigabit hosting, we'll probably still have SQUID on Linux with shed loads of memory accelerating the static content, because hardware is cheaper than rewriting the corporate applications.
  • by xiando ( 770382 ) on Saturday May 07, 2005 @04:21AM (#12460902) Homepage Journal
    First of all, performance is in many cases the main issue when buying a system. If you have a huge load then the choice may be between buying one well-performing system or two or three systems to take that load. And when you say "price is" I assume you mean the retail price, which is to me almost totally irrelevant (at least in the long run). The real price for any kind of software is the cost of teaching humans how to use it, costs related to maintaining it and most importantly, costs related to downtime and lost productivity if the software is slow. Performance DOES matter.
  • Re:Easy (Score:5, Insightful)

    by dtfinch ( 661405 ) * on Saturday May 07, 2005 @04:29AM (#12460919) Journal
    IIS was heavily tweaked for this benchmark. But for most uses, both IIS and apache are fast enough out of the box.
  • by zoefff ( 61970 ) on Saturday May 07, 2005 @04:45AM (#12460972)
    Do it like this:
    -invite MS and RH (or other) for the contest, but ask what software versions to buy off the shelf and what to install. Install it yourself and have them tweak it with the configuration, without having extra's installed.
    -Let them agree between each other on as many things as possible: hardware, time to tweak, do's and don'ts, type of tests
    -Compile several benchmark tests: load, response time, static, dynamic. Ofcourse all on the same type of hardware platform.
    -Run the tests with them witnessing and report here!

    One note: Make sure to get the experts on both sides. At MS not a problem, but the linux arena is much bigger.
  • by iamacat ( 583406 ) on Saturday May 07, 2005 @04:53AM (#12460992)
    Well, I wouldn't use them for purchasing decisions, but this kind of studies are sure useful for pointing out weak spots in your favorite product. Time for Apache hackers get busy and fix such embarrassing performance scenarios.

    I don't think Apache is the right server for static pages and simple CGIs though. It has so many modules and settings that the code path from filesystem to socket has to be much longer than necessary and longer than the feature-limited competition. They should try a simple server like Boa.
  • by Arimus ( 198136 ) on Saturday May 07, 2005 @05:10AM (#12461061)
    We'll be reasonable when companies carrying out these kind of performance sutdies compare apples with apples (and ideally with no hint of GM involved):

    1. Use identical hardware...
    2. Use the default un-optimized settings...
    3. Hand tune using experts on the software under test...
    4. Rerun the identical tests...
    5. Ensure that clients used to test server software are identically configured.

    That would be being reasonable...

  • by zobier ( 585066 ) <> on Saturday May 07, 2005 @05:26AM (#12461102)
    What someone should do in these kind of tests is get an expert Windows team and an expert GNU/Linux team, identical servers and let them configure them as best as they can. That seems fair.
  • by Xabraxas ( 654195 ) on Saturday May 07, 2005 @05:44AM (#12461146)
    For hardware, windows support is vastly more comprehensive and complete than linux... though linux has better support of hardware more than a year old, as things improve...

    Hardware support maybe more complete on the x86 platform but that's it. Linux has far superior hardware support over all.

    would like to see more effort towards binary compatability in the kernel to support binary drivers a bit more consistantly though.

    There is a reason that binary compatibility doesn't exist in proprietary drivers. Linux is a free system and was never intended to support 3rd party, proprietary drivers.

    IMHO BSD is probably a better OS option than Linux is in a lot of ways (free/open/net)... though linux has the fame, glory, and fanatic following. I like windows, I use windows...

    I see this posted all the time but It just leads me to beleive the poster has never used Linux or BSD. It seems like people are using this line to try and ward of criticism by saying "look, I like free software, just not Linux", or "I like Unix, just not Linux".

  • by inode_buddha ( 576844 ) on Saturday May 07, 2005 @06:12AM (#12461210) Journal
    How is an "independent agency" "authorized"? I would *love* to see those NDA's.
  • by cofaboy ( 718205 ) on Saturday May 07, 2005 @06:17AM (#12461220)
    Don't forget that MS changed the EULA so that you are no longer allowed to benchmark windows and MS products without written permission. The only commercial people who can benchmark are those who will use a framework defined by MS.

    Any other options will mean no study and no money.

    He who pays the piper calls the tune.

  • The red flag (Score:3, Insightful)

    by jav1231 ( 539129 ) on Saturday May 07, 2005 @08:03AM (#12461466)
    The red flag here is "300%." I don't think anyone can take it seriously with such a large desparity. That's like two hybrid car makers competing and the salesman tells a customer "Yeah, they get 50mpg but ours gets a bagillion-zillion!"
  • Almost (Score:2, Insightful)

    by Eternally optimistic ( 822953 ) on Saturday May 07, 2005 @08:33AM (#12461537)
    If you require identical hardware, some might complain that it's more suited to a particular system. Give them a fixed amount of money for the server. Or a fixed amount of other resources that might be the bottleneck (power, floorspace, maintenance time/month,...)
  • by tacocat ( 527354 ) <> on Saturday May 07, 2005 @08:50AM (#12461586)

    You don't have to load any of those modules if you don't want to. Thereby simplifying the code path...

    That's why they choose to make them modules rather than baking them into the application itself.

  • by Anonymous Coward on Saturday May 07, 2005 @09:06AM (#12461631)
    A lot of people like to complain about Microsoft, especially Windows. They usually think Linux is 'cool', but then when they try it and see how much doesn't work (eg half the hardware in my laptop) without installing a plethora of patches from various sources, and find out that some of it doesn't work as well as it does under Windows, they come to the conclusion that Windows isn't really so bad after all.

    In my own experience, there's also a big difference based on the usage. For example, my servers almost never go down unless there's a power outage (no matter which OS they're running). Desktops tend to have more stability problems (mainly with video-intensive stuff like games, where a dodgy video driver can cause no end of problems), and laptops have all the problems of desktops, plus more power-management issues. Overall, I help friends troubleshoot Windows crashes from time to time, and they're invariably caused by dodgy, unsigned device drivers (usually for hardware that doesn't work at all under any other OS), not bugs or flaws in the OS itself. (For the uninformed, a dodgy driver will crash UNIX, Linux or BSD just as easily as Windows.)

    There are plenty of good things about a lot of other Microsoft software, like Word and Excel. I used to be somewhat hostile towards Microsoft (because of my UNIX background), but despite my best efforts, I could never find a spreadsheet competitive with Excel. I still prefer typesetting tools common on UNIX to Word, but the fact that copy/paste works pretty well on Windows is a huge benefit.

    These days, I generally just use what best gets the job done (ie I'm not inherently pro- or anti- anything), and that means I mostly use MS Windows and MS Office for day-to-day work, and MS .NET for GUI programming. For non-GUI programming, I still prefer the UNIX command line and GCC, but I don't really do much programming at all any more.

    At the end of the day, Microsoft software is just like any other: it has strengths and weaknesses. In some cases, it's the best tool for the job, and in others it isn't. Anyone who can't see that must have either (a) very limited experience, or (b) an irrational bias against Microsoft.
  • by Anonymous Coward on Saturday May 07, 2005 @09:38AM (#12461737)
    Of course, the budget has to buy both software and hardware.

    After all, we want to count TCO and performance, right? :-)
  • by eno2001 ( 527078 ) on Saturday May 07, 2005 @10:14AM (#12461902) Homepage Journal
    Actually what is needed is a public, non-profit benchmark competition. Both Windows and Linux enthusiasts are welcome to join in. Limit the contest to 100 teams of up to ten people. The 100 teams are all suprvised by the people who run the contest. The contest itself should make no money of any kind in order to keep away any monetary incentive. Hardware donations from the big players are acceptable with the understanding that the hardware will be returned after the compeition. In this way, the ugly little trait called "competition" gets in without any monetary incentive. At that point it's enthusiasts trying to outdo each other on both platforms. With this set up, you really get to test the performance of both OSes in a fair way because enthusiasts are likley to know all the tricks to get their OS and application to perform best. This means you'll likely see Windows outperforming a typical Windows system and Linux outperforming a typical RedHat/Mandrake/Debian/Gentoo/SuSE Linux system. Sounds like fun. SO who wants to get this party started? :)
  • by Bobzibub ( 20561 ) on Saturday May 07, 2005 @10:25AM (#12461957)
    I never understood why this is not illegal. Is preventing benchmarks not a restriction on trade?
    Imagine if the automobile industry did it?
  • by NickFortune ( 613926 ) on Saturday May 07, 2005 @10:42AM (#12462045) Homepage Journal
    The best tool for the job? Well sometimes, maybe. Perhaps. But look - a spanner is a tool, right? Well in the case of MS we're talking about spanners that we can't buy but only licence. Spanners that MS can take back from us any time, despite the fact we paid good money. We're talking spanners that have been designed to stop working if we move to a house with a bigger garage.

    A lot of people would say "if you don't like that, don't use MS spanners". Fine. Done deal! :)

    Just a little way down the metaphorical road, there's a shop that sells spanners at a fraction of the price that MS does. They may not be as pretty, and for some jobs they aren't quite as exact - but they've been getting better for years and the difference is scarcely noticeable these days. And if you can do without the fancy packaging, you can go online and get that same tool free..

    And it's then yours to use legally, wherever and however you wish - so long as you don't try and claim you designed it.

    So the question is: by what criteria do you evaluate best? None free software, security holes, forced upgrades... with many people these things carry a hefty negative.

  • by NickFortune ( 613926 ) on Saturday May 07, 2005 @10:58AM (#12462111) Homepage Journal
    Does the test setup matter? Apparently Veritest thought so. They spent some time tweaking both machines. It seems like the tweaks sped up windows and slowed down linux.

    I have to applaud the way you take a positive stance and look at how apache can be improved. I expect efforts in that direction form an ongoing part of apache development, but the positive attitude is appreciated. It's just a bit sad that your post reads as an endorsemnt of a blatant piece of paid-for propaganda

  • by rben ( 542324 ) on Saturday May 07, 2005 @11:38AM (#12462295) Homepage
    Sadly, money is everything.

    Not so. If it were, there would be far less support for Open Source projects. Fortunately, as FOSS has demonstrated, large numbers of human beings are quite capable of being motivated by interesting problems and the knowledge that thier work will benefit everyone else.

    Be cynical if you like, but every day you use Linux or Open Office; every day you see a website served by Apache; know that it's because some people value contribution to society enough to donate their time and creative energies.

  • by ChrisCampbell47 ( 181542 ) on Saturday May 07, 2005 @11:51AM (#12462343)
    A real competition would give 2 (or more) teams identical budgets, not identical hardware. They would need to PAY all consultants at market rates. No free consultation from Microsoft (or Red Hat) R+D allowed, unless they are paid, and paid at market rates.

    The budget has to buy software, hardware and setup labor.

    This eliminates the problem of "that hardware favors Microsoft" or "that team had better engineers". It all comes down to money and value.

    Of course the competition would need to state up front exactly how performance would be measured and how the various different tasks (static pages, cgi, etc.) would be weighted to come up with any overall scores. That would dictate the design choices made by each team.

  • by Jester998 ( 156179 ) on Saturday May 07, 2005 @11:56AM (#12462365) Homepage
    Surprisingly (controversially?) enough, some EULAs forbid public criticism - I wonder if such clauses would ever be found valid in court, I seriously hope not - judges should declare void in whole any EULA that includes any anticonstitutional demands.

    Ever hear of a non-disclosure agreement? Are those 'unconstitutional' in your fairyland world too?

    The US Constitution only affects dealings between the government and private citizens, it has nothing to do with dealings between two private citizens. But then again, this is Slashdot. Carry on with your groundless bitching.
  • by o517375 ( 314601 ) on Saturday May 07, 2005 @12:03PM (#12462384)
    Or maybe a soon-to-be-Google-employee
  • by magadass ( 234992 ) on Saturday May 07, 2005 @12:04PM (#12462389) Homepage
    Usually unless they otherwise state so, benchmarks are done with the out-of-box settings, you can also do several things to enhance IIS performance, so I am not sure if your analysis is really valid. Although I do have to say IIS 6 and IIS 5.1 are nearly completely different, microsoft really took a leap forward when they designed IIS 6, so its interesting.

    As to the Microsoft sponsored benchmark, Im not 100% convinced that if you pay for a benchmark its going to be biased, unless you black-mail the company doing the benchmark, I would like to think that the company holding the benchmark would be lookin out for there name also, and not skew results to make them look bad.

    But thats what I would like to think, to much past with how microsoft has handled these things, but of course microsoft isnt the only one guilty of this, intel is just as guilty as well as AMD, Nvidia, ATI, etc.

    If you had a business would you like your benchmarks skewed a bit to benefit your company? I'm not sure if I would or not, money is tempting but technology doesnt advance for money it advances for competition.

Q: How many IBM CPU's does it take to execute a job? A: Four; three to hold it down, and one to rip its head off.