Measuring The Benefits Of The Gentoo Approach

An anonymous reader writes "We're constantly hearing how the source based nature of the Gentoo distro makes better use of your hardware, but no-one seems to have really tested it. What kind of gains are involved over distros which use binary packaging? The article is here."

Interesting, but I have to wonder.

[Enahs]

I have to wonder what the reviewers consider to be a "default" install. For example, did the reviewers remember to build in support for their IDE controller (if that's what they use)? If so, is DMA enabled for the Gentoo box, and is it for the others? What kernel did they use? Did they use gentoo-sources or did they use another?

Maybe to the uninitiated this seems informative, but to me it doesn't.

Re:Misses the point

[ecchi_0]

Finally, what's with measuring compile times? How is that a fair way of measuring performance? Hey, look, my distcc + ccache + lots of CPUs system with gcc3.2 can compile stuff faster than your single CPU gcc2 system... It's like comparing chalk and oranges

Except in this case they all had the same hardware on each machine...

The Mindcraft method, against itself

[lavalyn]

Besides, before doing any comparisons on Debian vs. Gentoo they should have compared Gentoo vs. Gentoo on different optimizations. Like using -O2, -Osize, -mfp-math=sse. Comparing video drivers. Trying different filesystem types. And a whole gaggle of other configurables at compile-time.

You'd be yelling bloody murder if Microsoft sponsored a study without doing this sort of research before pitting Windows vs. Linux.

Makes an excellent point

[JanneM]

There are a lot of issues one can bring up with the test - not identical versions of various software; different X drivers, one distro will have patches missing in the others and so on. Clearly, that greatly influences the results.

And that is a good point to take home. Optimizing compiles is _not_ the panacea for speed and responsiveness that - a minority, I believe - of source-based distros tend to bring up. There are so many other factors intimately involved in it that any benefits are generally lost in the noise.

For some specific components, it can be a good idea - but for those, most distros tend to ship several optimized versions that the installer chooses between at installation time.

Another domain that benefits are specialized, compute-intensive applications; things like simulators or other technical stuff. But then, those apps are generally tweaked and compiled by their users no matter what distro is used anyway.

Re:Misses the point

[aboyce]

first, I'd love to see a distro be faster than "up2date package_name" or even "aptget package_name".

Next, they said right in the article that they used an identical copy of the kernel source on each machine, so patches shouldn't make a difference.

Finally, its not that I dont agree with you, their tests did have flaws, it just seems that some of your facts are wrong in attacking them. There are some points that need to be examined, even if some of their conclusions are premature.

Re:Misses the point

[scotch]

Why the hell would you introduce a distributed computing tool in a discussion about evaluating the performance of a single machine/OS? Pure obfuscation. Typical gentoo-missing-the-point behavior. Either compiling the kernel is a fair measure of the speed of the system or it isn't. distcc doesn't play into it. Here's a fun analogy. You want to see which is faster a porche 911 or a chevy corvette. So some thoughtful guys put together a series of tests, one of which is a 1000 mile race. Then along comes user keesh (202812) who says "Bad test, I wouldn't drive 1000 miles, I would take the train."

A hearty helping of wtf is in order. Some of your other points are ok, though ;).

Re:Remember the KDE mandrake/gentoo fiasco?

[Anonymous Coward]

This (I think) Is because Mandrake runs it's X server with a priority of -10 (high priority). Gentoo, by default, runs X at priority 0 (normal priority).

Unfair test

[periscope]

There seems to be little attention given to the fundamental unfairness of this test presented.

The distributions were running with different software versions initially and although this was corrected there seems to have been little consideration given to the minor tweaks given to each different installation used. Which services were running on each system? Were the kernel settings identical in use? Were the machines experiencing differences in performance due to the X setup causing X to add different loads?

etc.

Fundamentally this test was probably not complete enough to suggest anything in particular. Perhaps it would have been better to boot a single machine three times and perform the sequence of events exactly the same each time as this would have also ruled out some other potential factors.

Jon.

Catch 22

[Alethes]

It seems that the people that would benefit the most from a source-based distro and optimizing binaries specifically for their hardware are the ones with the slow hardware that will take too much time to get everything installed for it to be a worthwhile investment of time.

Re:Slow?

[antiMStroll]

....these users are power desktop users and not everyone could wait 3 days for X to build.

Desktop power users on 386's are a rare breed nowadays. I don't know how long it took X to build on my P2 366 w/ 192 meg RAM because I started it before going to bed and it was done in the morning. Maybe these power users should consider hardware before choosing distros.

And so what I saw gentoo suddenly doing was having a lot of pre-complied binaries ....

Lots? OpenOffice has a precompiled option, Opera does because it isn't open-source, but which other packages do you mean? To the best of my knowledge almost all Gentoo packages are still source-based and more than a few have new options to pull the newest source directly from the CVS tree. Gentoo is expanding into more source-based options, not pre-compiles.

None of this is to say that Debian isn't a better developer's platform, or faster than Gentoo. I wouldn't know, but your post is very misleading.

Why I like Gentoo

[SpineZ]

I've tinkered with various distributions over the years and the main reason I like Gentoo is the package management system, Portage. It is by far the best package mgmt system I've ever used in a Linux environment. The thought of RPMs frighten me.

It's hard to say if it feels more optimized than say RH 9 or the latest Mandrake, but it's perfect for my needs as a desktop system. I am able to play Warcraft III under Transgaming's WineX without a hitch.

Also, with the install process and actually compiling from sources, you will learn more about linux and how it works in the first week than in the first few months with other distros - my opinion/personal experience of course.

Sites to help you get started:
Gentoo Installation Guide for x86 [gentoo.org]
Optmized CFLAGS for compiler [freehackers.org] - Be wary of Pentium 4 options. See forums for more information.
Gentoo Forums [gentoo.org] - arguably the best linux support site on the 'net

Re:Misses the point

[Vellmont]

I think you're probbably right about the reasons for using Gentoo (though I admit I've never actually used Gentoo).

However, I think that a kernel compile _is_ a fair measure of overall system performance. It involves lots of disk, memory, and processor access, so it's a decent indicator of across the board performance.

As far as kernel compile versions go, from the article:

The same 2.4.21 source was copied to all machines and compiled using the same options. However, it should be noted that the Debian system used gcc 3.3.1 whilst the Mandrake and Gentoo installations used gcc 3.3.2 .

So the kernel source was the same, not Gentoo source.

You say the performance problems are because they got the CFLAGS wrong. If this is the case it only seems to underscore how easy it is to screw up optimizations with Gentoo. It's great for people that know all the proper optimizations for a particular piece of hardware, but I think the majority of people just don't know this offhand.

In any case I find it very interesting the big differences you can see in performance between distributions on the same hardware (and I'm assuming similar kernel versions).

No distro is the king for "developers"

[0x0d0a]

Yeah...my PII/266 definitely takes less than 24 hrs to build X.

That being said, I'm dubious that blowing the time on compiling your ftp server with all optimizations every time you download a new version really is a worthwhile use of time and effort.

Maybe xmame. Maybe glibc. Maybe the kernel. That's about it. Definitely not 99% of the software on the system.

I don't really think any one distro is much better than the others for development. I happen to use Red Hat, which I do plenty of development on. I have a friend that uses SuSE and Mandrake, and another that uses Gentoo and Debian. All of us write software pretty happily -- all the tools we need are packaged or easy to build for whatever system we want to use. Most folks that I've seen that dislike a particular distro just plain don't know how to use the tools on that distro. I've liked just about every one I've tried, though they all have little things that one does better than the others -- RH shouldn't have shipped gcc 2.96, SuSE should put version numbers in their RPM names (actually, I believe they do these days), SuperRescue should be updated more often.

Heck, the vast majority of Linux users, not so long ago, *were* developers. By that metric, SuSE, Debian, and RH would be the most favored developer distros, though I doubt that contains one iota of useful information. :-)

Re:Misses the point

[lspd]

It definitely is not unless they were using unpatched sources in all three systems. The Gentoo sources applies bunches of patches to the stock kernel which would affect compile time.

RTFA. "The same 2.4.21 source was copied to all machines and compiled using the same options. However, it should be noted that the Debian system used gcc 3.3.1 whilst the Mandrake and Gentoo installations used gcc 3.3.2"

Default Install = Stage1?

[aSiTiC]

I didn't see anywhere in the story if the Gentoo installation was done from scratch stage1 or from stage3. I would think this would be a very important piece of information to mention.

The Things that Really Matter in a distro

[0x0d0a]

It's not worth it. Moving from distro to distro for performance is pretty ridiculous.

Here's what I'd consider, since this is where the biggest differences lie:

* How frequently are new releases announced? Frequent new releases may be better for hobbyists, but a pain in the ass for servers sitting in a back room somewhere. (It's the reason RH can see an enterprise edition that's simply not released as frequently).

* How do you like the packaging system? Try out apt, emerge, up2date (actually, don't -- up2date truly sucks. Everyone using RH who cares about automatic updating has long since started using apt or (IMHO, better) yum).

* How do you like the config system? Most vendors have their own interface to let you configure the system. RH used to use linuxconf, and is now using Redhat-config. SuSE uses yast.

* How much do you care about commercial support? A few widely used distros tend to get the only commercial support. Mandrake gets a little, but if you're going to be running packages that require support (especially binary-only), you're probably best off with Red Hat.

* Which desktop environment do you want to use? Mandrake puts more work into KDE on their system, Red Hat into GNOME.

Arguments about speed or features is really pretty meaningless -- common software is generally packaged for most of these, and rare software for none (use checkinstall to *make* packages -- you'll be much happier). It's still Linux with the GNU suite present.

People that switch from distro to distro (or maintain *multiple* distros on their machine) are nuts, IMHO. It's a fair amount of work to relearn the quirks of each

Those cflags are nowhere near optimized

[SiliconJesus101]

Looks to me like they compiled using pretty much generic flags. As an exaple, my cflags are as follows: CFLAGS="-march=pentium4 -O3 -pipe -fomit-frame-pointer -fprefetch-loop-arrays -falign-functions=4 -funroll-loops -ffast-math -fforce-addr -mmmx -msse -msse2 -mfpmath=sse,387" Also, we may have to question their ability to properly compile a kernel. Possibly they did a generic kernel with everything but the kitchen sink thrown in. The idea of Gentoo is that you compile the kernel for YOUR system and not generically.

I beg to differ subjectively

[marienf]

Although I never actually *measured* anything, I have been moving all my boxen (except for one Duron on which I have found it quite impossible to compile Gentoo) to Gentoo 1.4rc4. I was actually in the process of building my own compile-in-place GNU/Linux called "Q-Gnu/Linux" when I discovered Gentoo did it all, and did it better. I was all RedHat before that (going so far as to wear a red fedora on parties - I have two of those). I find Gentoo as opposed to RedHat quite impressive, at least. My professional workhorse (on which I'm currently typing) is a Toshiba Satellite Pro 4300:

model name : Celeron (Coppermine)
stepping : 3
cpu MHz : 597.077
cache size : 128 KB

..with 384MB RAM.. and was becoming annoyingly slow in things requiring major GUI complexity, like OpenOffice, and at compiling many Java classes.

Compiling Gentoo on there allowed the machine a third chance at life, the second one being when I got it (already old then) and installed RedHat on it, over that would-be-OS it came with. It just feels that much faster again. I am no longer annoyed by it at all. It took more than 4 days to compile all I wanted from the Gentoo 1.4rc4, but it was *well* worth it.

I moved my personal little server, an Athlon Thunderbird, with the same impression. Currently running

emerge system

on my brand new Athlon XP 2600, expecting much from it.

Bottom line: Nothing but Kudos for Gentoo, wondering what went wrong during the tests described, or whether somehow the subjective speedups I have experienced are just auto-suggestion. I think not. I have been staring at CRT's since 1980, thats 23 years folks! And I tell ya compiling stuff yourself is worth it. So if you have time on your side, go for LFS [linuxfromscratch.org], which I did, and slowly ground into Q-GNU/Linux. If you have some time, but not *that* much time, go for Gentoo [gentoo.org], if you have no time, you poor shmuck, either get a life, or install SuSe :-), and pretend.. :-) :-)..

Re:right now

[0x0d0a]

If you invest a lot of time in learning a distro, you're terrified that it might not be the best, and will spend ridiculous amounts of time insulting the others.

Hence, the distro wars.

This article is flawed...

[eWarz]

They optimized Gentoo for the p3 platform? Celeron 1.4 ghz and above is based on the p4 core.

Re:Misses the point

[arkanes]

You're right, I don't, and neither does anyone else. Even in your case, memory IO is likely to be as much of a bottleneck as the CPU, if not more. One of the reasons the new Mac walks over PCs in Photoshop benchmarks is massive memory bandwidth.

Moore's Law

[nagora]

The time to install from source halves every 18 months. Already, entry-level systems can compile a Gentoo desktop system up from stage1 to everything-except-OO in a day (and OO can be installed as a binary) and a server can be up and running in a couple of hours so compile time is not a big deal and gets less so every day.

Given how much better Portage is than any of the other management systems, I'd say Redhat is going to suffer big loses at the hands of Gentoo (Debian would too but the effect will be drowned out by the damage Debian is doing to Debian).

So far I've converted six machines to Gentoo, all from Redhat because I couldn't face upgrading with RPMs anymore.

TWW

Re:Mod this up

[archen]

"I'm a real Linux user because I don't use Redhat or Mandrake"

I think that additude is more because those distro's tend to hide a lot of the fundamental parts of really administering a Linux box. Mainly because there seems to be a growing ammount of people who can use Linux, and perhaps even get around okay, but have never even compiled/installed from source before.

In some ways it is an elitist additude, and in some ways it's sort of a valid concern. It's also a problem because there's a lot of nice software out there with no binary form (rssh as one example). Not that it's hard to check an MD5, and read a readme file - but some have never done it and you wonder what other areas they're probably lacking in understanding as well. Like a windows admin who's never used regedit - using it may not be neccesary on a regular basis, but not knowing how to use it is probably shifty at best.

Re:Diminishing returns

[Rhone]

So when does the time taken to compile the app with extra optimizations exceed the time you save on tasks performed in that app?

Does it matter when you're letting stuff compile overnight, or while you're at work/school/whatever? Or in the background during times when you're doing something that isn't CPU-intensive?

I mean, it's not like Gentoo users sit in front of their computers twiddling their thumbs, just waiting for a compile to finish before they can move on with their lives.

And pentium 4

[r6144]

I have heard that P4 runs code compiled for i386/P2/P3 rather slowly compared to code compiled for it. For example, P4 runs traditional (stack-based) FPU instructions rather slowly, while it prefers SSE/SSE2-based instructions. Therefore on P4 systems compiling with the right options may well give a significant speed boost.

However, AFAIK PPro/P2/P3/Athlon runs these "legacy" code quite well, so relatively little gain can come from compiler option tweakings.

Re:Misses the point

[Deadplant]

CPUs still aren't fast enough for me.
maybe I'm not representative of 'most users'

at work I do video processing and at home i play games and encode DVDs to mpeg4.
Even 2.4ghz cpus take hours to encode entire movies... I can get a little better than realtime encoding to mpeg4, but when you add two-pass, and the fact that the videos are so damn long... I wish i could get a terahertz CPU...

you try running a 2 pass encode on a 6 hour 720x480 DV video file and then tell me your CPU is fast enough.

it always makes me chuckle when people say that 'this or that' part of the PC is as fast as it needs to be.
There all sorts of things we can't do because our systems aren't nearly fast enough.

Define 'same'

[The Monster]

Except in this case they all had the same hardware on each machine...

That is frankly impossible. Even though the machines were supposed to be identical:

Upon testing with hdparm, it was apparent that this machine was having troubles setting above udma2. Eventually this problem was traced to the HD cable, a salutary lesson in the

variability of identical hardware setups.

This is just the difference they caught. Who knows how many other subltle variations exist between nominally identical machines? An honest attempt to determine how fast 3 distros do the same thing would be to really use the same hardware, by running the tests on one or more machines with one distro, then wiping the HDs and installing the second and repeating the tests, then on to the third.

The only way to have the same hardware is to use the same machine for each distro. Period.

Re:right now

[_Knots]

Of course, some realize that distros are good at different things and don't (seriously) insult any of them.

I use Gentoo on my two main boxes, simply because I toyed with it and haven't found a good reason to switch away yet (two just so I have the same environment on my two main machines). That said, my router runs Debian-testing. I maintain Mandrake machines at work, know my way around a RedHat machine... etc. My 486 ran Slack 8 while it was up. My DECStation runs NetBSD (though that's a different issue entirely).

I catch flack for using Gentoo, RedHat, Mandrake... my point is, it doesn't matter at all. Use the right tool for the job - that's what open source, and moreso just diversity in the computing environment, is good for! As long as everything speaks TCP+UDP/IP, who cares? ^^

--Knots;

ravings about debian

[Anonymous Coward]

"Debian would too but the effect will be drowned out by the damage Debian is doing to Debian"

Speaking as a Debian Developer, and Gentoo user (i know, sacrilege and all that).

Debian has a very conflicted nature, it evolves in a mob metality, any changes must first pass the gauntlet of debian-devel-ml where fire and brimstone await.

IF you make it through the gauntlet and have the support of the cabal (sshh, they dont really exist) and ALL the individuals who your changes effect, onlythen do you have a chance of getting the change through.

In the short to medium term it means things a more likely to remain the way they are, which is good if things are already running well.

In the long term, it means debian is doomed, ever hear of the expression "evolve or die" ?

Gentoo's USE settings are the best packaging inovation since apt, however its too radical to make it into debian in the forseeable future.

Debian installer is way better than Gentoo's, but still debian installer only narrowly made it through the gauntlet and much of what they are doing is shrinking debian without violating their precious policy. (they earnt a backdoor key)

Debian is structured in bazzar style, but many bazzar dwellers are trying to modify their stand into a cathedral, these are the elitists who also control the gauntlet.

Debian doesnt have much competition, there arent many non-profit distirbutions at all, i beleive gentoo is listed as non-profit but only exists to make money for drobbins, look at zynot for the explanation.

Whats needed to fix debian is for some real competition to emerge in the same space as debian (and no, mandrake and redhat are not competition to debian). Only true competition will prompt debian to get its head out of its arse.

Enough Already...

[khyronmaetel]

Ok, I'm a gentoo user. I'll admit a sizable percentage of our ranks dont know what they're talking about, i'll even admit that most distro "speed" is in the users head. But most of you are missing the point. Many gentoo users (including myself) installed gentoo as an ongoing learning experience. Sure, there's really no difference between the "l337ness" of typing emerge foobar and typing rpm -ivh foobar. But those of us who have taken the time to understand the portage system have learned a great deal. As an aspiring programmer, this was my distro of choice because it enabled me to learn about gcc. Also, i like the idea of (although most install standard packages) being able to beta test bleeding edge applications. While there are a lot of phoney gentoo users who are under the impression that theyre furthering the opensource movement by emerging packages, gentoo's backbone a highly active community of volunteers who are really interested in Open Source. Basically, all i'm trying to say is that any idiot can probably get gentoo installed and working, but the real point is to understand the OS that you've built, and i've found that gentoo helps me and others do this better than package based distros.

Re:Misses the point

[GlassHeart]

it looks like the guys doing the testing got their CFLAGS wrong. Gentoo's performance should never be worse than Mandrake

Makes you wonder how many Gentoo users actually get their compiler flags right, doesn't it?

Learning

[xRelisH]

I must say, that Gentoo is a great Distro for actually learning some of the things of the Linux environment.
Gentoo was/is my second distro (I tried redhat first), I was intrigued by the idea of doing everything from almost scratch.
Starting from Stage 1, I was able to learn quite a few things that I probably would not have learnt till some time if I had not decided to do a Gentoo Stage 1 setup as you're somewhat forced to learn the basics and have a fairly good foundation to go learn more stuff yourself.
Although I am sure there are other distros out there (including linux from scratch, if that's even technically a distro ) that are like Gentoo, but I find it to be a rather good Distro.
However though, I may decide some time from now to try the "from scratch" route, and perhaps learn even more. Although, I think the think that will keep me with Gentoo, like others have said is Portage, I think it would be great if there was a well maintained Portage-like system for any distro, as a piece of software, even though it's probably possible to get portage working on other distros, but probably not without some major tinkering.

Valid tests, not likely; Gentoo's real calling.

[mindmaster064]

While it really isn't likely that Gentoo is more than 10% faster than other systems in any given operation there are reasons I don't feel these numbers reflect anything.

First, the tests fail to mention kernel versions they're running or the filesystem of the machines (which has a big impact on processing efficiency, different fs being better or worse at different things) and they didn't even mention what version of X11 or gnome they're even running on these boxes. Are we testing Gentoo? Gentoo is portage, gentoo-sources (and the other modified kernels), ebuilds, prelink, evms, XFS, and a host of other features the others simply don't have.

Second, no comparisions are made on i/o bound tasks (reading/writing/copying files). Things that do not stretch resources in linux may simply be preempted out of importance. (Linux puts higher priority on a soon-to-be-overstuffed disk buffer than the display/computation of a spreadsheets). It would be hard for ANY of these distros to even compete on i/o bound tasks mostly because of Gentoo's EVMS or XFS options. EVMS is simply the best Software RAID I have ever used. XFS seems to be one of the filesystems of the future as well due to it's i/o guarantees, journalling, and sickly-disgustingly-large filesizes.(I feel the jobs XFS will do simply cannot be done by anything else at this point).

Thirdly, another advantage is simply not having 3-4 disks worth of iso install that you will NEVER need. Mandrake and Redhat (though not tested here) are completely bloatware. They don't install what you need and work properly, they install EVERYTHING or work poorly (because the distro makers hide some silly dependencies). Gentoo's "emerge -p packagename" is a whole hell of a lot friendlier. But, if you don't use the deep options it's possible to lose a lot of the "roll your own" advantages. Gentoo is subject to a high degree of user stupidity, it is possible to negate all of the advantages of source-based distro simply by being careless.

Lastly, Gentoo's major advantage -- it's best.. the Gentoo users! That's right. Gentoo's forums are so good (and full of know-how types) that put virtually any goofy problem you can think of is within searching distance. I simply have found Gentoo's support forums probably the best repository of linux information on the net. I don't say that lightly either. Something to be said about being able to get problems resolved quickly and correctly -- and having it at your fingertips. I like to go to one place for answers, and likely you do to.

As you can see, it's very hard to get a good feel for what Gentoo is by reading that review. I would encourage any techie freak to take her for a test drive. It's not for everyone and not even for most, but it's still a very different experience in comparison to other linux distributions. (I have used Redhat, Mandrake, and switched to Gentoo -- I have not had any need or thought of moving to something else; my own 2 cents).



-Mind

Re:Misses the point

[DrXym]

And it's binary patches which make it such a pain in the arse to keep Linux up to date. Really, everytime someone finds an exploit in glibc or the kernel which involves a few line patch, everyone is expected to download a 20Mb package!

That's fine if you happen to be broadband but sucks if you don't. In an ideal world, everyone would be motivated to waste the hours to grab the update, but how many bother, especially with the workings of Linux becoming more opaque and the users less knowledgable? The consequence is lot more unpatched Linux machines with all that entails.

Linux desperately needs binary patch support for this reason. I wonder how hard it might be to do. If someone has RH8 - i.e. a known configuration with RPMs of a certain version, it should be possible to produce incremental patches from that baseline that were a mere fraction of a full download. Instead of a 20Mb download, users would be faced with patches measuring in a few hundred k. Instead of hours to download, we're talking minutes at most. It even opens the possibilty to have an automatic download (and installation) functionality built into Linux, much like that found in MS Windows.

But MS Windows only uses automatic updates for small hotfixes for the same reason that service packs are humungus. How long before MS produce incremental patches for their OS? Here is the chance for Linux to take the lead for once.