Gentoo On Server Considered Harmful

Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."

This article makes good points.

At the same time, the "your system is always approaching the bleeding edge" way of doing things solves one problem that I've always been bothered by with running user servers for suso.org [suso.org]. Eventually, the OS on the server reaches the age where it is no longer supported and updates are no longer coming out for it. This isn't always X years where X is the number of years that a distribution claims to provide package updates for. Its usually X-1. This is because you'd be foolish to use the very latest hasn't been available for more than a day version of Linux. Usually you wait for 6-12 months for it to be mature and have special packages of whatever available for it. Then you spend another month or two setting up the machine and getting it ready for production. By that time, you've already burned over a year of support time. Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left. Upgrading such a system to the latest version of whatever distro means taking the server down for several hours to upgrade, hope to hell that special packages you've built and configurations aren't broken and in nightmare situations, roll back because something is broken and can't be fixed.

The promise of Gentoo for me is being able to continually upgrade and never get outside of that window of support.

I actually have a new shared user system that is running Gentoo that is kinda in beta right now. This article was very useful for me because it brings up those points about stability that concern me. Its kinda an experiment.

I think I may try Debian next.

Re:This article makes good points.

Gentoo has proven troublesome in a production environment.
The problem isnt updating often... it's when you DONT update often.

We had one system which we didnt bother to update. (Dont fix what isnt broken)
Then one day we had to upgrade some of the services.. which in turn required lots of libraries to be upgraded.

In the end, we had to upgrade kernel.. cause libraries didnt support 2.4 kernel.
Stuff change too much in gentoo to put it simple.. It's easier to keep updating often

emerge sync && emerge -u world
Then iron out all config-changes. Find out which undocumented features were changed, which keys to add to startup script etc.

Lesson learnt: Dont use gentoo on production systems. Run it on your desktop computer you play around with...

Re:This article makes good points.

Then one day we had to upgrade some of the services.. which in turn required lots of libraries to be upgraded.

In the end, we had to upgrade kernel.. cause libraries didnt support 2.4 kernel.
Stuff change too much in gentoo

How is it Gentoo's fault that the services you run require updated libraries? How is it Gentoo's fault that the libraries you use require a 2.6 kernel?

Seems to me the blame lies with the services and the libraries respectively, and performing the same upgrade would require the same kernel update on other distros too.

Re:This article makes good points.

In the case of Red Hat, they'll backport changes for you so that you don't need to upgrade 50 other packages in order to get a security patch for Apache to work.

So in a way, yes, it is Gentoo's fault. It's just the way the distro is designed. Everything at the latest revisions possible. Great for a home system, not good for a server you have to maintain.

Re:This article makes good points.

It's not. The issue here is not which distro is better than the other in some very personal sense, it's whether or not it makes sense to update all the time. I personally feel that, yes, gentoo does require lots of time to update constantly, but it's meant for a park of desktops, not specifically servers, or else you'd better have a number of machines you have a servers + 1 to run updates and then just use packages compiled on your external machine.
Yes new patches come out all the time, but the real question is whether you trust developers to improve their code over time, or to destroy it. We've seen one end of the spectrum with what MS did between 98 and ME, and I believe that gentoo shows us the other end. While you theoretically always ARE at the bleeding edge with Gentoo, it does have a "safe window" built in, the way it handles portage with the keyword system. New packages are usually in CVS within 48 hours of release. If they compile and run, they get thrown into the ~arch (testing) rapidly. Then, depending on what kind of update has been done on it, you'll have to wait anywhere from 2 days to 5 months to see it come down into the actual arch repository, which is deemed the "stable" gentoo. I personally run ~arch, yet I can't seem to recall a problem that portage couldn't solve with minimum input on my part.
Yes, I'm a gentoo fanboy, but I'm not so glued down into distro patriotism to refuse to see flaws where they are.
Some people seem to want to spend time in maintenance to keep a system up to date and continually tinker and let their knowledge grow by frequent maintenance, and other people seem more interested in setting something up and being lazy about having to deal with updates/upgrades. I personally trust that most open source coders, and especially the ones for the big projects like apache, ssh, mysql and others of that caliber, usually improve the code from release to release, not damage it. Security fixes, bug fixes, and plain new features are usually the goal of coders, and I trust that they do that.

Re:Bukd your own binaries

Yeah, but you have to admit ..... that's getting on for as much effort as it takes to be a Distributor!

If you want something that you know isn't going to change much, and certainly never in a way that breaks anything, use Debian Stable -- and be prepared to build the odd package from source {it really isn't as bad as it's made out to be} if you have to have a massively up-to-date version of something. They have a more-than-King-size package repository.

Re:This article makes good points.

Lesson learnt: Dont use gentoo on production systems.

I would see that lesson instead as don't experiment on your production systems. Obsolete hardware is useful for testing out stuff like this.

The reason I don't run gentoo on production systems is simply becuase I am not familiar enough with it and it is different enough from other distributions of linux and other versions of *nix to make things confusing. It's the same reason I don't use reiserfs - if it all messes up how can I or any moderately skilled linux user get things back into operation?

Re:This article makes good points.

The joy of portage all the way. Continuous upgrade versus release cycles. 15 years of dealing with both have convinced me that portage is good only in two places:

• your own workstation where you want to look at how the bleeding edge looks before unleashing it on the unsuspecting user population. This one is updated continuously and rebuild as necessary.
• single special purpose dedicated servers (not run of the mill 10+ servers with same load). You build these once and after that you leave them alone until they die (preferrably).

Everything in between - forget it. Update hell, dll hell, etc. If you use portage (either the BSD or Gentoo incarnation) you die and releases are the exact and only solution to this. You can stamp servers with a "released" OS out of workshop by truckload and you can be more or less confident that updates will not break a lot of things. The only problem is upgrades to next release but if you are using The One OS to rule them all [debian.org] even that is not a problem.

Re:This article makes good points.

In an average company you need 2-3 packages at most that need to be pushed to newer versions. If you need to maintain locally more than 3 packages for infrastructure (and you are not making a living out of it) you are doing something seriously wrong. The most likely reason is the magpie syndrom (love for all things new and shiny). Time to stand back, look at what are you doing and think: "Do I really need all these shiny latest superduper things or I can make with a verified version and a well known workaround".

If you are dealing with 2-3 packages you can do that by using backports.org or backporting yourself. If you need more and these are an essential part of the business there is no difference between portage and backporting/local packaging. In ether case they have a tendency to break and you need local developer/sysadmin time allocated to that. Portage gives you no advantage whatsoever because the resource you gain in keeping more than 3-4 packages synced to their projects HEADs you will lose in infrastructure upgrade creep. Every time I have looked at this in the past taking out the numbers out of the ticketing and workflow control systems have proven that this is the case. I have yet to see one case where this is not.

Re:Not if you're using Debian

Which is exactly the way I like my infrastructure. 3-6 months freeze with all bugs known, worked around or fixed in the meantime. Once I have gotten it to this point I build on top of that for the actual services which can run something very bleeding edge if necessary, but this is as I pointed out "your daily bread". For the stuff that is not, you need to be sure that it works and if you are a manager to be severely anal about it. So debian stable + 2-3 unavoidable backports and local builds is about right. This is also the reason corporations buy RedHat ES/AS/WS like hot bread. They finally see a model where the base has been frozen long enough to be relied on for building your own services.

Many itadmins and most developers have a problem with understanding of the "establish a platform and build on it" and "platform freeze before development" ideas. They think that everything is a fair game and the results (in man hours wasted on piecing everything together for release) are usually quite obvious.

Re:This article makes good points.

Gentoo has proven all right in my production enviroment - and that is an ISP.
First reason, is that you don't have to upgrade those production machines all that often. I sit down and read any security advisory that seems to affect me. And, not surprisngly, there are actually very few remote vulnerabilities that hit Gentoo-hardened. Furthermore, those tend to be in software right in a leaf of the dependency tree, or software I might consider disabling (or limiting to trusted hosts) to the next maintainance cycle.
And there comes it - once in 6 months a massive emerge -uDB world && emerge -uDk world && revdep-rebuild && perl-cleaner (better don't omit the latter two). The system is nicely trimmed down and the build runs on a few machines I have available, so it doesn't take any epic amounts of time. In fact, I even seen it done within half an hour. Still, back when it did take a better part of the day, I simply run the first command a day earlier and then used the packages, what of course is a breeze.
Finally comes the configuration updating. I haven't seen it easier anywhere. The first nice thing is that Gentoo developers don't toy around them - they usualy come as the original software developers intended. But what really makes a difference is the toolchain. By far, I have seen no other distro that automagicaly within the standard package system uses revision control for configs. And then, it gets the trivial updates done for me, and puts me into vimdiff anytime any decision is required.
At most times, this means no downtime at all, as everything runs smoothly. In case of a kernel upgrade, or anything going wrong (once till now), we still have redundancy. So there are no visible drawbacks of using Gentoo on those servers... Unless I, and my boss, am missing something.

Redhat 6.2

Don't fix it if it ain't broke: up 292 days, 22:26 The reason for the short uptime, is PSU upgrades...

Re:Redhat 6.2

Are you kidding? This reminds me of the old adage about the 20 year old broom. 7 new handles and 12 new brushes and it still cleans as well as it did when it was new. :)

I had an OpenBSD/postfix box

That would have had around 900 days uptime if my reboot-happy Windows-only-admin coworkers wouldn't have reset it in a panic on multiple occasions to "troubleshoot" (no it was never a problem with my OpenBSD box) mail problems.

I don't know what the hell it is with Windows-only admins and rebooting. The kind of instability that required reboots all the time was reduced drastically with Win2k and win2k3, yet that insatiable urge to reboot first and ask questions later still plauges my Windows-only counterparts.

Re:Redhat 6.2

Don't fix it if it ain't broke: up 292 days, 22:26 The reason for the short uptime, is PSU upgrades...

My Gentoo system was up 309 days [snerk.org] before I realized that the PSU fan had stopped turning and the motherboard overheated and blew 6 capacitors which is why the clock got so far out of sync (the computer thought it was April when I rebooted it back in November) which explains the graph weirdness.

Prior to that I had an uptime well over 200 days ruined by a blackout that outlasted my UPS.

I perform updates here and there on my server periodically and perform a full-scale "bleeding edge" upgrade whenever I'm forced to reboot the machine.

Re:Redhat 6.2

Dude, install NTP. Then you could have kept the machine going without those useless capacitors!

Re:This article makes good points.

The promise of Gentoo for me is being able to continually upgrade and never get outside of that window of support.
I agree. Every now and then a program's latest version doesn't agree with a config script somewhere, but that's what etc-update is for. If something borks, you can always ask the gentoo forums [gentoo.org], which is an invaluable source of information for all things gentoo. That and the gentoo-wiki [gentoo-wiki.com].

Also, no one is 'requiring' anyone to upgrade. I administer hundreds of gentoo servers and you don't always need to keep up to date to be secure. Part of the nice thing about gentoo is that you're only installing the packages you need, so if you know of a vulnerability in a script you use, you don't have to upgrade your whole portage tree just to plug a hole.

Re:This article makes good points.

but that's what etc-update is for.

dispatch-conf is an improved tool for managing configuration files.

Re:This article makes good points.

Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left.

Which is why IT Pros prefer Red Hat Linux or its unencumbered variants link CentOS, White Box, and Scientific. Better testing up front thanks to the Red Hat gang, and longer shelf life. Which is why most commercial software chooses to support it first, it provides a stable base.

Re:This article makes good points.

Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left.

What kind of dope uses Fedora on a production server?

Use CentOS - I'm running CentOS 4, and anticipate not having to do *ANYTHING* to my production systems except use them, keep them turned on, and keep them updated (which is about 5 min/week) until 2010 or so.

Re:CentOS updates

At risk of exposing my ignorance here (I'm a Debian person; the last time I did anything RedHat-based was before automatic package management), what is CentOS's automatic-update feature like? Does it have one?

Yes, it's yum.

I assume it uses yum, or something like it, being RedHat, but does it pull from RedHat's servers directly, or are there separate CentOS repositories?

CentOS Repositories

In that case, how closely do the CentOS repos track the 'official' RHEL ones, in terms of patches and bugfixes?

The official RHEL ones are publicly available, and tracked by CentOS very well. The only changes they make are for trademark requirements. Thus far it has been bug for bug compatible with RHEL.

Not that you'd probably want to do it on a true 'production' system, but can you do the CentOS equivalent of 'apt-get upgrade' and be reasonably assured of not breaking things?

Yes

I've always been intrigued with CentOS, and it does seem to have a good reputation as far as stability is concerned, but after growing up with apt-get (and before that, nightmarish experiences with dependency hell on some very early RedHat systems), I've developed a certain perhaps-unwarranted negative bias of everything else.

I prefer yum myself. I used apt when it first came out, and loved it. Since I got my first 64 bit machine I just prefer something that handles the dual architecture a little better. For the most part they're about the same though.

Re:This article makes good points.

And that my friend, is the niche Opensolaris will quickly start filling.

Re:This article makes good points.

So, you upgraded from the old 1.x branch to a radically different 2.x branch, known to be a substantial partial rewrite, and expect everything to work out ok all by magic? You also seem to failed the "sentient sys-admin test" by not using 'google' to do some research. Things like say "http://www.gentoo.org/doc/en/apache-upgrading.xml " perhaps?

I run Gentoo on my own machine, and most of my users WANT bleeding edge versions, a lot of custom options here and there. The system is using a hardened kernel, stack protection and everything is compiled for 64bit (k8). I don't know of any distros that can do that for every package. So far I have had 1 package problem, and that was resolved by 'uncaching' some stuff and redo the emerge of that package. In general, gentoo is easy to maintain, provided you update regularly. As for the people whining about compile times, this is a server, using it at 100% cpu now and then, provided the compilation has a low priority impacts noone. Compiler time is a non-issue, i'm not running X, soundcards, usb, video drivers, gui-browsers etc, there's not all that much to upgrade.

It should be noted that I sync the portage tree from a euro-mirror to a local mirror 6 times a day, and having 3-4 meg a sec to the files-repository makes downloads take an average of 2-3 seconds. Coupled with two beefy processors and lots of ram, Gentoo is brilliant for me. And yes, I have permission from the rsync-maintainer to synch that often.

Some serious crack smoking...

Gentoo allows you to be on the cutting edge, just like all the other distributions. The primary difference is it makes it very easy for those who don't know what they are doing to be there. Most folks running SuSE, RH, or one of the other 'package' based distributions won't build their own RPM, etc. There is nothing stopping one of the 'normal' distributions from upgrading the kernel with each release. I certainly don't update everything on my Gentoo box because it is there, on my server.

I run Gentoo on a server. The server is stripped down beyond what a typical 'router' distro looks like - one of the reasons I went with Gentoo is I could really trim the system down for the job at hand. My server only gets updates for security, and once in a while a bug fix that impacts the applications running on the server. Not often. When I need to compile something big, the last place I'd do it on is the server itself - it has another task. I take one of my workstations with far more GCC horsepower and let distccd [gentoo.org] do the work for the poor little pizza box. Beyond the initial build, I doubt those boxes have ever compiled anything.

Since it is a source-based distro, I also am not trapped by RPM's or other packages no longer getting provided for my system. One of the applications I had was using RH9 (with paid support) only to have them drop maintenance on it and have the vender drag their feet moving to another platform (clue stick, they had issues with the 2.6 kernel, so would not 'support' any platform but RH 8 and later 9. The enterprise editions? Forget about it... You want to live in the suck, you try keeping one of those boxes alive and secure years after it EOL.

Re:Some serious crack smoking...

You are essentially describing a Slackware system after 20 minutes of install.

And??

"I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything?"

I agree... so why does this preclude using Gentoo?

Just because you _can_ update all the time doesn't mean you should. I've used gentoo for various purposes (server, desktop, laptop). What I usually do is get it setup and install all the packages I need and then leave it for a _long_ time... only upgrading packages that I either need the new capability of or for security purposes.

Look... I personally don't think Gentoo is the best server OS out there... but I also don't think that just because the package system makes it really easy to tinker with the system that Gentoo is inherently unstable...

Friedmud

Part of "article" not quite correct.

There is no 'stable' version of Gentoo. Gentoo is rather a moving target where emerge will forever cause your system to approach the cutting edge.

Yea. Not quite. This is what the "ACCEPT_KEYWORDS=" setting in make.conf is for. If you don't have it set, you get "stable" packages. If you do have it set, you get the unstable stuff.

Further, with the use of the files in /etc/portage, you can have a stable system, but have one or more packages be unstable without having it a system-wide setting.

Haven't read the rest yet, but wanted to point that out.

Re:It's a dirty job

Servers are not the place for bleeding tech. Servers are the place for stability.

That is, unless you really dislike your customers that much, be they actual customers or other divisions in your business.

Not for me!

I certainly wouldn't want a Gentoo on my servers. Sure, it wouldn't weigh [wikipedia.org] much, but think of the poop you'd have to clean up!

I use to run Gentoo on a Personal Server

I had a colo box that ran gentoo. Then one day, a standard stable package update broke mysql [alexvalentine.org].

* MySQL DATADIR is /var/lib/mysql * Previous datadir found, it's YOUR job to change * ownership and have care of it * Sorry, plain up/downgrade between different version of MySQL is (still) * un-supported.

I vowed never to use Gentoo again, and promptly moved that machine to Debian. I use to run Gentoo on all my desktop machines in the pre-ubuntu days, because it had the most bleeding edge desktop packages and optimizations. After Ubuntu came on the seen, Gentoo had no advantage for me. Its still a great learning too though. I highly recommend for aspiring Linux geeks.

Agreed.

I have been a server admin for web/database for about 3 years now. I agree that bleeding edge is *not* where server admins want to be. There's a reason that Debian is widely considered the best server OS despite being rather far behind the bleeding edge. Tried and tested is better than the latest and greatest when you rely on the machine being up. It's also worth noting that the military doesn't use any COTS technology within 5 years of it being released.

The Problem With Gentoo...

The problem with Gentoo Linux is not the system itself, it's the stereotypes that people put against it.

Gentoo is only good for ricers, Gentoo is bleeding edge and unstable, Gentoo is only good for X deployment

The truth about Gentoo is that it is not really a distribution. Gentoo Linux does not make "releases" and it does not aim to cover one area of the market alone.

In Gentoo's packaging system, called portage, the aim is not only to provide up-to-the-minute packages (which it does) but also to provide a wide variety of both tested and verified "stable" packages as well as more bleeding-edge, testing packages.

This, along with a properly configured make.conf and /etc/portage file system, allows you to pull down the packages you want that have been verified as stable (and are also under watch by the Gentoo security project) and keep track of their libraries with revdep-rebuild.

Stop branding Gentoo with stereotypes that label it as X distribution, the project even calls itself a "metadistribution" capable of dropping into multiple roles.

Re:The Problem With Gentoo...

The problem with Gentoo is that Gentoo users assume that most people care about configuration options. They assume that people want the most up-to-date packages. They assume that there's no reason to have stable, long-term supported releases.

Huh? We assume no such thing. In fact, we really don't care what "most people" want, at all. We make no assumptions about support. It is Gentoo detractors who tend to claim that we do. We don't. What we care about is making Gentoo. If Gentoo doesn't fit your needs, don't friggin' use it! Trust me, you won't hurt our feelings. If you think Debian is better, use it. If you think Windows is better, use it. You aren't harming us in any way by using what you feel is the best tool for the job. In fact, that is exactly what we try to give to our users. We give them a set of tools to allow them to build what they want.

I think the biggest issue is that people seem to have this closed-minded view of software and Gentoo. They're stuck in this way of thinking that lends towards doing what the vendor tells you to do. They run Red Hat. They run Debian. They don't think that you can build what you want. Gentoo provides the tools to do just that. For many of my clients, I have built custom Gentoo-based distributions. What they get themselves is slightly different than Gentoo. They get pre-compiled packages. They get a very nice Internet-based update system for these packages. They don't jump into make.conf, at all. They don't need to make these kind of changes. Instead, I have built a custom distribution with the software that the customer wants on it. They install it from CD, and it has exactly what they want on it and nothing else. Gentoo is the tool that builds this system. I am using Gentoo as it was intended, to build exactly what I want. People tend to forget that it is impossible to make something that fits every need. Rather than try to do so, like other distributions do, we instead provide the tools to allow you to build it on your own. It's a completely different philosophy, which is why I understand that so many people simply don't get it.

some truth, but for many Gentoo is appropriate

First of all, I find it interesting that FreeBSD never seems to get these complaints and hate about having to recompile packages with portupgrade all the time, and being able to tweak the flags, etc. In this respect, it's just like gentoo!!! Except without a lot of the fancy features like etc-update and slots and masking and multiple supported versions. Yes, the "base system" is more stable on FreeBSD (which is both a blessing and curse), but what is it about Gentoo that attracts so many haters/inexperienced admins, hmm??

Anyway, I run Gentoo on servers. (Also FreeBSD). I think it's great. I can't stand stuff like Red Hat, which makes it difficult to customize anything, so I'd always resort to installing stuff "by hand", which was a huge pain. Or creating a custom RPM, which was an even bigger pain (RPM is basically a huge clusterfuck in general).

Being able to set up ebuild "overlays" is great. Being able to set up custom profiles that contain all the software needed for a particular app is great. Writing ebuilds is a piece of cake. Turning on/off various features system-wide is very helpful. The mechanism for merging configs (etc-update or dispatch-conf) is nice. Being able to pin down specific versions with masking is good. Etc. For the record, I've never tweaked the CFLAGS in my life.. that's just not why I use Gentoo.

The author writes this:

A profile update will touch a very large number of configuration files, and it may even alter your startup process. Obviously this is not something you want to do to any server. ................. The end result: the machine had to be resuscitated on-site with associated downtime.

I have no idea what happened to him. Updating your profile is basically moving a symlink, which changes some lists of base packages and other high-level build configuration. It doesn't "touch" anything in your system. Sure, you have to some upgrades afterwards, but you have to do that regularly anyway on Gentoo. Compare it to upgrading FreeBSD from 5.x to 6.x, which is much more involved.

As you might be aware, FreeBSD has a nice little program called portaudit........... Now, Gentoo also has something like portupgrade. What it doesn?t have is portaudit. ............ In all fairness, Gentoo has an experimental command called "glsa-check".

I've been using glsa-check for a while now, it works great. It tells me what's got known holes and I just update those packages, and their dependencies. What problem did he have with it, besides the "experimental" status? Yeah it can "do stuff", but I don't use those options, I just use it to get a list of packages with known holes. Heck I could probably write a script to do the very same thing.

Suppose you need to patch one of your installed packages by the way.. it's very easy to create custom ebuilds on Gentoo. Sometimes I plug security holes that I've found on my own for instance.

I have a simple strategy with Gentoo servers: keep an identical test/staging server nearby and do your updates on that machine first. Run your application tests and then upgrade the production machine. If you want, build binary packages on the staging machine. I would do this even with Red Hat, Debian, etc.

Another point: I've NEVER run "emerge -u world". I always do the packages in small groups or chunks and then updated configs, restarted daemons, and run tests after each one. This seems like a much better strategy than what some people do.

Also, I gotta say, it's probably not a good idea to run Gentoo on a production server unless you've got at least 5 years of Linux admin under your built. You also need to FOLLOW the Gentoo newsletter, AT LEAST, so you can get a heads-up when config files change or files are moved around. It happens from time to time.

Really, the only valid point he makes that generalizes to servers other than his own is the following: Gentoo takes more time to keep runni

Re:some truth, but for many Gentoo is appropriate

You wrote, "First of all, I find it interesting that FreeBSD never seems to get these complaints and hate about having to recompile packages with portupgrade all the time, and being able to tweak the flags, etc. In this respect, it's just like gentoo!!!."

As was pointed out in an earlier post, gentoo is a meta-distribution, whereas FreeBSD is complete operating system. Overall, the "FreeBSD experience" is significantly different from the "Gentoo experience." FreeBSD feels much more polished, and is therefore less likely to produce frustrated blog entries.

I administer Gentoo, FreeBSD, and RHEL boxes, and have several years of Solaris experience. There is a lot to like about gentoo but the final point that you acknowledge, "Gentoo takes more time to keep running," is extremely important, and worth elaborating on in a whole paragraph of its own.

It does require more time and effort to build a gentoo box in the first place; it take more time/effort to provide a secure environment (glsa-check is still in beta, for good reasons); it requires more time/effort to ensure that your dev, staging, and production environments are all in sync. Yes, it can be done, and quite elegantly, but it costs more (time == money) to do that on gentoo than using other solutions.

That is the core frustration of every negative gentoo review that I've read. The most common counter-argument to those complaints boils down to, "You just haven't spent enough time to appreciate the elegant beauty that is gentoo." Allow me to offer a counter-counter-argument.

Once upon a time, I took the time to fully appreciate the beauty that is emacs. I accepted the truism that emacs doesn't meet you halfway, that you have to go to emacs; I read books on the subject; I made it my default editor; I created a highly customized .emacs file; I got tired of pushing my customzied .emacs file, and all associated libraries, onto every new machine; my pinkies started to hurt all the time; and I noticed that when I was REALLY in a hurry I used vi. Eventually I just stopped using emacs.

I think of gentoo as the "emacs" of operating systems - really cool, but with a high pain threshold before the cool starts paying for itself.

*sigh*

The article makes it sound as if gentoo installs the ~unstable profile by default. The stable one's no more bleeding-edge than Ubuntu.