Gentoo On Server Considered Harmful 372
Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."
Some serious crack smoking... (Score:5, Interesting)
I run Gentoo on a server. The server is stripped down beyond what a typical 'router' distro looks like - one of the reasons I went with Gentoo is I could really trim the system down for the job at hand. My server only gets updates for security, and once in a while a bug fix that impacts the applications running on the server. Not often. When I need to compile something big, the last place I'd do it on is the server itself - it has another task. I take one of my workstations with far more GCC horsepower and let distccd [gentoo.org] do the work for the poor little pizza box. Beyond the initial build, I doubt those boxes have ever compiled anything.
Since it is a source-based distro, I also am not trapped by RPM's or other packages no longer getting provided for my system. One of the applications I had was using RH9 (with paid support) only to have them drop maintenance on it and have the vender drag their feet moving to another platform (clue stick, they had issues with the 2.6 kernel, so would not 'support' any platform but RH 8 and later 9. The enterprise editions? Forget about it... You want to live in the suck, you try keeping one of those boxes alive and secure years after it EOL.
I use to run Gentoo on a Personal Server (Score:4, Interesting)
* MySQL DATADIR is /var/lib/mysql
* Previous datadir found, it's YOUR job to change
* ownership and have care of it
* Sorry, plain up/downgrade between different version of MySQL is (still)
* un-supported.
I vowed never to use Gentoo again, and promptly moved that machine to Debian. I use to run Gentoo on all my desktop machines in the pre-ubuntu days, because it had the most bleeding edge desktop packages and optimizations. After Ubuntu came on the seen, Gentoo had no advantage for me. Its still a great learning too though. I highly recommend for aspiring Linux geeks.
Agreed. (Score:5, Interesting)
some truth, but for many Gentoo is appropriate (Score:5, Interesting)
First of all, I find it interesting that FreeBSD never seems to get these complaints and hate about having to recompile packages with portupgrade all the time, and being able to tweak the flags, etc. In this respect, it's just like gentoo!!! Except without a lot of the fancy features like etc-update and slots and masking and multiple supported versions. Yes, the "base system" is more stable on FreeBSD (which is both a blessing and curse), but what is it about Gentoo that attracts so many haters/inexperienced admins, hmm??
Anyway, I run Gentoo on servers. (Also FreeBSD). I think it's great. I can't stand stuff like Red Hat, which makes it difficult to customize anything, so I'd always resort to installing stuff "by hand", which was a huge pain. Or creating a custom RPM, which was an even bigger pain (RPM is basically a huge clusterfuck in general).
Being able to set up ebuild "overlays" is great. Being able to set up custom profiles that contain all the software needed for a particular app is great. Writing ebuilds is a piece of cake. Turning on/off various features system-wide is very helpful. The mechanism for merging configs (etc-update or dispatch-conf) is nice. Being able to pin down specific versions with masking is good. Etc. For the record, I've never tweaked the CFLAGS in my life.. that's just not why I use Gentoo.
The author writes this:
I have no idea what happened to him. Updating your profile is basically moving a symlink, which changes some lists of base packages and other high-level build configuration. It doesn't "touch" anything in your system. Sure, you have to some upgrades afterwards, but you have to do that regularly anyway on Gentoo. Compare it to upgrading FreeBSD from 5.x to 6.x, which is much more involved.
I've been using glsa-check for a while now, it works great. It tells me what's got known holes and I just update those packages, and their dependencies. What problem did he have with it, besides the "experimental" status? Yeah it can "do stuff", but I don't use those options, I just use it to get a list of packages with known holes. Heck I could probably write a script to do the very same thing.
Suppose you need to patch one of your installed packages by the way.. it's very easy to create custom ebuilds on Gentoo. Sometimes I plug security holes that I've found on my own for instance.
I have a simple strategy with Gentoo servers: keep an identical test/staging server nearby and do your updates on that machine first. Run your application tests and then upgrade the production machine. If you want, build binary packages on the staging machine. I would do this even with Red Hat, Debian, etc.
Another point: I've NEVER run "emerge -u world". I always do the packages in small groups or chunks and then updated configs, restarted daemons, and run tests after each one. This seems like a much better strategy than what some people do.
Also, I gotta say, it's probably not a good idea to run Gentoo on a production server unless you've got at least 5 years of Linux admin under your built. You also need to FOLLOW the Gentoo newsletter, AT LEAST, so you can get a heads-up when config files change or files are moved around. It happens from time to time.
Really, the only valid point he makes that generalizes to servers other than his own is the following: Gentoo takes more time to keep running. But you have to weigh that against the flexibility you get, just like any "build vs. buy" decision.
Re:This article makes good points. (Score:4, Interesting)
What kind of dope uses Fedora on a production server?
Use CentOS - I'm running CentOS 4, and anticipate not having to do *ANYTHING* to my production systems except use them, keep them turned on, and keep them updated (which is about 5 min/week) until 2010 or so.
Lack of support contract considered harmful (Score:4, Interesting)
Re:This article makes good points. (Score:3, Interesting)
Here is where I make myself sound like an old man talking to his children about walking through the snow both ways. I knew someone would have to make a remark like this.
I've been using RedHat and thus Fedora for 10 years now. I started out on Linux on the RedHat track. And thus I'm more familiar with it. CentOS wasn't even in diapers and there weren't many other choices. Now that there are things like CentOS, I've actually gotten tired of dealing with rpm dependency issues that Fedora/CentOS/RHEL have and don't want to use it anymore. I once had an error about something like kernel-source requires some audio library.
Cannot say I disagree. (Score:5, Interesting)
Gentoo is so system dependent compared to other distros. The end result, instead of having 1 package for some function, you have 1^n packages for that same function. Given 'n' amount of users with differing hardware and compile time arguments. The Qaulity Assurance ends at the user, always. You ultimately have a quality control department that consists of one, the user.
Any system upgrade or maintenance procedures in production environments are usually limited to a few hours at most. It does not make sense to spend six hours compiling what could have been installed, configured, and tested in 6 minutes with a pre-compiled package. In the event of a hardware failure, I find it reassuring when a Linux distro can be loaded onto a spare box in 15 minutes. Then spend a few more minutes restoring configurations from a good backup.
But that's just my opinion. To each his own. If it works for you, then go with it. Otherwise, I'd say it is a fairly level-headed review.
Re:This article makes good points. (Score:5, Interesting)
CentOS updates (Score:2, Interesting)
I assume it uses yum, or something like it, being RedHat, but does it pull from RedHat's servers directly, or are there separate CentOS repositories? I assume it's the latter. In that case, how closely do the CentOS repos track the 'official' RHEL ones, in terms of patches and bugfixes? Not that you'd probably want to do it on a true 'production' system, but can you do the CentOS equivalent of 'apt-get upgrade' and be reasonably assured of not breaking things?
I've always been intrigued with CentOS, and it does seem to have a good reputation as far as stability is concerned, but after growing up with apt-get (and before that, nightmarish experiences with dependency hell on some very early RedHat systems), I've developed a certain perhaps-unwarranted negative bias of everything else.
Re:Agreed. (Score:1, Interesting)
Actually, I was thinking of one case in particular, the USS Yorktown, which made the news in 1997 for being dead in the water because their systems crashed. It was reported that they were running Windows NT 4.0, which was released in 1996, making it a year old when the USS Yorktown incident happened.
Re:This article makes good points. (Score:5, Interesting)
I would see that lesson instead as don't experiment on your production systems. Obsolete hardware is useful for testing out stuff like this.
The reason I don't run gentoo on production systems is simply becuase I am not familiar enough with it and it is different enough from other distributions of linux and other versions of *nix to make things confusing. It's the same reason I don't use reiserfs - if it all messes up how can I or any moderately skilled linux user get things back into operation?
FreeBSD (Score:3, Interesting)
Daniel's original premise seems to have been (which I agree with) that there are some elements of FreeBSD which are highly desirable, which at the time, Linux didn't have. Ports, portaudit, portupgrade...they're all good things. Ubuntu has an equivalent of portaudit and portupgrade combined, and of course the Red Hat autoupdate was probably the first on Linux, but the difference between those and the two commands I mentioned is that the Ubuntu and Red Hat services both focus on binaries...portupgrade anywayz focuses on source, which is something that at least some of us want.
I don't advocate using source compilation all the time, or if I do, at least not during the day or when you're active...set something up to do it while you're asleep or while the system isn't being used...that way it won't bother you. To be honest also, the main reason why I advocate compiling from source is simply for the reason that if you stop doing a certain thing for long enough, the ability to do said thing when you *do* want to has a tendency to disappear. If you maintain the attitude of compiling from source when it doesn't matter, there'll still be enough people doing it that the option to do so will still be there when it *does*.
There are a lot of people out there who don't want to do anything that even vaguely resembles self-responsibility or proactivity, at least where using a computer is concerned. That's fine, but said people need to realise that the fascist nature of such things as Vista is merely the ultimate logical extension of them wanting multinational corporations to act as their wetnurse. It's been an eternal truth in politics and other areas as well as IT that freedom and proactivity genuinely go hand in hand...If you don't want one, you're not going to get the other.
Re:some truth, but for many Gentoo is appropriate (Score:4, Interesting)
As was pointed out in an earlier post, gentoo is a meta-distribution, whereas FreeBSD is complete operating system. Overall, the "FreeBSD experience" is significantly different from the "Gentoo experience." FreeBSD feels much more polished, and is therefore less likely to produce frustrated blog entries.
I administer Gentoo, FreeBSD, and RHEL boxes, and have several years of Solaris experience. There is a lot to like about gentoo but the final point that you acknowledge, "Gentoo takes more time to keep running," is extremely important, and worth elaborating on in a whole paragraph of its own.
It does require more time and effort to build a gentoo box in the first place; it take more time/effort to provide a secure environment (glsa-check is still in beta, for good reasons); it requires more time/effort to ensure that your dev, staging, and production environments are all in sync. Yes, it can be done, and quite elegantly, but it costs more (time == money) to do that on gentoo than using other solutions.
That is the core frustration of every negative gentoo review that I've read. The most common counter-argument to those complaints boils down to, "You just haven't spent enough time to appreciate the elegant beauty that is gentoo." Allow me to offer a counter-counter-argument.
Once upon a time, I took the time to fully appreciate the beauty that is emacs. I accepted the truism that emacs doesn't meet you halfway, that you have to go to emacs; I read books on the subject; I made it my default editor; I created a highly customized
I think of gentoo as the "emacs" of operating systems - really cool, but with a high pain threshold before the cool starts paying for itself.
Re:Redhat 6.2 (Score:4, Interesting)
My Gentoo system was up 309 days [snerk.org] before I realized that the PSU fan had stopped turning and the motherboard overheated and blew 6 capacitors which is why the clock got so far out of sync (the computer thought it was April when I rebooted it back in November) which explains the graph weirdness.
Prior to that I had an uptime well over 200 days ruined by a blackout that outlasted my UPS.
I perform updates here and there on my server periodically and perform a full-scale "bleeding edge" upgrade whenever I'm forced to reboot the machine.
Re:Redhat 6.2 (Score:3, Interesting)
I've got to admit, I've got a relatively short uptime right now on Gentoo, only 98 days, 15:27. (There was a power outage due to a storm.) But in that time, I've upgraded versions of asterisk, postgresql, apache, squid, samba, PHP, and mythTV. I've also recompiled the system using a new compiler, and the only service downtime I had was when the recompile of PHP was finished, merging it in crashed Apache. Total downtime? Less than 15 minutes. Now this is a home server, and I wouldn't be nearly so aggressive on a production machine.
I've got a production Gentoo server running Xen (I use custom CFLAGS to remove thread local storage, which makes for a HUGE improvement for xen virtual machines.) I have duplicates of the production virtual servers which are upgraded while out of circulation, and when tested, they are placed into live circulation, and the live machines are pulled out of service.
While people can complain about being able to shoot their own feet with Gentoo, I've never seen someone take a Redhat server from 6.2 up to Fedora Core without having to completely reinstall.
Re:This article makes good points. (Score:2, Interesting)
However, for a lot of the packages I use, I require a recent version. There are plenty of bugs in Samba's domain support which have only been recently fixed. Bacula, a backup tool, is fantastic but the only version in Debian stable has been showing its age for months, if not years.
Possible Solutions:
Run a mix of unstable/stable. What's the point in a supposedly "stable" distro then?
Run "stable" and live with the old software. Not an option for me, I can't very well tell my staff that the reason such a feature doesn't work (even though the developers fixed it over a year ago) is because it hasn't hit Debian stable yet. They simply will not accept this.
Run unstable. What's the benefit of using Debian if I'm using the unstable branch? Granted, apt-get et al are great tools, but so is portage.
Compile my own newer versions of packages (and any dependencies which need updating). Eurgh. The whole point of a tool like portage is that the dependency hell which so often crops up when compiling a major package is effectively eliminated.
None of these are an issue for reasonably mature software where it's unlikely you need the latest version and you're probably better off with a proven version - cf. Apache or Postfix, for instance.
Re:This article makes good points. (Score:4, Interesting)
If you are dealing with 2-3 packages you can do that by using backports.org or backporting yourself. If you need more and these are an essential part of the business there is no difference between portage and backporting/local packaging. In ether case they have a tendency to break and you need local developer/sysadmin time allocated to that. Portage gives you no advantage whatsoever because the resource you gain in keeping more than 3-4 packages synced to their projects HEADs you will lose in infrastructure upgrade creep. Every time I have looked at this in the past taking out the numbers out of the ticketing and workflow control systems have proven that this is the case. I have yet to see one case where this is not.
Re:This article makes good points. (Score:1, Interesting)
Re:This article makes good points. (Score:3, Interesting)
It's arrogant, elitist (and ignorant) comments like this that really drive me crazy.
What you use in your production environment depends on different things; Knowledge and preference of the admins, business needs, type of environment, etc.
At my current employer, we're moving away from debian towards fedora for a very specific reason: Our requirements dictate that we *need* functionality that doesn't exist in 1000 year old software that's housed in debian packages. Don't get me wrong, I love debian. I personally find apt to be a better package manager than rpm, but we simply can't wait for 6 months/2 years/whatever to get the features that exist right now in more "bleeding edge" distros.
Being a long time fedora user I can say with a great deal of confidence that fedora is very stable on a server. We run one version behind on our servers to let the brand new version mature a bit (we're running FC5 on our servers now), and have *never* had an issue with stability.
Can we all just stop with the two rediculous Fedora FUD comments please?
Dependency Hell(tm) no longer exists for redhat (and hasn't for a LONG time) and FC is a stable OS.
Just stop.
It's old now.
Re:This article makes good points. (Score:3, Interesting)
Gentoo does reasonably well with configuration stuff (certainly better than any other system I've seen), but I still think it should be better; it'd be really nice if upgrades that change config files would be built but not installed, and then you'd be guided through updating the config file stored in the new package, and then it would install the package, overwriting your old config file with the version you prepared separately. And then it could stop the service right before installation and start it again right after, because the new configuration would already be in place.
The other thing that would be great would be if it could build packages with their dependancies built but not yet installed. Then it could do the revdep-rebuild *before* there were any non-working programs on the system.
But really, the only case where I've had problems with Gentoo on my server is when there's a difference between what services are running now and what services would be started if I rebooted and got to the current runlevel. (I.e., you install something and start it but don't add it to the default runlevel, and when you eventually reboot, you don't realize it's not running until somebody complains that the thing they need isn't working.)