Stories
Slash Boxes
Comments

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Gentoo On Server Considered Harmful

Posted by kdawson on Sun Jan 28, 2007 11:24 PM
from the not-a-desktop dept.
Siker writes in to point out his blog post — Why Gentoo Shouldn't Be On Your Server — which seems to have stirred up a lot of discussion, including a thread on the Gentoo forums. From the post: "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything? If you update anything you will undoubtedly need to update configuration files. You will need to fix things that break in the upgrade process... This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge."
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • by suso (153703) * on Sunday January 28 2007, @11:27PM (#17795014) Homepage Journal
    At the same time, the "your system is always approaching the bleeding edge" way of doing things solves one problem that I've always been bothered by with running user servers for suso.org [suso.org]. Eventually, the OS on the server reaches the age where it is no longer supported and updates are no longer coming out for it. This isn't always X years where X is the number of years that a distribution claims to provide package updates for. Its usually X-1. This is because you'd be foolish to use the very latest hasn't been available for more than a day version of Linux. Usually you wait for 6-12 months for it to be mature and have special packages of whatever available for it. Then you spend another month or two setting up the machine and getting it ready for production. By that time, you've already burned over a year of support time. Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left. Upgrading such a system to the latest version of whatever distro means taking the server down for several hours to upgrade, hope to hell that special packages you've built and configurations aren't broken and in nightmare situations, roll back because something is broken and can't be fixed.

    The promise of Gentoo for me is being able to continually upgrade and never get outside of that window of support.

    I actually have a new shared user system that is running Gentoo that is kinda in beta right now. This article was very useful for me because it brings up those points about stability that concern me. Its kinda an experiment.

    I think I may try Debian next.
    • by lordsilence (682367) * on Sunday January 28 2007, @11:37PM (#17795118) Homepage
      Gentoo has proven troublesome in a production environment.
      The problem isnt updating often... it's when you DONT update often.

      We had one system which we didnt bother to update. (Dont fix what isnt broken)
      Then one day we had to upgrade some of the services.. which in turn required lots of libraries to be upgraded.

      In the end, we had to upgrade kernel.. cause libraries didnt support 2.4 kernel.
      Stuff change too much in gentoo to put it simple.. It's easier to keep updating often

      emerge sync && emerge -u world
      Then iron out all config-changes. Find out which undocumented features were changed, which keys to add to startup script etc.

      Lesson learnt: Dont use gentoo on production systems. Run it on your desktop computer you play around with...
      • by dbIII (701233) on Monday January 29 2007, @01:40AM (#17796004)

        Lesson learnt: Dont use gentoo on production systems.

        I would see that lesson instead as don't experiment on your production systems. Obsolete hardware is useful for testing out stuff like this.

        The reason I don't run gentoo on production systems is simply becuase I am not familiar enough with it and it is different enough from other distributions of linux and other versions of *nix to make things confusing. It's the same reason I don't use reiserfs - if it all messes up how can I or any moderately skilled linux user get things back into operation?

      • by arivanov (12034) on Monday January 29 2007, @03:05AM (#17796498) Homepage

        The joy of portage all the way. Continuous upgrade versus release cycles. 15 years of dealing with both have convinced me that portage is good only in two places:

        • your own workstation where you want to look at how the bleeding edge looks before unleashing it on the unsuspecting user population. This one is updated continuously and rebuild as necessary.
        • single special purpose dedicated servers (not run of the mill 10+ servers with same load). You build these once and after that you leave them alone until they die (preferrably).

        Everything in between - forget it. Update hell, dll hell, etc. If you use portage (either the BSD or Gentoo incarnation) you die and releases are the exact and only solution to this. You can stamp servers with a "released" OS out of workshop by truckload and you can be more or less confident that updates will not break a lot of things. The only problem is upgrades to next release but if you are using The One OS to rule them all [debian.org] even that is not a problem.

        • by Anonymous Coward on Monday January 29 2007, @01:29AM (#17795924)
          In the case of Red Hat, they'll backport changes for you so that you don't need to upgrade 50 other packages in order to get a security patch for Apache to work.

          So in a way, yes, it is Gentoo's fault. It's just the way the distro is designed. Everything at the latest revisions possible. Great for a home system, not good for a server you have to maintain.
    • Redhat 6.2 (Score:5, Funny)

      by flyingfsck (986395) on Sunday January 28 2007, @11:42PM (#17795160)
      Don't fix it if it ain't broke: up 292 days, 22:26 The reason for the short uptime, is PSU upgrades...
        • by toadlife (301863) on Monday January 29 2007, @04:01AM (#17796738) Journal
          That would have had around 900 days uptime if my reboot-happy Windows-only-admin coworkers wouldn't have reset it in a panic on multiple occasions to "troubleshoot" (no it was never a problem with my OpenBSD box) mail problems.

          I don't know what the hell it is with Windows-only admins and rebooting. The kind of instability that required reboots all the time was reduced drastically with Win2k and win2k3, yet that insatiable urge to reboot first and ask questions later still plauges my Windows-only counterparts.
    • by ePhil_One (634771) on Sunday January 28 2007, @11:48PM (#17795228) Journal
      Then you get users onto it and now you only have X-1.5 years of support. On Fedora, this means practically no time is left.


      Which is why IT Pros prefer Red Hat Linux or its unencumbered variants link CentOS, White Box, and Scientific. Better testing up front thanks to the Red Hat gang, and longer shelf life. Which is why most commercial software chooses to support it first, it provides a stable base.

    • by saleenS281 (859657) on Monday January 29 2007, @01:06AM (#17795752) Homepage
      And that my friend, is the niche Opensolaris will quickly start filling.
        • Re:CentOS updates (Score:5, Informative)

          by DA-MAN (17442) on Monday January 29 2007, @01:40AM (#17796000) Homepage
          At risk of exposing my ignorance here (I'm a Debian person; the last time I did anything RedHat-based was before automatic package management), what is CentOS's automatic-update feature like? Does it have one?

          Yes, it's yum.

          I assume it uses yum, or something like it, being RedHat, but does it pull from RedHat's servers directly, or are there separate CentOS repositories?

          CentOS Repositories

          In that case, how closely do the CentOS repos track the 'official' RHEL ones, in terms of patches and bugfixes?

          The official RHEL ones are publicly available, and tracked by CentOS very well. The only changes they make are for trademark requirements. Thus far it has been bug for bug compatible with RHEL.

          Not that you'd probably want to do it on a true 'production' system, but can you do the CentOS equivalent of 'apt-get upgrade' and be reasonably assured of not breaking things?

          Yes

          I've always been intrigued with CentOS, and it does seem to have a good reputation as far as stability is concerned, but after growing up with apt-get (and before that, nightmarish experiences with dependency hell on some very early RedHat systems), I've developed a certain perhaps-unwarranted negative bias of everything else.

          I prefer yum myself. I used apt when it first came out, and loved it. Since I got my first 64 bit machine I just prefer something that handles the dual architecture a little better. For the most part they're about the same though.
  • Gentoo allows you to be on the cutting edge, just like all the other distributions. The primary difference is it makes it very easy for those who don't know what they are doing to be there. Most folks running SuSE, RH, or one of the other 'package' based distributions won't build their own RPM, etc. There is nothing stopping one of the 'normal' distributions from upgrading the kernel with each release. I certainly don't update everything on my Gentoo box because it is there, on my server.

    I run Gentoo on a server. The server is stripped down beyond what a typical 'router' distro looks like - one of the reasons I went with Gentoo is I could really trim the system down for the job at hand. My server only gets updates for security, and once in a while a bug fix that impacts the applications running on the server. Not often. When I need to compile something big, the last place I'd do it on is the server itself - it has another task. I take one of my workstations with far more GCC horsepower and let distccd [gentoo.org] do the work for the poor little pizza box. Beyond the initial build, I doubt those boxes have ever compiled anything.

    Since it is a source-based distro, I also am not trapped by RPM's or other packages no longer getting provided for my system. One of the applications I had was using RH9 (with paid support) only to have them drop maintenance on it and have the vender drag their feet moving to another platform (clue stick, they had issues with the 2.6 kernel, so would not 'support' any platform but RH 8 and later 9. The enterprise editions? Forget about it... You want to live in the suck, you try keeping one of those boxes alive and secure years after it EOL.
  • And?? (Score:5, Informative)

    by friedmud (512466) on Sunday January 28 2007, @11:35PM (#17795080) Homepage
    "I firmly believe in updating server software only when you need to. If you don't need new features, and things are working, why change anything?"

    I agree... so why does this preclude using Gentoo?

    Just because you _can_ update all the time doesn't mean you should. I've used gentoo for various purposes (server, desktop, laptop). What I usually do is get it setup and install all the packages I need and then leave it for a _long_ time... only upgrading packages that I either need the new capability of or for security purposes.

    Look... I personally don't think Gentoo is the best server OS out there... but I also don't think that just because the package system makes it really easy to tinker with the system that Gentoo is inherently unstable...

    Friedmud
  • by michrech (468134) on Sunday January 28 2007, @11:35PM (#17795088)
    There is no 'stable' version of Gentoo. Gentoo is rather a moving target where emerge will forever cause your system to approach the cutting edge.

    Yea. Not quite. This is what the "ACCEPT_KEYWORDS=" setting in make.conf is for. If you don't have it set, you get "stable" packages. If you do have it set, you get the unstable stuff.

    Further, with the use of the files in /etc/portage, you can have a stable system, but have one or more packages be unstable without having it a system-wide setting.

    Haven't read the rest yet, but wanted to point that out.
  • Not for me! (Score:5, Funny)

    by MarkRose (820682) on Sunday January 28 2007, @11:36PM (#17795104) Homepage
    I certainly wouldn't want a Gentoo on my servers. Sure, it wouldn't weigh [wikipedia.org] much, but think of the poop you'd have to clean up!
  • Agreed. (Score:5, Interesting)

    by MrNaz (730548) on Sunday January 28 2007, @11:39PM (#17795136) Homepage
    I have been a server admin for web/database for about 3 years now. I agree that bleeding edge is *not* where server admins want to be. There's a reason that Debian is widely considered the best server OS despite being rather far behind the bleeding edge. Tried and tested is better than the latest and greatest when you rely on the machine being up. It's also worth noting that the military doesn't use any COTS technology within 5 years of it being released.
  • by mattdev121 (727783) on Sunday January 28 2007, @11:41PM (#17795154) Homepage
    The problem with Gentoo Linux is not the system itself, it's the stereotypes that people put against it.

    Gentoo is only good for ricers, Gentoo is bleeding edge and unstable, Gentoo is only good for X deployment

    The truth about Gentoo is that it is not really a distribution. Gentoo Linux does not make "releases" and it does not aim to cover one area of the market alone.

    In Gentoo's packaging system, called portage, the aim is not only to provide up-to-the-minute packages (which it does) but also to provide a wide variety of both tested and verified "stable" packages as well as more bleeding-edge, testing packages.

    This, along with a properly configured make.conf and /etc/portage file system, allows you to pull down the packages you want that have been verified as stable (and are also under watch by the Gentoo security project) and keep track of their libraries with revdep-rebuild.

    Stop branding Gentoo with stereotypes that label it as X distribution, the project even calls itself a "metadistribution" capable of dropping into multiple roles.

  • by Anonymous Coward on Sunday January 28 2007, @11:43PM (#17795168)

    First of all, I find it interesting that FreeBSD never seems to get these complaints and hate about having to recompile packages with portupgrade all the time, and being able to tweak the flags, etc. In this respect, it's just like gentoo!!! Except without a lot of the fancy features like etc-update and slots and masking and multiple supported versions. Yes, the "base system" is more stable on FreeBSD (which is both a blessing and curse), but what is it about Gentoo that attracts so many haters/inexperienced admins, hmm??

    Anyway, I run Gentoo on servers. (Also FreeBSD). I think it's great. I can't stand stuff like Red Hat, which makes it difficult to customize anything, so I'd always resort to installing stuff "by hand", which was a huge pain. Or creating a custom RPM, which was an even bigger pain (RPM is basically a huge clusterfuck in general).

    Being able to set up ebuild "overlays" is great. Being able to set up custom profiles that contain all the software needed for a particular app is great. Writing ebuilds is a piece of cake. Turning on/off various features system-wide is very helpful. The mechanism for merging configs (etc-update or dispatch-conf) is nice. Being able to pin down specific versions with masking is good. Etc. For the record, I've never tweaked the CFLAGS in my life.. that's just not why I use Gentoo.

    The author writes this:

    A profile update will touch a very large number of configuration files, and it may even alter your startup process. Obviously this is not something you want to do to any server. ................. The end result: the machine had to be resuscitated on-site with associated downtime.

    I have no idea what happened to him. Updating your profile is basically moving a symlink, which changes some lists of base packages and other high-level build configuration. It doesn't "touch" anything in your system. Sure, you have to some upgrades afterwards, but you have to do that regularly anyway on Gentoo. Compare it to upgrading FreeBSD from 5.x to 6.x, which is much more involved.

    As you might be aware, FreeBSD has a nice little program called portaudit........... Now, Gentoo also has something like portupgrade. What it doesn?t have is portaudit. ............ In all fairness, Gentoo has an experimental command called "glsa-check".

    I've been using glsa-check for a while now, it works great. It tells me what's got known holes and I just update those packages, and their dependencies. What problem did he have with it, besides the "experimental" status? Yeah it can "do stuff", but I don't use those options, I just use it to get a list of packages with known holes. Heck I could probably write a script to do the very same thing.

    Suppose you need to patch one of your installed packages by the way.. it's very easy to create custom ebuilds on Gentoo. Sometimes I plug security holes that I've found on my own for instance.

    I have a simple strategy with Gentoo servers: keep an identical test/staging server nearby and do your updates on that machine first. Run your application tests and then upgrade the production machine. If you want, build binary packages on the staging machine. I would do this even with Red Hat, Debian, etc.

    Another point: I've NEVER run "emerge -u world". I always do the packages in small groups or chunks and then updated configs, restarted daemons, and run tests after each one. This seems like a much better strategy than what some people do.

    Also, I gotta say, it's probably not a good idea to run Gentoo on a production server unless you've got at least 5 years of Linux admin under your built. You also need to FOLLOW the Gentoo newsletter, AT LEAST, so you can get a heads-up when config files change or files are moved around. It happens from time to time.

    Really, the only valid point he makes that generalizes to servers other than his own is the following: Gentoo takes more time to keep running. But you have to weigh that against the flexibility you get, just like any "build vs. buy" decision.

  • *sigh* (Score:5, Insightful)

    by Ant P. (974313) on Sunday January 28 2007, @11:45PM (#17795188) Homepage
    The article makes it sound as if gentoo installs the ~unstable profile by default. The stable one's no more bleeding-edge than Ubuntu.
  • by God of Lemmings (455435) on Monday January 29 2007, @12:00AM (#17795318)
    There is NOTHING forcing you to "emerge world", "emerge system", and "emerge --sync" every single time Gentoo
    updates portage... Emerge flags include "--pretend", "--ask" and "--fetchonly" among several others, learn to
    use them.
  • by RyogaHibiki (969138) on Monday January 29 2007, @12:23AM (#17795476)
    (I posted this on the gentoo forums)

    If someone is running a server room with many live production systems where downtime must be in seconds per year, they should ALWAYS have a test environment and a production environment. Gentoo makes it extremely easy to produce this setup. Imagine if you will, this setup:

    1) Master rsync system (contains the portage sync used by all the systems)

    2) Test boxes for each role needed (perhaps you have 3 different kinds of servers, WWW, Mail, DB)

    3) Many production boxes

    What you would end up doing is creating a fairly generic gentoo install (by generic, I mean hardware independent - like i686 or whatever you feel comfortable that will be supported for the lifecycle of the servers). All production servers are identical to the test boxes at the beginning of this example and have a simple backup of the whole test environments (perhaps a large tarball saved on a separate drive). A new update is necessary for apache so you do an emerge --sync on the master rsync system. Then you rsync all the test boxes so they have the same portage tree. You then run the necessary installs on the test systems to make sure that it works, if it doesn't, then you research why and figure out if its easier to fix after the update, or if the update needs to be done differently, if you need to, you can restore the test system from the backup and start over. After you have all the test boxes running well, you can then rsync the production boxes and reproduce the steps necessary to get them updated.

    Once all this is said and done, the production boxes will all be updated successfully (and the updates were tested on the test boxes) and the test boxes will at this point have the same configuration as the production boxes. You would make a new backup of the test boxes and wait for the next time you have to do this cycle. As long as the boxes really are identical, you could even run konsole (or another xterm that allows you to send your input to multiple console windows) and perform the identical steps on all the same type of boxes (sending your update commands to 20 or even 50 servers at once).

    I'm sorry, but in any real production environment, I see NO issues with this setup. It may be a bit time consuming if you have a lot of etc-updates to do, but still, the basic update should be painless to that point.

    -Jason Pf.
  • Nonsense (Score:5, Insightful)

    by loxosceles (580563) on Monday January 29 2007, @12:30AM (#17795532)
    You say Gentoo wants to change a lot of stuff?

    Any binary distribution has two modes of updates. One is an updated package within the same release; the other is a mass-update from one release to another. Gentoo combines the two, since the distinction is artificial. What you call "changing a lot of stuff" is merely keeping packages reasonably current so that you never have to do a mass-update or complete reinstall.

    Anyone who considers the Gentoo update process too difficult either hasn't used Gentoo (upgrades are easy, and there aren't that many of them if you stick to stable x86) or has never dealt with package conflicts in binary distributions. That is the real horror I want to avoid, and I avoid it nicely by running Gentoo.
  • It's been said before by many. I cannot say I disagree with the article. With more traditional distributions of Linux, you always have standardized packages with some amount of quality control. Bugs and security holes slip through to the end users all the time. Often your end users report these bugs to the upstream maintainer. Occasionally, the end user even submits fixes upstream.

    Gentoo is so system dependent compared to other distros. The end result, instead of having 1 package for some function, you have 1^n packages for that same function. Given 'n' amount of users with differing hardware and compile time arguments. The Qaulity Assurance ends at the user, always. You ultimately have a quality control department that consists of one, the user.

    Any system upgrade or maintenance procedures in production environments are usually limited to a few hours at most. It does not make sense to spend six hours compiling what could have been installed, configured, and tested in 6 minutes with a pre-compiled package. In the event of a hardware failure, I find it reassuring when a Linux distro can be loaded onto a spare box in 15 minutes. Then spend a few more minutes restoring configurations from a good backup.

    But that's just my opinion. To each his own. If it works for you, then go with it. Otherwise, I'd say it is a fairly level-headed review.
  • Not anymore. (Score:5, Insightful)

    by a9db0 (31053) on Monday January 29 2007, @01:19AM (#17795850)
    Gentoo on a server? No longer.

    I used Gentoo for several years. I learned an awful lot about Linux from it. And I appreciate the work that goes into it. But my servers run Debian now, for one reason - quick, reliable updates. I support several small businesses, I don't have the resources to maintain test environnments to check the impact of upgrades. And not having multiple powerful systems at many sites means distcc is not an option. And the recompiles occasionally necessary for apache or samba or postfix or mysql put an unreasonable strain on servers that are typically not high powered and are supporting multiple users. So for quick, reliable system updating apt-get beats emerge every time.

    I'm not knocking gentoo. It's a great system for testing stuff, and evaluating software. But in the 3 minutes it took me to type this post, I could update 5 servers that hadn't been updated in a week.
  • Not at all (Score:5, Insightful)

    by vandan (151516) on Monday January 29 2007, @03:30AM (#17796612) Homepage
    I've been using Gentoo on our database / web / email / many-other-goodies server since August 2003 ( I keep emerge --sync logs ). I'm running the stable branch on our server, and the unstable ( ~x86 ) branch on desktops. I certainly agree that updates on the unstable branch have to be done thoughtfully, but building binary packages when emerging helps a great deal with disaster recovery. It's nothing that can't be fixed with a little searching.

    But on the stable branch, I've actually been very surprised with how ... stable ... it is ( coming from the ~x86 branch ). I keep a separate binary packages repository for the server ... just in case ... but haven't actually had to back-track to anything yet. I do updates outside of work hours, and revdep-rebuild when upgrading major parts. I haven't had any catastrophes yet. Actually I haven't even had any mishaps yet. What can I say? If you are confident enough to run Linux on a server, I say you can handle the stable branch of Gentoo.

    As for the points the author raised against Gentoo:

    1) Too long to do initial install.

    This one gives it away from the start. You only install once. But this is at the top of the list. I can't remember how long it took me to install Gentoo on this server, but it was probably 2 days or something. Who cares? That's what time I take installing *any* server. You don't just whack it together and put it into production. You install, you read, you test, you frig around some more. What's wrong with that? The author is no server administrator.

    2) Same as point one, just repeated

    WTF? Seriously, this author has his head up his arse. On the one hand, he later says that you shouldn't update willy-nilly on servers, and yet then says that it takes ages to update everything. So what, exactly, is he trying to achieve? It takes me about 10 - 15 minutes to update MySQL, which is the most common package I update. What's wrong with that? I back things up, shut down MySQL, emerge the new MySQL package, test, and import form backups if required. No problem? Where is this guy's problem, seriously?

    3) Don't like updates, even if they are to more stable packages

    Nothing forces you to update packages. Also, no-one claims that packages updates *won't* break things ( though my experience is that in the stable branch, updates *don't* break things ). But if you don't want to update, don't. No problem. If you do want to update, the tools are there to update easily. Sure you should pay attention to what you're doing. It goes without saying.

    4) Same as point 3, but with the update impetus being security instead of stablity

    Doesn't deserve a response really.

    I challenge this author to prove that he's actually used Gentoo Linux for more than 7 days without running crying back to Linspire.