Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Facebook Linux IT

A Look at Facebook's Use of Systemd (phoronix.com) 84

Posted by msmash from the growing-adoption dept.
At an event this month (you can find the video of it here), Davide Cavalca, a production engineer at Facebook, spoke about the growing adoption of systemd at the data centers of the company. From a report: Facebook continues making use of systemd's many features inside their data centers. Some of their highlights for systemd use in 2018 includes: Facebook's servers have been relying on systemd for about the past two years. Facebook is using CentOS 7 everywhere from hosts to containers. While relying on CentOS 7, Facebook backports a lot of packages including new systemd releases, Meson, other dependencies, and of course new Linux kernel releases. Facebook is working on "pystemd" as a Python (Cython) wrapper on top of SD-BUS.
This discussion has been archived. No new comments can be posted.

A Look at Facebook's Use of Systemd More

A Look at Facebook's Use of Systemd

Comments Filter:

  • Ugh (Score:2, Insightful)

    by Miser ( 36591 )

    Let Facebook HAVE systemd. They can keep it.

    • Re: (Score:3, Informative)

      by Tailhook ( 98486 )

      Amazon Linux — the image used on most AWS EC2 Linux instances — has also changed over to systemd with Amazon Linux 2, released last Jun 26. Amazon Linux was an important sysvinit holdout; if your users expected to use your work on EC2 with Amazon's distro you needed to care about sysvinit. Now that Amazon Linux has moved on there are no important distributions based on sysvinit remaining and it may be safely ignored.

    • I actually watched the video.
      Seems like Facebook is finding systemd useful.
      They really present some extreme use cases.
      At one point they call systemd "awesome" so no doubt they will "keep it".

  • Systemd (Score:4, Funny)

    by 93 Escort Wagon ( 326346 ) on Friday October 05, 2018 @12:24PM (#57432502)

    #systemctl enable usertracking.service
    #systemctl start usertracking.service

    • Re: (Score:2)

      by jmccue ( 834797 )

      #systemctl enable usertracking.service #systemctl start usertracking.service

      Meant to moderate funny but mouse slipped and it hit troll. Undoing mod.

  • systemd (Score:5, Insightful)

    by Anonymous Coward on Friday October 05, 2018 @12:25PM (#57432520)

    triggered

    • Re: (Score:2, Funny)

      by Anonymous Coward

      SYSTEMD!

      Slowly I turned...step by step...inch by inch...

      Then I unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep.

  • It is a floor wax it is a dessert topping (Score:3, Insightful)

    by bobstreo ( 1320787 ) on Friday October 05, 2018 @12:30PM (#57432568)

    Name three things that deserve each-other:

    Systemd, Bookface, Python,

  • Wait (Score:3)

    by ArchieBunker ( 132337 ) on Friday October 05, 2018 @12:41PM (#57432654)

    A Python wrapper for the systemd API? So now they're going through Python to go through systemd to make some calls...?

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      Combining Python and systemd is like mixing vomit with diarrhea.

    • Re: (Score:1)

      by Anonymous Coward

      Don't forget d-bus and its xml-wrapping of requests and the various libraries that do the encoding and the interfacing to the encoding and the talking to d-bus and so on. This of course is obviously why d-bus needs to be in the kernel. Because logic.

      I'm a little surprised zuckbook uses python, though. They're a php shop. Doesn't php have systemd api wrapper interfaces?

      • Re: (Score:2)

        by mvdwege ( 243851 )

        kdbus hasn't been an issue for at least 3 years [github.com], everyone is focusing on improving the userspace implementations of the DBus protocol.

        But do go on, keep demonstrating that the anti-systemd side is mostly composed of idiots. The more you do your best, the sooner we can write you lot off as irrelevant.

  • Phew (Score:1)

    by Anonymous Coward

    That explains the continual smell of decaying horse shit that wafts out of Facebook HQ.

  • They’re using CentOS huh? (Score:5, Funny)

    by 93 Escort Wagon ( 326346 ) on Friday October 05, 2018 @01:01PM (#57432804)

    I can understand... these startups can’t afford the license fees Red Hat charges.

  • systemd has logged your complaint (Score:5, Insightful)

    by jfdavis668 ( 1414919 ) on Friday October 05, 2018 @01:06PM (#57432848)
    If anyone can ever figure out how to read the logs, someone may respond.

    • Re: (Score:3, Interesting)

      by Anonymous Coward

      LOL. It is a pain to remember things like:

      systemctl restart openvpn@dev.service

      The command isn't named systemd, there's an @ sign in the name, and the dev invokes magic to find /etc/openvpn/dev.ovpn.

      But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console. Having to resort to using strace just sucks.

      • Re: (Score:1, Informative)

        by Xenolith0 ( 808358 )

        But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console.

        Yeah, I know what you mean! Typing "journalctl -u sshd" is impossible to remember! Much easier to just jump on the systemd-hate bandwagon and use strace.

        It's SO-FUCKING-ANNOYING that systemd logs all stdout and stderr to the journal! I mean ugh! having to type journalctl. I can't even.

        • Re:systemd has logged your complaint (Score:5, Informative)

          by DCFusor ( 1763438 ) on Friday October 05, 2018 @05:33PM (#57434784) Homepage
          Yeah, you're one of the ones who can't even, that's obvious. When this crap just gets added to your system, how are you even supposed to know there's such a thing as journalctl? Some of us, do, you know, real work on this stuff, and only do sysadmin if it's required to set something nifty up. Having to also fix things someone else broke is extra work that isn't needed. The pushback isn't for no reason - systemd really screwed up a lot of people's day. If it just worked...probably would have been another story, despite the terrible architecture.

          • Re: (Score:3, Interesting)

            by Xenolith0 ( 808358 )

            So you're mad that you might have to read a manual to a highly complex system? I'm impressed that you do "real work" on a system that you have no idea how it works!

            If you don't want to bother reading manuals because you're too busy doing "real work". Why did you even bother upgrading to RHEL7? Stick with RHEL6, which is supported until 2024.

            And yes, systemd is incredibly well documented. Just the man pages alone on the system:
            $ man -k systemd | wc -l
            147
            But for online documentation: https://www.freed [freedesktop.org]

            • Re: (Score:2)

              by jrumney ( 197329 )

              $ man -k systemd | wc -l
              147

              You're not really helping your point there. Something as complex as systemd needs far more than 3 or 4 pages for its manual.

              • thats just for specific part of it. Description "systemd, init - systemd system and service manager"
                I got 178 lines in my "man systemd" attempt and read this at the end of it (plus there are more notes after these ones)
                "SEE ALSO The systemd Homepage[10], systemd-system.conf(5), locale.conf(5), systemctl(1), journalctl(1), systemd-notify(1), daemon(7), sd-daemon(3), systemd.unit(5), systemd.special(5), pkg-config(1), kernel-command-line(7), bootup(7), systemd.directives(7) "
              • Not sure if you're joking... But man -k lists all man pages about systemd. There are 147 man documents on a RHEL7 system, each multiple pages long.
          • "When this crap just gets added to your system, how are you even supposed to know there's such a thing as journalctl?" - how did you learn to use anything on a computer? clue - RTFM

        • Why did you feel the need to rewrite all the traditional text processing tools over again? So now you have binary logs and these new tools that do exactly what the old ones did...? Did someone use AIX one day and say hey now there's an idea Linux could use!

          • See this for a detailed answer: https://www.loggly.com/blog/why-journald/ [loggly.com]

            You can pipe the output of journalctl to whatever you want. So I'm not sure I understand the actual complaint. But further, while journal is great for catching start-up messages from services, it is not something you would use in an enterprise for true logging. Which is why on RHEL systems, journald is configured to forward everything to rsyslog which works the same as it always has. So... Nothing is really different, and you can do

            • A centralized place to store all a sytem's configuration, with a security model and lots of magic redirection that is different from the file system. All wired together with GUIDs.

              SystemD is weak. It is a Registry would make *nix truly great.

  • systemd hacking (Score:1)

    by Anonymous Coward

    So that's why it's so easy to hack facebook.

  • Facebook's use? (Score:5, Interesting)

    by jeff4747 ( 256583 ) on Friday October 05, 2018 @01:25PM (#57432986)

    I can't watch the video because work, but from the text this doesn't look much like Facebook is using systemd. Facebook is using an operating system that uses systemd. And may someday have a python api to do what everyone else does with bash.

    • And may someday have a python api to do what everyone else does with bash.

      If you think people are using bash to work with the SD-BUS API you either don't know Bash, or don't know the API being talked about.

  • Only one good use-case (Score:5, Interesting)

    by DCFusor ( 1763438 ) on Friday October 05, 2018 @03:44PM (#57434104) Homepage
    for systemd - tons of identical instances, usually virtual so they can really be the same. The moment you customize with shares mounted on other machines, use wifi instead of wired ethernet, need to run custom daemons and so on - all of which worked fine here for years - under systemd, you find out that nope, shares didn't mount because they tried and failed before there was an IP address (now fixed, which fix broke the workaround you needed at first) - things like conky get started in an endless timeout loop till the system dies from no memory (maybe fixed, so hard to get right I gave up on conky) and on and on for a long list of stuff that worked before systmd, and LP says "just don't do that, EWONTFIX". The whole list is too long for the margin here.
    .

    And his clan call us haters. They didn't have to make a ton of interlinked custom real hardware on prem systems work - broken time and time again by upgrades to that nasty piece of work...the web is full of workarounds that used to be the only things that worked, that are now broken as the systemd coders finally listened and fixed them but in such a way as to break the workaround, again, and again, and again.
    Shades of lost productivity, near windows-like. Which of course doesn't affect RedHat revenue as they support - tons of identical instances...llike farcebook. The simplest case, get it right once and it's right everywhere. No edge cases. Wow, I'm er, underwhelmed.
    In other words, this far fan-dancier init system doesn't (or didn't) handle anything complex, just added complexity to only handle the simplest stuff correctly. And was forced on us long before ready.
    It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount. Glad it wasn't far away and I could just do that.
    Not a hater, exactly. Just gheez - it caused me a ton of unnecessary unpaid work and stress and was arrogant about how it was somehow my fault for using what had been the documented standard ways of doing things. Way to win friends and influence people.

    • Re: (Score:1, Informative)

      by thegarbz ( 1787294 )

      all of which worked fine here for years

      Seriously you realise that systemd was build precisely because the things you listed didn't work fine for years.

      shares didn't mount because they tried and failed before there was an IP address

      Fix your startup order in the unit file.

      things like conky get started in an endless timeout loop

      Fix the start conditions in the unit file.

      and LP says "just don't do that, EWONTFIX".

      He was being polite. He should just tell people to RTFM.

      And his clan call us haters.

      No. Ignorant maybe, but not haters.

      Shades of lost productivity, near windows-like.

      Yeah that's what happens when you adopt a system and spend more effort complaining than simply reading the manual.

      It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.

      It never ceases to amaze me how people can get their system in that kind of a state.

      it caused me a ton of unnecessary unpaid work

      Systemd can't help you with a

      • Re: (Score:1)

        by Anonymous Coward

        It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.

        It never ceases to amaze me how people can get their system in that kind of a state.

        While not a failed mount, I just ran into this shutdown issue yesterday.

        Clean Debian 9 install into a newly created VirtualBox VM.

        "shutdown -r now" hangs halfway through the process with systemd starting a 15 minute timeout counter because it can't unmount "/"

        Seeing as the only changes I made after boot were to set the hostname and lookup the MAC address.
        At that point I added a dhcp reservation with fixed IP (on another system of course) and then came back to reboot the VM and make sure it got the right IP.

        • Re: (Score:1)

          by Anonymous Coward

          While not a failed mount, I just ran into this shutdown issue yesterday.

          Clean Debian 9 install into a newly created VirtualBox VM.

          You are certainly not the first person to claim I fucked something up, but I am really really hoping you will be the first to tell me exactly what above was the fuckup and hopefully what to do instead. (Emphasis added)

          Use Devuan [devuan.org] instead of Debian.

      • Re:Only one good use-case (Score:5, Informative)

        by DCFusor ( 1763438 ) on Friday October 05, 2018 @04:51PM (#57434620) Homepage
        What order, I did an NFS mount in /etc/fstab, which is the documented way that worked for years, and not only did systemd fail to mount it, it failed to respect the bg (backgrounding try again) flag, and then refused to let the system shut down since it couldn't unmount what it had failed to mount - which I verified by mounting manally at the CLI - and then it would unmount it and shutdown normally.
        //

        There was no fine manual to read that I'm aware of when this happened. Just oddball notes all over if you googled about how to fix just this particular bug - which workarounds worked until the next update. You might not have been paying attention.
        //

        Got my system in such a state? What a jerk, you're proving my point. Systemd got it into that state, I did zero, nada, nothing but allow an update to happen.
        //

        I don't have an employment contract - I work for my own outfit, which is a physics lab. It needs a lot of fairly custom machines from little to huge and fast for data aq and control - which is needed as the fusion reactor I'm developing works well enough to make too many neutrons to be safe near....
        //

        If it wasn't a bug that affected a lot of people, why did they complain and get the snotty remark - and then list workarounds findable on google?
        //

        That lovely restart loop on slow starting things is well documented as well, and has a workaround involving setting a longer timeout, which is a nasty hack...
        //

        Pretty sure it's not ignorant to trust "security" updates. I wouldn't have known systemd existed, much less have had time to retrain on hard to find or nonexistent documents that were changing daily...I'm REAL sure that something they broke isn't my fault, and people who tell me otherwise might be the problem.

  • Did it have a part in them getting hacked? Would not surprise me. These people are incompetent.

Slashdot Top Deals

In the long run, every program becomes rococco, and then rubble. -- Alan Perlis

Close