A Look at Facebook's Use of Systemd (phoronix.com) 84
At an event this month (you can find the video of it here), Davide Cavalca, a production engineer at Facebook, spoke about the growing adoption of systemd at the data centers of the company. From a report: Facebook continues making use of systemd's many features inside their data centers. Some of their highlights for systemd use in 2018 includes: Facebook's servers have been relying on systemd for about the past two years. Facebook is using CentOS 7 everywhere from hosts to containers. While relying on CentOS 7, Facebook backports a lot of packages including new systemd releases, Meson, other dependencies, and of course new Linux kernel releases. Facebook is working on "pystemd" as a Python (Cython) wrapper on top of SD-BUS.
Ugh (Score:2, Insightful)
Let Facebook HAVE systemd. They can keep it.
Re:Ugh (Score:4, Insightful)
They just had a major security leak of 50 million users? Competent you say?
Re: (Score:3)
pfft, they make $86-93K, they are not highly skilled admins
Re: (Score:1, Offtopic)
pfft, they make $86-93K, they are not highly skilled admins
This is evidence that they're not highly *paid* admins, and only has a passing correlation to skill
Re: (Score:1)
Nobody with any stroke wants to be a slave for Big Tech (tm).
Re: (Score:2)
A competent admin can make 50% more than that top end *away* from the bay area.
Re: (Score:3)
that is crap in the Bay area.
Re: (Score:3, Insightful)
Yeah, all the highly skilled sysadmins updated their startup scripts years ago to start under systemd. It's only the lazy weak sysadmins who have spent the last four of five years whining that systemd once broke a script somewhere.
The highly skilled admins forked Debian [devuan.org], to create Devuan [devuan.org], a Debian free of systemd.
Re: Ugh (Score:2)
Re: (Score:2)
Ahahahahahahaa, you think getting hacked proves competence?
Re: (Score:3, Informative)
Amazon Linux — the image used on most AWS EC2 Linux instances — has also changed over to systemd with Amazon Linux 2, released last Jun 26. Amazon Linux was an important sysvinit holdout; if your users expected to use your work on EC2 with Amazon's distro you needed to care about sysvinit. Now that Amazon Linux has moved on there are no important distributions based on sysvinit remaining and it may be safely ignored.
Re: (Score:1, Flamebait)
You appear to think that sysvinit is the alternative to systemd. How quaint.
Re:Ugh (Score:4, Insightful)
Ignoring good technology in favor of hip but bad technology is not a sign of competence.
Re: (Score:1)
Re: (Score:2)
I actually watched the video.
Seems like Facebook is finding systemd useful.
They really present some extreme use cases.
At one point they call systemd "awesome" so no doubt they will "keep it".
Systemd (Score:4, Funny)
#systemctl enable usertracking.service
#systemctl start usertracking.service
Re: (Score:2)
#systemctl enable usertracking.service #systemctl start usertracking.service
Meant to moderate funny but mouse slipped and it hit troll. Undoing mod.
systemd (Score:5, Insightful)
triggered
Re: (Score:2, Funny)
SYSTEMD!
Slowly I turned...step by step...inch by inch...
Then I unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep.
It is a floor wax it is a dessert topping (Score:3, Insightful)
Name three things that deserve each-other:
Systemd, Bookface, Python,
MOM! DAD! DON'T TOUCH THAT! (Score:3)
Re: (Score:3)
a real "Nexus of Evil" to coin a phrase...
I am pretty sure Alphabet/Google has a trademark on Nexus (of Evil)
Re: (Score:2)
It is a floor wax it is a holy war. (Score:1)
Oh! Holy Wars. Maybe this one will be better than the Vi-Emacs one.
Re: (Score:2)
Oh! Holy Wars. Maybe this one will be better than the Vi-Emacs one.
Nah. Either way, all the combatants do is type vicious comments on their devices.
Re: (Score:2)
Hahahaha, I use Joe! I am a heretic to _both_ camps!
Re: (Score:2)
Systemd, Bookface, Python, Vegans
Re: (Score:3)
I disagree. Python is actually pretty nice, but it requires somebody with real skill to be used competently.
Re: (Score:2)
Wait (Score:3)
A Python wrapper for the systemd API? So now they're going through Python to go through systemd to make some calls...?
Re: (Score:3, Insightful)
Combining Python and systemd is like mixing vomit with diarrhea.
Re:Wait (Score:4, Funny)
It's end-to-end integration!
Re: (Score:2)
Re: (Score:1)
Don't forget d-bus and its xml-wrapping of requests and the various libraries that do the encoding and the interfacing to the encoding and the talking to d-bus and so on. This of course is obviously why d-bus needs to be in the kernel. Because logic.
I'm a little surprised zuckbook uses python, though. They're a php shop. Doesn't php have systemd api wrapper interfaces?
Re: (Score:2)
kdbus hasn't been an issue for at least 3 years [github.com], everyone is focusing on improving the userspace implementations of the DBus protocol.
But do go on, keep demonstrating that the anti-systemd side is mostly composed of idiots. The more you do your best, the sooner we can write you lot off as irrelevant.
Phew (Score:1)
That explains the continual smell of decaying horse shit that wafts out of Facebook HQ.
They’re using CentOS huh? (Score:5, Funny)
I can understand... these startups can’t afford the license fees Red Hat charges.
Re: (Score:2)
Re:They're using CentOS huh? (Score:1)
So, you're saying that packages built for Centos won't work on the corresponding version of RHEL? That doesn't match my experience at all.
Facebook are freeloaders, not freeriders [wordpress.com].
systemd has logged your complaint (Score:5, Insightful)
Re: (Score:3, Interesting)
LOL. It is a pain to remember things like:
systemctl restart openvpn@dev.service
The command isn't named systemd, there's an @ sign in the name, and the dev invokes magic to find /etc/openvpn/dev.ovpn.
But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console. Having to resort to using strace just sucks.
Re: (Score:1, Informative)
But the logging issue is a bigger problem. I miss SysV scripts where even if something didn't get logged, you could at least see it on the console.
Yeah, I know what you mean! Typing "journalctl -u sshd" is impossible to remember! Much easier to just jump on the systemd-hate bandwagon and use strace.
It's SO-FUCKING-ANNOYING that systemd logs all stdout and stderr to the journal! I mean ugh! having to type journalctl. I can't even.
Re:systemd has logged your complaint (Score:5, Informative)
Re: (Score:3, Interesting)
So you're mad that you might have to read a manual to a highly complex system? I'm impressed that you do "real work" on a system that you have no idea how it works!
If you don't want to bother reading manuals because you're too busy doing "real work". Why did you even bother upgrading to RHEL7? Stick with RHEL6, which is supported until 2024.
And yes, systemd is incredibly well documented. Just the man pages alone on the system:
$ man -k systemd | wc -l
147
But for online documentation: https://www.freed [freedesktop.org]
Re: (Score:2)
$ man -k systemd | wc -l
147
You're not really helping your point there. Something as complex as systemd needs far more than 3 or 4 pages for its manual.
Re: (Score:2)
I got 178 lines in my "man systemd" attempt and read this at the end of it (plus there are more notes after these ones)
"SEE ALSO The systemd Homepage[10], systemd-system.conf(5), locale.conf(5), systemctl(1), journalctl(1), systemd-notify(1), daemon(7), sd-daemon(3), systemd.unit(5), systemd.special(5), pkg-config(1), kernel-command-line(7), bootup(7), systemd.directives(7) "
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Why did you feel the need to rewrite all the traditional text processing tools over again? So now you have binary logs and these new tools that do exactly what the old ones did...? Did someone use AIX one day and say hey now there's an idea Linux could use!
Re: (Score:2)
You can pipe the output of journalctl to whatever you want. So I'm not sure I understand the actual complaint. But further, while journal is great for catching start-up messages from services, it is not something you would use in an enterprise for true logging. Which is why on RHEL systems, journald is configured to forward everything to rsyslog which works the same as it always has. So... Nothing is really different, and you can do
*nix needs a Registry (Score:2)
A centralized place to store all a sytem's configuration, with a security model and lots of magic redirection that is different from the file system. All wired together with GUIDs.
SystemD is weak. It is a Registry would make *nix truly great.
Re: (Score:2)
Re: (Score:2)
I hadn't touched Linux in a while and decided to use a BeagleBone board for a project. Even setting the IP address was a chore. I guess ifconfig and route weren't trendy enough so they had to be replaced with ip. Then all I needed was to set some IO pins at startup. Hmm no rc.local anymore. Try writing a new service in systemd. Oh it didn't work and doesn't say why. Ok let's try enabling rc.local from systemd. Oh that didn't work either. There isn't even a systemd manual, just people telling you how to get
systemd hacking (Score:1)
So that's why it's so easy to hack facebook.
Facebook's use? (Score:5, Interesting)
I can't watch the video because work, but from the text this doesn't look much like Facebook is using systemd. Facebook is using an operating system that uses systemd. And may someday have a python api to do what everyone else does with bash.
Re: (Score:2)
And may someday have a python api to do what everyone else does with bash.
If you think people are using bash to work with the SD-BUS API you either don't know Bash, or don't know the API being talked about.
Re: (Score:2)
Only one good use-case (Score:5, Interesting)
.
And his clan call us haters. They didn't have to make a ton of interlinked custom real hardware on prem systems work - broken time and time again by upgrades to that nasty piece of work...the web is full of workarounds that used to be the only things that worked, that are now broken as the systemd coders finally listened and fixed them but in such a way as to break the workaround, again, and again, and again.
Shades of lost productivity, near windows-like. Which of course doesn't affect RedHat revenue as they support - tons of identical instances...llike farcebook. The simplest case, get it right once and it's right everywhere. No edge cases. Wow, I'm er, underwhelmed.
In other words, this far fan-dancier init system doesn't (or didn't) handle anything complex, just added complexity to only handle the simplest stuff correctly. And was forced on us long before ready.
It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount. Glad it wasn't far away and I could just do that.
Not a hater, exactly. Just gheez - it caused me a ton of unnecessary unpaid work and stress and was arrogant about how it was somehow my fault for using what had been the documented standard ways of doing things. Way to win friends and influence people.
Re: (Score:1, Informative)
all of which worked fine here for years
Seriously you realise that systemd was build precisely because the things you listed didn't work fine for years.
shares didn't mount because they tried and failed before there was an IP address
Fix your startup order in the unit file.
things like conky get started in an endless timeout loop
Fix the start conditions in the unit file.
and LP says "just don't do that, EWONTFIX".
He was being polite. He should just tell people to RTFM.
And his clan call us haters.
No. Ignorant maybe, but not haters.
Shades of lost productivity, near windows-like.
Yeah that's what happens when you adopt a system and spend more effort complaining than simply reading the manual.
It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.
It never ceases to amaze me how people can get their system in that kind of a state.
it caused me a ton of unnecessary unpaid work
Systemd can't help you with a
Re: (Score:1)
It was really fun trying to figure out why a system with a failed mount wouldn't shut down clean without a hard power off - because it couldn't unmount what it had failed to mount.
It never ceases to amaze me how people can get their system in that kind of a state.
While not a failed mount, I just ran into this shutdown issue yesterday.
Clean Debian 9 install into a newly created VirtualBox VM.
"shutdown -r now" hangs halfway through the process with systemd starting a 15 minute timeout counter because it can't unmount "/"
Seeing as the only changes I made after boot were to set the hostname and lookup the MAC address.
At that point I added a dhcp reservation with fixed IP (on another system of course) and then came back to reboot the VM and make sure it got the right IP.
Re: (Score:1)
While not a failed mount, I just ran into this shutdown issue yesterday.
Clean Debian 9 install into a newly created VirtualBox VM.
You are certainly not the first person to claim I fucked something up, but I am really really hoping you will be the first to tell me exactly what above was the fuckup and hopefully what to do instead. (Emphasis added)
Use Devuan [devuan.org] instead of Debian.
Re:Only one good use-case (Score:5, Informative)
//
There was no fine manual to read that I'm aware of when this happened. Just oddball notes all over if you googled about how to fix just this particular bug - which workarounds worked until the next update. You might not have been paying attention.
//
Got my system in such a state? What a jerk, you're proving my point. Systemd got it into that state, I did zero, nada, nothing but allow an update to happen.
//
I don't have an employment contract - I work for my own outfit, which is a physics lab. It needs a lot of fairly custom machines from little to huge and fast for data aq and control - which is needed as the fusion reactor I'm developing works well enough to make too many neutrons to be safe near....
//
If it wasn't a bug that affected a lot of people, why did they complain and get the snotty remark - and then list workarounds findable on google?
//
That lovely restart loop on slow starting things is well documented as well, and has a workaround involving setting a longer timeout, which is a nasty hack...
//
Pretty sure it's not ignorant to trust "security" updates. I wouldn't have known systemd existed, much less have had time to retrain on hard to find or nonexistent documents that were changing daily...I'm REAL sure that something they broke isn't my fault, and people who tell me otherwise might be the problem.
Crap Facebook uses crap systemd? Fits.... (Score:2)
Did it have a part in them getting hacked? Would not surprise me. These people are incompetent.