The Story Behind Google's In-house Desktop Linux (computerworld.com) 60
"For more than a decade, Google has been baking and eating its own homemade Linux desktop distribution," writes Computerworld.
Long-time Slashdot reader waspleg shared their report: The first version was Goobuntu. (As you'd guess from the name, it was based on Ubuntu.) In 2018, Google moved its in-house Linux desktop from the Goobuntu to a new Linux distro, the Debian-based gLinux. Why? Because, as Google explained, Ubuntu's Long Term Support (LTS) two-year release "meant that we had to upgrade every machine in our fleet of over 100,000 devices before the end-of-life date of the OS."
That was a pain. Add in the time-consuming need to fully customize engineers' PCs, and Google decided that it cost too much. Besides, the "effort to upgrade our Goobuntu fleet usually took the better part of a year. With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases."
So, when Google had enough of that, it moved to Debian Linux (though not just vanilla Debian). The company created a rolling Debian distribution: GLinux Rolling Debian Testing (Rodete). The idea is that users and developers are best served by giving them the latest updates and patches as they're created and deemed ready for production.
Google's using what appears to be an automated build system (along with virtualized test suites, and eventually "incremental canarying"), the article points out. The end result?
"The entire gLinux development team consists of a single on-duty release engineer position that rotates among team members."
Long-time Slashdot reader waspleg shared their report: The first version was Goobuntu. (As you'd guess from the name, it was based on Ubuntu.) In 2018, Google moved its in-house Linux desktop from the Goobuntu to a new Linux distro, the Debian-based gLinux. Why? Because, as Google explained, Ubuntu's Long Term Support (LTS) two-year release "meant that we had to upgrade every machine in our fleet of over 100,000 devices before the end-of-life date of the OS."
That was a pain. Add in the time-consuming need to fully customize engineers' PCs, and Google decided that it cost too much. Besides, the "effort to upgrade our Goobuntu fleet usually took the better part of a year. With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases."
So, when Google had enough of that, it moved to Debian Linux (though not just vanilla Debian). The company created a rolling Debian distribution: GLinux Rolling Debian Testing (Rodete). The idea is that users and developers are best served by giving them the latest updates and patches as they're created and deemed ready for production.
Google's using what appears to be an automated build system (along with virtualized test suites, and eventually "incremental canarying"), the article points out. The end result?
"The entire gLinux development team consists of a single on-duty release engineer position that rotates among team members."
Whoever infiltrates this distro's supply chain (Score:2, Insightful)
will have hit gold. With a little care and patience before deploying the payload, it doesn't even sound too complicated, what with the lone Google dude in charge or rolling out the updates.
Re: Whoever infiltrates this distro's supply chain (Score:4, Insightful)
Re: (Score:3)
It sounds like they have a canary system and automated testing to detect much of that. It might not be perfect, but it doesn't make it any worse than any other distribution.
A team of one? (Score:2)
will have hit gold. With a little care and patience before deploying the payload, it doesn't even sound too complicated, what with the lone Google dude in charge or rolling out the updates.
Hm, it seems your hypothesis (thought experiment) is based off this apparently paradoxical (but I suspect only poorly copy-edited) sentence that appears in both the summary and article:
If we remove the unnecessary modifiers, the sentence basically asserts: "The entire team consists of a single position that rotates among members of the team."
So is the team plural
Google Corporate Officer of the Day (OOD) (Score:2)
Doesn't the military have a system of rotating some thankless responsibility to the "Officer of the Day"?
Something like Colonel Potter barking at Hawkeye,
"Pierce, you are now in charge of keeping all the Linux boxes in the 4077 patched and up-to-date."
Am I missing something? (Score:1)
Why would Google have to upgrade their equipment every two years because of the distro? Wouldn't everything still work on the hardware they have?
Someone fill me in because either I'm missing something completely obvious (wouldn't be the first time) or Google is making more out of this than is necessary.
Re:Am I missing something? (Score:4, Informative)
"Upgrade every machine" means upgrade the Linux on it, not upgrade the actual hardware. Yes, that perhaps could have been worded more clearly. So, they're upgrading to the supported version to stay in support. (Just keepin' it real here: I keep an unsupported Windows 7 box around because it can still see SMB 1 shares of logs on our way-out-of-support Red Hat servers. Sec Eng folks "see" and and ask about it every few weeks and let me keep it so I can see the logs. The Red Hat boxes are upgrading next month, though and I'll gladly shut it down. Yay!)
Re: (Score:2)
Ok, thanks, that sounds *somewhat* better. But why worry about the supported version? It still works, right? It's not going to stop working just because it's no longer supported (ala your Windows 7 box). Does it matter if they're behind a year or two on the distro?
Again, to me, this sounds like Google is making more out of this than is necessary.
Re: Am I missing something? (Score:2)
Re: (Score:2)
Security updates and a consistent build environment so you don't add random bugs to the mix that are version-dependent.
Re: Am I missing something? (Score:3)
No support means you're screwed when you have an audit, penetration test or a breach. Also, you're looking at significant afterhours if there is a major issue.
Re: Am I missing something? (Score:3)
If you haven't installed security patches in the last 2 weeks you're vulnerable to something known publicly. Out of support means no more upstream fixes.
Re: (Score:1)
Never felt like turning on SMB1 support in W10?
Re:Am I missing something? (Score:4, Insightful)
I think I'm still missing something. Every Ubuntu LTS release since Ubuntu 12.04 has had five years of free support, with the option to pay for Extended Security Maintenance to extend that by two or five years. (Ubuntu 10.04 had a shorter support period for desktop use and no ESM.) Why did Google think they needed to update every two years rather than every five years?
Re:Am I missing something? (Score:4, Insightful)
I think it's the requirement to upgrade the distro version to get the latest package versions, not that support was removed from the distro after 2 years. Older releases wouldn't push to receive the bleeding edge versions of everything.
Re: (Score:2)
To upgrade from, say, 18.04 LTS to 22.04 LTS you need to install the interim release (20.04 LTS) first.
Re: (Score:2)
That may be, but it seems like they could just do those updates back to back. Sometime after 22.04 LTS comes out, they finalize the update process for both, and jump from 18.04 LTS to 20.04 LTS to 22.04 LTS -- the disruption to a given workstation is minimized, they only have to update every 4 years, and the update process should probably be heavily automated for any enterprise of that scale.
Re: (Score:2)
Because most LTS distros, while they're up-to-date on security fixes, are woefully behind on features. For instance, 18.04, which is good until April 2023 for support, has php 7.2 by default-- which hit EOL 20 months ago. Even with backports, there is no 7.4 option (which hits EOL in November).
Yes, there are 3rd party repositories (and I support those for some of my customers), having it in the base OS is rather nice.
I gave up on fixed version distros some time ago, because the LTS "stable ABI" paradigm w
Re: (Score:2)
Isn't that the definition of a traditional OS release, though? You get definite versions of the packages in it. You don't get newer features, but you also don't have to deal with your applications or libraries having breaking changes.
For my home system, I use Debian testing and an happy with the rolling release model. For systems I use at work, where we deploy an application and expect it to work for years, we use a traditional OS release.
Re: (Score:2)
And now we have come full circle to Google's justification to moving to their own rolling release of Debian. :)
Re: (Score:2)
No, we haven't. We've rehashed the difference between traditional releases and rolling releases. Did Google just not realize what they wanted from a desktop OS for the first however many years? Or did their requirements change sometime during those years?
Re: (Score:2)
Re: (Score:2)
Ubuntu LTS support is 5 years, not two. I don't know what this article is about but the whole premise is blatantly false.
Re: (Score:2)
Because there's a huge difference between "security updates" and "application updates". LTS gets the former, but not the latter.
Best part of story (Score:2)
I struggled to understand what the heck the summary was saying, so I actually read the article. My takeaway:
It is a shame that more of the work of making a distribution work reliably isn't as automated as it cou
My summary was like 80% of the actual article. (Score:2)
You can click my name to see what was actually submitted vs what showed up.
Re: (Score:2)
That's an interesting example of providing a summary...
Re: (Score:2)
I wouldn't do it at all if I could just submit stories. Google News has that shit automated - really most people only look at the title anyway.
Re:Best part of story (Score:5, Funny)
I struggled to understand what the heck the summary was saying, so I actually read the article.
That's a good way to get banned here.
Re: (Score:2)
Microsoft built a testing lab with thousands of computers and after a certain point "ate their own dog food" (running beta OS in production in some limited fashion), just to try and produce a stable release - Linux employs something kinda similar, but it more closely resembles an "infinite number of monkeys with an infinite number of typewriters..." than a professional, automated testing facility.
Something doesnâ(TM)t pass the smell test (Score:1)
Re:Something doesnâ(TM)t pass the smell test (Score:5, Informative)
In another story about a Google project, someone pointed out that to get to L7 or above [levels.fyi], you need to do some kind of project that affects the company beyond your own team.
That creates an incentive to do projects that are high profile and flashy, whether they are actually useful or not.
And for my L7 project... (Score:2)
I'm going to remove this unnecessary distro from our company :D.
Re: (Score:2)
An excellent plan. Think of all the complaints it will solve!
Re: Something doesnâ(TM)t pass the smell test (Score:2)
Re: Something doesnâ(TM)t pass the smell test (Score:2)
Right? I *really* canâ(TM)t imagine Google going âoeman, we really canâ(TM)t be arsed rolling out even one update every two years, letâ(TM)s change the distro so we can have a massively outdated OS on all our machines!â
Google's Linux - see also Debconf22 presentation (Score:3, Informative)
It exists, it works, they chose Debian because it worked - see also https://meetings-archive.debia... [debian.net] where much is explained.
Wasted effort. (Score:5, Interesting)
Imagine, instead, that they decided to base their rolling distro on another publicly available rolling distro, like Arch (mentioned in the article). They could make both technical and financial contributions to make Arch even more smoother to upgrade (as well as bugfixes, usability improvements etc).
Other users and even companies can see all the value that's been pumped into the project and will be more inclined to use it, which means it will be tested even more widely, resulting in bug fixes going back Google's way for free.
On the face of it, companies can "freeload" off of Google's efforts. But Google would be in greater control of the upstream simply by the sheer mass of their own contributions in comparison. Those companies will find themselves having to maintain their own separate distro, with the upstream "breaking" them all the time. So the only sensible option for companies is not to freeload and to contribute back to the upstream.
You get a virtuous circle of contributions that only improve on each other, creating value for each one involved - far more than what they could ever spend money on their own. Even Google.
But instead, Google wastes money on maintaining a distro all on its own.
Re: (Score:2)
I'm guessing this is just they want complete control, and won't promise to match any other system. I doubt they wrote it all from scratch (sorry didn't read TFA yet).
I get your point of putting that effort into a name that someone else already made, but then you inherit all their bad decisions. Past, present, and future.
Re: (Score:2)
But instead, Google wastes money on maintaining a distro all on its own.
Wastes money? How much could it possibly cost them? $20/googler? So what? It's cheaper than a MS software assurance license, and in case you forgot, Google already had an OS group in-house, working on Android...
Re: (Score:1)
"Cheap" does NOT negate whether it is WASTED or not. If something is unnecessary, then by DEFINITION it is a waste.
Re: (Score:3)
Sorry, but these assumptions are way off base.
A fleet of hundreds of thousands of Linux desktops would probably need some customization. They might want to ensure all updates go though another security review. They might want to have pre-installed in-house applications, or maybe some security settings. Who knows, they might even have a custom wallpaper in the mix.
And, maybe, just maybe, Google is actually one of the top upstream open source contributors out there: https://www.freecodecamp.org/n... [freecodecamp.org] (interest
Re: (Score:2)
The issue they have was NOT about customization but about, what ended up becoming yet another rolling distro, of which there already exist.
I could go on, but "it works fine" does NOT negate that it's a "wasted effort".
I thought Ubuntu LTS... (Score:3)
I thought Ubuntu LTS was supported for five (5) years after release, why the need to replace every install every two years?
A new LTS version is released every two years. In previous releases, a Long Term Support (LTS) version had three years support on Ubuntu (Desktop) and five years on Ubuntu Server. Starting with Ubuntu 12.04 LTS, both versions received five years support. There is no extra fee for the LTS version; we make our very best work available to everyone on the same free terms. Upgrades to new versions of Ubuntu are and always will be free of charge.
Source [ubuntu.com]
Re: (Score:2)
Common platform concerns? Maintaining currency while providing the update benefits? I don't quite understand how they would pull off a "major" upgrade, but I guess they thought of that...
Windows banned (Score:5, Informative)
The elephant in this room is that Windows is banned at Google because of the unacceptable attack surface, famously exploited by China.
Re: (Score:2)
Re: (Score:2)
extremely rare common sense
Proper System Admin... (Score:1)
If you know what you are doing, the number of servers you administer does not matter much.
Congratulations, Google. :)
Google is still the most untrustworthy company beh (Score:1)
Re: Google is still the most untrustworthy company (Score:1)
Re: Google is still the most untrustworthy company (Score:1)
Two year support window?! (Score:1)
They release an LTS every two years, yes, but they're supported for five. You're doing lots of extra work if you're updating every two years.
Re: (Score:2)
Re: (Score:1)