'Uncertainty' Drives LinkedIn To Migrate From CentOS To Azure Linux (theregister.com) 79
The Register's Liam Proven reports: Microsoft's in-house professional networking site is moving to Microsoft's in-house Linux. This could mean that big changes are coming for the former CBL-Mariner distro. Ievgen Priadka's post on the LinkedIn Engineering blog, titled Navigating the transition: adopting Azure Linux as LinkedIn's operating system, is the visible sign of what we suspect has been a massive internal engineering effort. It describes some of the changes needed to migrate what the post calls "most of our fleet" from the end-of-life CentOS 7 to Microsoft Azure Linux -- the distro that grew out of and replaced its previous internal distro, CBL-Mariner.
This is an important stage in a long process. Microsoft acquired LinkedIn way back in 2016. Even so, as recently as the end of last year, we reported that a move to Azure had been abandoned, which came a few months after it laid off almost 700 LinkedIn staff -- the majority in R&D. The blog post is over 3,500 words long, so there's quite a lot to chew on -- and we're certain that this has been passed through and approved by numerous marketing and management people and scoured of any potentially embarrassing admissions. Some interesting nuggets remain, though. We enjoyed the modest comment that: "However, with the shift to CentOS Stream, users felt uncertain about the project's direction and the timeline for updates. This uncertainty created some concerns about the reliability and support of CentOS as an operating system." [...]
There are some interesting technical details in the post too. It seems LinkedIn is running on XFS -- also the RHEL default file system, of course -- with the notable exception of Hadoop, and so the Azure Linux team had to add XFS support. Some CentOS and actual RHEL is still used in there somewhere. That fits perfectly with using any of the RHELatives. However, the post also mentions that the team developed a tool to aid with deploying via MaaS, which it explicitly defines as Metal as a Service. MaaS is a Canonical service, although it does support other distros -- so as well as CentOS, there may have been some Ubuntu in the LinkedIn stack as well. Some details hint at what we suspect were probably major deployment headaches. [...] Some of the other information covers things the teams did not do, which is equally informative. [...]
This is an important stage in a long process. Microsoft acquired LinkedIn way back in 2016. Even so, as recently as the end of last year, we reported that a move to Azure had been abandoned, which came a few months after it laid off almost 700 LinkedIn staff -- the majority in R&D. The blog post is over 3,500 words long, so there's quite a lot to chew on -- and we're certain that this has been passed through and approved by numerous marketing and management people and scoured of any potentially embarrassing admissions. Some interesting nuggets remain, though. We enjoyed the modest comment that: "However, with the shift to CentOS Stream, users felt uncertain about the project's direction and the timeline for updates. This uncertainty created some concerns about the reliability and support of CentOS as an operating system." [...]
There are some interesting technical details in the post too. It seems LinkedIn is running on XFS -- also the RHEL default file system, of course -- with the notable exception of Hadoop, and so the Azure Linux team had to add XFS support. Some CentOS and actual RHEL is still used in there somewhere. That fits perfectly with using any of the RHELatives. However, the post also mentions that the team developed a tool to aid with deploying via MaaS, which it explicitly defines as Metal as a Service. MaaS is a Canonical service, although it does support other distros -- so as well as CentOS, there may have been some Ubuntu in the LinkedIn stack as well. Some details hint at what we suspect were probably major deployment headaches. [...] Some of the other information covers things the teams did not do, which is equally informative. [...]
Switching flavors of linux sucks (Score:5, Insightful)
Re:Switching flavors of linux sucks (Score:5, Insightful)
I have no idea why anyone would stick with CentOS now.
Re:Switching flavors of linux sucks (Score:5, Interesting)
Red Hat made a lot of insane decisions since their purchase by IBM. And they've since lost a lot of the sensible people who kept it a stable and attractive operating system.
Re:Switching flavors of linux sucks (Score:5, Informative)
Re:Switching flavors of linux sucks (Score:5, Funny)
Honestly that sounds like taking your balls out of the fire and quenching them in some boiling oil.
Re: Switching flavors of linux sucks (Score:2)
You will always always always get burned trustng anything Oracle.
Re: (Score:2)
I understand where you're coming with that, but it doesn't really apply to Oracle Linux specifically.
Oracle Linux is just RHEL repackaged, just like CentOS, RockyLinux and AlmaLinux. They do change it up a bit with their own "unbreakable" kernel, but I don't know that this is really any better than the stock kernel, though I'm pretty sure if I asked they'd tell me a thousand ways that it was.
In any event, if Oracle Linux changes in some way that you don't like, you just jump ship to the equivalent version
Re: (Score:2)
Oracle have form for changing the terms from underneath you. Either for the sources or the binaries. Look at how they fucked over Java users between 8 and 11 - runtime binaries that were previously free were now not and it opened up companies to suprise audits by Oracle's inquisitors. It's the reason that many orgs now use Corretto instead which is binaries built from the same sources but not by Oracle.
Re: (Score:2)
Yes, but you're missing my point, and that is Oracle Linux is a special case. They *can't* change the sources or the binaries (or the license), because Oracle Linux is just a repackaged RHEL, which is under the GPL.
With Java they could, because they wrote it and had all the rights, but not Oracle Linux.
All they could do with Oracle Linux is stop supporting it, or change future releases to be something other than a repackaged RHEL -- which anybody else repackaging RHEL could also do, and CentOS actually did
Re: (Score:3)
And that kids, is how I met your mother.
Re: (Score:1)
Re: (Score:2)
We did not need it. Some of our MySQL queries sped up on identically configured VMs and some of our PHP workloads slowed down
I still maintain a central beefed up MySQL VM for all php vms using MySQL. It makes it easier to find out which vm does stupid queries :)
Sometimes, I wonder if it's worth anymore. Any expert opinion around here in a virtualization context where the central database runs on the same bare metal hardware than the vms, that you can use ksm to deduplicate memory and that you can use zfs to deduplicate storage space?
Any opinion is welcomed in advance.
Re: (Score:2)
Oh, easier to manage pseudo-real time replication too thinking of it...
Re:Switching flavors of linux sucks (Score:5, Funny)
Blink twice is if an Oracle rep is standing behind you.
Re: (Score:2)
But can anything top the insanity of moving from Redhat to Microsoft?
Re: (Score:2)
But can anything top the insanity of moving from Redhat to Microsoft?
Changing from IBM to Microsoft has been a cost-effective move for most of history. It's difficult to say which is worse today, unless it's both of them. I see this as a lateral move, which does make it dumb, but not necessarily insane.
Re: (Score:2)
It's not insane if you happen to be Microsoft. It's not like your vendor is going to screw you. But it's Microsoft, so who knows?
Re: (Score:2)
I wouldn't call it insane but short minded greed. Surely they expected some migrations to paid RHEL.Maybe they even got some.
Re: (Score:2)
To be fair, a company with the net worth of LinkedIn should be paying for their damn RedHat licenses. CentOS was unlicensed redhat thst benefited small companies where a redhat license was encumbering to the business model. Like say running your small 100 or less user phone system. Running your small website, etc. CentOS was 100% binary compatible with RedHat, meaning a paid for version was always an option.
Re: (Score:2)
I have no idea why anyone would stick with CentOS now.
CentOS is dead now. The forums were taken down, with all their history.
Re: (Score:2)
To be fair, CentOS's decision not to provide a migration path from 7 to 8 or even stream was absolutely insane.
I have no idea why anyone would stick with CentOS now.
There's Rocky Linux now.
Re: (Score:2)
There's also a big difference between using a distro maintained by someone else, and being the maintainers of your own distro. Part of the reason is probably CentOS being EOL'd, and another might be the chance to add features they need to integrate with WSL more tightly.
Plus, if they can offer support subscriptions to their version for half the cost of a Redhat subscription, perhaps they are exploring that as a new revenue stream?
Linux365 anyone?
Re:Switching flavors of linux sucks (Score:5, Interesting)
Since they are forced by their owners to use Azure I guess that Azure Linux makes some sense but I've never had a pleasant experience with rpm based distros (RedHat, Fedora, CentOS, Amazon Linux, ... to name a few).
The Debian based distros have always been a breeze in comparison and even a few years ago we were forced to switch our AWS EKS nodes to Ubuntu because the Amazon Linux nodes had a known kernel bug, fixed upstream for years, that caused our containers to drop network connectivity. That kernel bug had been fixed for years but RH based distros have very old kernels with a bunch of back ports. It mostly works but having to deal with years old bugs or not be able to use features released years earlier is just frustrating.
If you want RedHat because you need commercial support, pay Canonical, I don't have first hand experience but I heard their support teams actually help. My experience with RedHat support is similar to my experience with Azure support: waste of time.
Re: (Score:2)
Problem is Microsoft has too much motivation to strangle Linux to death with its own propri
Re: (Score:2)
Microsoft might have a cloud but they are not known for quality of that cloud or on any of their products.
Ubuntu just works. So does Debian. And the new CentOS alternatives Alma and Rocky. There is no reason to go use a Linux distribution from a company that would have to make vacuum cleaners in order to make something that doesn't suck.
Re: (Score:3)
Article author here.
> There is no reason to go use a Linux distribution from a company that would have to make vacuum cleaners in order to make something that doesn't suck.
Sure there is. There are several.
* You are already a customer of that company;
* You want to minimise your number of suppliers;
* You have already paid (for support or for hosting or whatever) and you don't want to pay more;
* You have already got investment in that company and want to keep it
* Guaranteed compatibility and/or interop
Those
Re: (Score:2)
IMHO, those are all of the worst excuses to use a specific piece of software. Excuses, not reasons. Dunno how many times a company (usually one I was working for) has purchased some software (or licenses, or got free seats with some other purchase), and that was enough for them to foist it on everyone and have everyone migrate off of the existing, and usually either inhouse developed or open source/free/no license needed implementations, to said software. And we'd often be limited on admin seats then, where
Re: Switching flavors of linux sucks (Score:3)
In case you missed it, Microsoft owns LinkedIn this is why they have been migrating to Microsoft solutions for the past few years. They migrated part of their infrastructure to Azure but the migration was so bad that they have been allowed the pause that for a while.
GitHub, also owned by Microsoft, runs part of their infrastructure in Azure now. For example GitHub Actions run on top of Azure Pipelines.
You can see how well that works for them by looking at their status history.
https://www.githubstatus.com/h. [githubstatus.com]
Re: (Score:2)
In case you missed it, Microsoft owns LinkedIn this is why they have been migrating to Microsoft solutions for the past few years.
Oh, I didn't miss it :-)
THAT sounds like the actual reason. All those other excuses are just that - BS excuses.
I personally suspect that part of Microsoft cloud growth is boosted by acquiring companies with loads of servers, and force them to use Azure.
Yup. Acquiring _OR_ using those stupid excuses to convince others to consolidate into their cloud.
I do think there are plenty of great use cases for the various cloud offerings and services. The one that breaks my brain is moving established, large deployments from data centers one already owns and operates into the cloud. You're going to pay more long term; Might as well invest that into yourself
I'm sure "in-house" is what won the day (Score:5, Informative)
But they'd have had a much easier path had they just shifted to AlmaLinux (or Rocky, for that matter).
Re: (Score:2)
If you're right, I'd say that was a pretty valid reason, because they can control its development. As long as they're using somebody else's Linux, the OS might change in undesirable ways, and MS might not be able to stop it.
Re: (Score:2)
But they'd have had a much easier path had they just shifted to AlmaLinux (or Rocky, for that matter).
I came on here to say this. They should have just gone to AlmaLinux.
Re: (Score:2)
[Article author here]
> a much easier path
But it's not always about what is _easy_. It's about what is the long-term better course for your company.
Re: I'm sure "in-house" is what won the day (Score:2)
The long term better course is to use the best product, because otherwise you will be bitten down the road. Fail to plan, plan to fail.
Re: (Score:2)
Re: I'm sure "in-house" is what won the day (Score:2)
Naturally it makes a certain amount of sense for a Microsoft owned company to use the Microsoft product. But that's not for technical reasons, so it will wind up leading to more pain in the future. In fact, Microsoft's special Linux for Azure has caused problems repeatedly, because... Microsoft.
Re: (Score:2)
Wait! What? (Score:4, Funny)
They jumped right over Fear and went straight to Uncertainty? Soon, it will be nothing but Doubt.
Re:Wait! What? (Score:5, Funny)
They jumped right over Fear and went straight to Uncertainty? Soon, it will be nothing but Doubt.
Tough economy these days, why buy the whole set when you only need the one?
Windows 1? will be linux (Score:3)
I've been saying it for years, but they're slowly moving towards windows being an abomination linux distro.
sadly that will be the year of the linux desktop. But itll still be called windows on the stat sheet.
Re: (Score:3)
The main hook of Windows is backwards compatibility. If the next version is just a reskinned Linux, it will have to rely on Wine (and have no advantage over any other distro), or MS will have to create a new proprietary solution that blows Wine away.
Re: (Score:3)
They won't need Wine. Windows already runs in a VM for most people (Virtualization-Based Security is now the default in Windows), and Linux running in WSL is also doing that as well. Windows is still treated as the primary for all this, but if they decided to shift to Linux being the primary, it would probably just involve a reversal of roles.
Re:Windows 1? will be linux (Score:4, Interesting)
Indeed. And there are rather strong indicators MS has lost control of the Windows codebase and can now only do cosmetics for fear of breaking too many things. Hence they need to scrap it and not in the distant future. The other factor is that MS makes most of its money with Azure these days.
Re: (Score:2)
Scrapping it is difficult. Eventually the technical debt swirling around in the middle of the Pacific needs to be cleared up, so it's time to fix it. But now you get the set of requirements which claim that thou shalt support 16-bit handles, thou shalt not fix this set of bugs because that will break everyone's workarounds, and thou shalt keep the CEOs favorite feature. It's honestly better to start over with a new product.
I see this problem in my own job. We try to get rid of a feature that we're confid
Re: (Score:2)
Well, as an IT security person, I think scrapping Windows will become necessary and without alternative in the not too distant future. At some point MS will get liability for its constant screw-ups and that will be it for Windows. I do not think Windows can be cleaned up to meet modern standards and the "innovations" MS has done in the last 10 years or so show this pretty clearly.
That said, you can still run it isolated if you really need to and there are some pretty good emulation layers around.
Re: (Score:2)
But it also causes slow progress to Windows because it always had to have some long term backwards compatibility. Of course, simultaneously, Windows will dump new stuff just because they were in that mood while keeping really ancient architectures in place just in case someone might want to run an ancient executable. So forget having a reliable long term backup solution because they will toss it out and reinvent it every couple of releases, but the 16-bit real mode support stuck around far long than made
Re: (Score:2)
This is some weird fantasy that a lot of people have, but it flat out won't happen. The whole point of Windows is perpetual compatibility. If MS made a Linux flavour it would have to work perfectly and performantly with Windows binaries, and windows DLLs, and all the things which makes Windows, Windows. That is just not something that makes sense to work on.
And this from a company that can't even get partially decent ARM emulator running.
Re: (Score:1)
Most of Windows in our days is C# / .Net
Good luck running a game like WarLords on modern Windows.
The 'hype' of backward compatible is completely overrated.
You hardly find a software older than ten years that runs on current Windows.
Mac's or Linux: a completely different story.
Re: (Score:3)
Warlords was released in 1990, so in all likelihood it's a 16-bit app. If that's the case, more than anything else, would be the primary reason it won't run on modern versions of Windows. The latter are all 64-bit, but support running 32-bit applications through Windows on Windows (WOW) thunking. On a 32-bit OS WOW can thunk down to run 16-bit applications, but the last version of Windows with a 32-bit release was Windows 10 190x (out of support), 1809 (available as LTSC), or Server 2019. The last two a
Re: (Score:3)
Re: (Score:1)
Nice :)
Re: (Score:2)
They were able to switch to the NT kernel. They were able to get Windows on ARM running (even if that limited application compatibility). They long ago ported SQL Server to Linux. Was .Net not meant to replace all the old cruft? (https://xkcd.com/927/) They even have their own Linux distribution, and run Linux for much of their infrastructure.
I'd call it fear more than a fantasy, but it's certainly within the realm of possibility, and especially so if we look very far out.
Re: Windows 1? will be linux (Score:2)
"The whole point of Windows is perpetual compatibility."
Then it has no point. Microsoft severely compromised backwards compatibility with Vista and again with 64 bit, and the situation has become only worse since. Even apps which technically work frequently don't quite work correctly in many cases. I remember bringing up Excel 97 on multi monitor Windows 7 and having pull down menus appear on the primary display even when the windows were on the secondary display. Classy!
Re: (Score:2)
True. But it will take a while still for them to realize. And it will take a lot of money and effort to remove the crappy MS technology.
Uncertainty? (Score:2)
Why "uncertainty"? CentOS 8 is clearly not a successor to CentOS 7, Rocky 8 is. Alma is another one. CentOS = 7 was a bug compatible clone of RHEL. Rocky is the new CentOS (RHEL clone). Maybe the uncertainty is what will IBM and RH do next in their attempts to thwart the RHEL clones? Ultimately, they had to get off the EOL CentOS 7, and probably there was an expectation to consolidate on to Microsoft's own tooling.
Re: (Score:2)
Fucking /. silently eating characters and lack of preview on the mobile version. Where I wrote "CentOS = 7", it should be "CentOS <=7".
FUD (Score:2)
Microsoft's in house Linux? (Score:2)
Did I wake up in the wrong universe?
Re: (Score:2)
Did I wake up in the wrong universe?
Yes. This one has Microsoft in it. Sorry.
'Embrace, Extend, Extinguish' (Score:2)
Re: (Score:2)
Doesn't seem the real reason (Score:2)
Sounds more like there's an attempt here to put Azure in the headlines, first for the service they sell, and second to try and reinforce their adoption of linux (the thing that runs the service on their platform).
I'd be more impressed if they were working closer with the distros to upstream hyper-v - can you move a hyper-v into KVM - genuinely I don't know - but that'd be more sane. Instead hyper-v (AFAIK) runs only with WinNT but the Linux kernel can be a guest.
Re: (Score:2)
[Article author here]
> there's an attempt here to put Azure in the headlines,
Nah. Read the whole article.
They've tried this at least once before, at the end of last year, and very publicly failed.
This is not about "hey look at us" because they did that before and it was not good. It is more like "hey, we made it work, listen, here's how".
Re: (Score:2)
There's no such thing as bad publicity, evidently.
Oh hey, that's one of mine! (Score:2)
Cool to see it on the front page of /. Thanks!
another ad (Score:3)
So LinkedIn (a Microsoft owned company), finds Azure (a Microsoft company), has a more stable Linux?
This is just a weird ad. They're only switching because of corporate policy.
Diss Track (Score:2)
There are plenty of good reasons for a Microsoft division to adopt Microsoft linux. It would be really stupid if they didn't. XFS as an example might have kept other potential customers away.
RHEL has plenty of problems (plus racial-discrimination lawsuits) and the known problems are not new. But to hear a Microsoft blog complain about IBM's competitor product
while promoting their own is nothing to be taken seriously especially since they uaed the U in FUD in their lede.
Oh, and the free version of the comp
Re: (Score:2)
... XFS as an example might have kept other potential customers away.
I don't get that part at all. I've run XFS for ages on multiple distros, and moved drives and volumes between distros, and have never had a single problem with that. They're already changing out the OS, so the OS disk can move to whatever FS (ext4, bcachefs, btrfs, xfs, etc..). OS taken care of, they can just modprobe in XFS support (assuming it's not built into the kernel already) and mount the XFS stuff. (FWIW, no - I didn't RTFA)
Closest I had to a problem like that was with ReiserFS (way back in the day
Azure Linux uses signed device drivers (Score:2)
One of the most eye-opening factoids, at least for me, is that Azure Linux uses signed device drivers and as such can't or won't use kernel modules.
Re: (Score:2)
Microsoft choosing a Microsoft product, Shocking (Score:2)
Forced to migrate... (Score:3)
CentOS 9 Stream is actually OK (Score:2)