AlmaLinux Discovers Working with Red Hat (and CentOS Stream) Isn't Easy (zdnet.com) 73
After Red Hat's decision to only share RHEL source code with subscribers, AlmaLinux asked their bug report submitters to "attempt to test and replicate the problem in CentOS Stream as well, so we can focus our energy on correcting it in the right place."
Red Hat told Ars Technica they are "eager to collaborate" on their CentOS Stream distro, "even if we ultimately compete in a business sense. Differentiated competition is a sign of a healthy ecosystem."
But Red Hat still managed to ruffled some feathers, reports ZDNet: AlmaLinux Infrastructure Team Leader Jonathan Wright recently posted a CentOS Stream fix for CVE-2023-38403, a memory overflow problem in iperf3. Iperf3 is a popular open-source network performance test. This security hole is an important one, but not a huge problem.
Still, it's better by far to fix it than let it linger and see it eventually used to crash a server. That's what I and others felt anyway. But, then, a senior Red Hat software engineer replied, "Thanks for the contribution. At this time, we don't plan to address this in RHEL, but we will keep it open for evaluation based on customer feedback."
That went over like a lead balloon.
The GitLab conversation proceeded:
AlmaLinux: "Is customer demand really necessary to fix CVEs?"
Red Hat: "We commit to addressing Red Hat defined Critical and Important security issues. Security vulnerabilities with Low or Moderate severity will be addressed on demand when [a] customer or other business requirements exist to do so."
AlmaLinux: "I can even understand that, but why reject the fix when the work is already done and just has to be merged?"
At this point, Mike McGrath, Red Hat's VP of Core Platforms, AKA RHEL, stepped in. He explained, "We should probably create a 'what to expect when you're submitting' doc. Getting the code written is only the first step in what Red Hat does with it. We'd have to make sure there aren't regressions, QA, etc. ... So thank you for the contribution, it looks like the Fedora side of it is going well, so it'll end up in RHEL at some point."
Things went downhill rapidly from there...
On Reddit, McGrath said, "I will admit that we did have a great opportunity for a good-faith gesture towards Alma here and fumbled."
Finally, though the Red Hat Product Security team rated the CVE as "'Important,' the patch was merged.
Coincidentally, last month AlmaLinux announced that its move away from 1:1 compatibility with RHEL meant "we can now accept bug fixes outside of Red Hat's release cycle."
This Thursday AlmaLinux also reiterated that they're "fully committed to delivering the best possible experience for the community, no matter where or what you run." And in an apparent move to beef up compatibility testing, they announced they'd be bringing openQA to the RHEL ecosystem. (They describe openQA as a tool using virtual machines that "simplifies automated testing of the whole installation process of an operating system in a wide combination of software and hardware configurations.")
Red Hat told Ars Technica they are "eager to collaborate" on their CentOS Stream distro, "even if we ultimately compete in a business sense. Differentiated competition is a sign of a healthy ecosystem."
But Red Hat still managed to ruffled some feathers, reports ZDNet: AlmaLinux Infrastructure Team Leader Jonathan Wright recently posted a CentOS Stream fix for CVE-2023-38403, a memory overflow problem in iperf3. Iperf3 is a popular open-source network performance test. This security hole is an important one, but not a huge problem.
Still, it's better by far to fix it than let it linger and see it eventually used to crash a server. That's what I and others felt anyway. But, then, a senior Red Hat software engineer replied, "Thanks for the contribution. At this time, we don't plan to address this in RHEL, but we will keep it open for evaluation based on customer feedback."
That went over like a lead balloon.
The GitLab conversation proceeded:
AlmaLinux: "Is customer demand really necessary to fix CVEs?"
Red Hat: "We commit to addressing Red Hat defined Critical and Important security issues. Security vulnerabilities with Low or Moderate severity will be addressed on demand when [a] customer or other business requirements exist to do so."
AlmaLinux: "I can even understand that, but why reject the fix when the work is already done and just has to be merged?"
At this point, Mike McGrath, Red Hat's VP of Core Platforms, AKA RHEL, stepped in. He explained, "We should probably create a 'what to expect when you're submitting' doc. Getting the code written is only the first step in what Red Hat does with it. We'd have to make sure there aren't regressions, QA, etc. ... So thank you for the contribution, it looks like the Fedora side of it is going well, so it'll end up in RHEL at some point."
Things went downhill rapidly from there...
On Reddit, McGrath said, "I will admit that we did have a great opportunity for a good-faith gesture towards Alma here and fumbled."
Finally, though the Red Hat Product Security team rated the CVE as "'Important,' the patch was merged.
Coincidentally, last month AlmaLinux announced that its move away from 1:1 compatibility with RHEL meant "we can now accept bug fixes outside of Red Hat's release cycle."
This Thursday AlmaLinux also reiterated that they're "fully committed to delivering the best possible experience for the community, no matter where or what you run." And in an apparent move to beef up compatibility testing, they announced they'd be bringing openQA to the RHEL ecosystem. (They describe openQA as a tool using virtual machines that "simplifies automated testing of the whole installation process of an operating system in a wide combination of software and hardware configurations.")
Not surprising! (Score:2)
Re: (Score:2)
AlmaLinux is everything IBM owned Corporate Red Hat hates and is working to destroy. Choice! Monopoly is not dead. It just looks different in 2023.
How do you say 'cluster' where you're from?
Re:Not surprising! (Score:5, Interesting)
For-profit: I need to budget my financial resources and may not have the financial resourced to implement it.
Re: (Score:2, Troll)
Missing the point, they need to allocate resources to check and test the AlmaLinux fix. A low priority CVE is insufficient to trigger the resource-allocation process.
If you look at the Red Hat CVE you will see that they assigned this one a score of 7.5 rather than the original 5.5, they also upgraded the "Attack Vector" and "User Interaction" fields, presumably to justify the higher score.
Re: Not surprising! (Score:2)
Re: (Score:2)
Re: Not surprising! (Score:1)
Re:Not surprising! (Score:5, Interesting)
AlmaLinux is everything IBM owned Corporate Red Hat hates and is working to destroy. Choice! Monopoly is not dead. It just looks different in 2023.
Actually, Redhat+IBM (PurpleHat?) is more pissed about Oracle's Redhat clone (and the rumored Suze's Redhat clone).
Alma and Rocky are just collateral damage.
Re: Not surprising! (Score:1)
Re: (Score:2)
I've heard differently from people claiming to be inside Red Hat.
While I have no way to evaluate the truth of these claims, they ring true to me. I can tell you that Oracle's lawyers are among the very few that are justifiably feared more than those of IBM. I don't think there's any way the latter risks confronting the former, without some type of backroom negotiations and secret agreements.
Meaning, if I am correct, Oracle's clone will continue to be a thing, while Rocky and Alma are no longer allowed to
Easy (Score:1)
stop helping red hat with anything they can fix their own bugs and security holes
Re: Red Hat and Ubuntu (Score:2)
What with systemD and the fuckup called pulseaudio ("simplify" alsa but end up making it more complex and less reliable, genius!) I sometimes wonder if lennart is actually working for MS.
Re: Red Hat and Ubuntu (Score:4, Insightful)
In case you actually don't know, he does work for Microsoft since 2022.
Maybe he unofficially was all along...
Re: Red Hat and Ubuntu (Score:2)
So the guy that single handedly divided the Linux community is working for a company that has engaged in years of dirty tricks against Linux and the computing industry as a whole.
Re: Red Hat and Ubuntu (Score:2)
I dont follow his career. Fuck sake, explains a lot. He really is a grade A arsehole.
Re: Red Hat and Ubuntu (Score:1)
What is CentOS stream's purpose? (Score:4, Interesting)
It's very strange what's coming out of Red Hat these days. They keep talking about Centos Stream being the community project and RHEL is their proprietary product that is built from Stream sources with a bit of proprietary special sauce added. The they went on about how if a third party wants to take CentOS Stream, build something from it, and contribute to it, that is great and what open source is all about. Yet here we are with RH treating CentOS stream like it's their own special thing and others really shouldn't be playing with it. Very discouraging. RH probably should just shut centos stream down and funnel everyone to Fedora because that's what they seem to want. I've not been impressed with the tone of the communications coming out of RH in the last few weeks. Seems like CentOS stream is a place where people can submit free labor for a possible benefit to RHEL (if they happen to like it). At this rate it's probably easier to contribute to MS's open source projects.
Re: (Score:2)
Re:What is CentOS stream's purpose? (Score:5, Informative)
Re: (Score:2)
Sure.
However, I think they just lost the possibly much larger market of "we used RHEL for production, but the binarily-equivalent CentOS / Alma / Rocky for development and testing, and now we realize we need to containerize our software so we can use Debian, support anything else running kernel version X.x, and not have to worry about going through this shizznit ever, ever again."
Re:What is CentOS stream's purpose? (Score:4, Informative)
They keep talking about Centos Stream being the community project
The first important clarification is that The CentOS Project is described as being "community-driven," but the CentOS Project is broader than merely CentOS Stream. The project includes the Special Interest Groups that extend CentOS Stream and build new variants or projects on top of Stream.
and RHEL is their proprietary product that is built from Stream sources with a bit of proprietary special sauce added
The second important clarification is that RHEL does not have "proprietary special sauce added". RHEL (the software) is nothing more than periodic snapshots of CentOS Stream that receive continued support in the form of fixes for critical bugs and security issues.
RHEL is better understood as a support program. I don't mean helpdesk style "support-me-when-something-breaks" support. Support isn't something that exists only during incidents, support is a relationship. It's periodic meetings with your account manager and engineers. It's discussing your roadmap and your pain points regularly, and getting direction from them. It's the opportunity to tell Red Hat what your needs and priorities are, and helping them make decisions about where to allocate their engineers time to address the real needs of their customers. It's setting the direction for the company that builds the system that sits underneath your technical operations. That kind of support is what makes RHEL a valuable offering.
Because there is no special sauce added after the fact, CentOS Stream has to conform to the policies for RHEL updates. It is the major-version stable branch for RHEL.
To understand that concept better, consider: https://medium.com/@gordon.mes... [medium.com]
Re: (Score:2)
Except that it's not a support program. RH tried that and it nearly bankrupted them. RHEL is a proprietary product with an SLA that forbids exercising your open source rights to redistribute. It doesn't say it quite that way, but the fact is if you redistribute binaries or source, RH reserves the right to terminate the SLA. The SLA is the de facto equivalent of a seat license, complete with auditing. This is the special proprietary sauce. I'm not sure why people would want to contribute to CentOS strea
Re: What is CentOS stream's purpose? (Score:2)
Re: (Score:3)
I'm very serious. RH is in the business of selling proprietary licenses (not called that, but SLAs) to their proprietary product, which consists of open source software. Just don't try to redistribute their versions of it, though. MS has moved to an open source development model for many parts of their products now. The bulk is closed source of course, but everything from Edge to Paint to Visual Studio Code (arguably the ultimate successor to Visual Studio eventually) is open source. Might not be free
Re: (Score:2)
My understanding is that most of C# and .NET Core are in fact Free (MIT license) as well as open-source.
It is on that basis that I continue to willingly develop for, using, and sometimes even recommending this platform.
There is no proven, sustainable mechanism to ensure profit from the "sale" of large numbers, which is after all what any file of any type ultimately is. Service in the broadest sense is the part that people are willing to pay for, not ones and zeros.
Re: (Score:2)
Except that it's not a support program. RH tried that and it nearly bankrupted them
I see Red Hat employees make a statement generally similar fairly often, but they are talking about "helpdesk" style support, and right now I am talking about a much broader enterprise support relationship.
Re: What is CentOS stream's purpose? (Score:1)
Re: What is CentOS stream's purpose? (Score:2)
Re: (Score:3)
It's very strange what's coming out of Red Hat these days. They keep talking about Centos Stream being the community project and RHEL is their proprietary product that is built from Stream sources with a bit of proprietary special sauce added. The they went on about how if a third party wants to take CentOS Stream, build something from it, and contribute to it, that is great and what open source is all about. Yet here we are with RH treating CentOS stream like it's their own special thing and others really shouldn't be playing with it. Very discouraging. RH probably should just shut centos stream down and funnel everyone to Fedora because that's what they seem to want. I've not been impressed with the tone of the communications coming out of RH in the last few weeks. Seems like CentOS stream is a place where people can submit free labor for a possible benefit to RHEL (if they happen to like it). At this rate it's probably easier to contribute to MS's open source projects.
Simple, Fedora is the Alpha test and A/B testing ground for RHEL N+1 , meanwhile, CentOS Stream is the Beta test for RHEL N.x+1.
If you submit patches to CentOS Stream instead off to the relevant upstream projects, that is more burden to Redhat, so, they will only take said patches if those patches are relevant to them.
This patch was not relevant to them, until al the bad PR made it relvant.
Re: What is CentOS stream's purpose? (Score:2)
To break compatibility with RHEL.
Re: (Score:3)
Re: What is CentOS stream's purpose? (Score:1)
IBM Processes (Score:3)
This has IBM's footprint all over it. I had to interface with IBM in a project around 5-7 years ago and they had stringent rules as to when they were allowed to do anything, even something which obviously needed doing.
Once that project ended the hardware supplier wanted to remove the the (leased) hardware that had been in use there. They could not get access to the building because they were no longer suppliers to that business unit (or probably IBM altogether). Afaik the hardware was there for a while, powered up (the UPS had caught fire once because it was overloaded), taking up space they needed for other purposes. I imagine they resolved the situation eventually, hopefully without the hardware supplier going through the courts to get their hardware back, I had left there and lost social contact with the people I had worked with there when the Covid rules hit.
Towards the end of that project we'd only go through their processes if it benefited our group, if a change was needed to save them money we'd tell them informally and let them decide what to do.
Most of the IBM people there had been taken over from another company and a lot of them jumped ship because they just could not work like that. Someone told me virtually all of the others were subsequently released when some period of time had elapsed - part of the contract had been to keep these new employees on for n years and their time was up.
Re: (Score:2)
Re: (Score:2)
Various RH employees have independently stated several times over the last month that there has been no, zero, nadda, communication from or with anyone up in the IBM corporate structure about any of this. No, this originates from RH. And it's been over a decade in the making. The Software Freedom Conservancy has spoken several times over the years about how RH's service agreements---to not redistribute RHEL binaries or source despite open source licensing allowing this---are in a very grey area of GPL a
Re: (Score:2)
When you say "this", do you mean this specific CVE or the more general policy change wrt. availability of sources?
My experiences are with IBM, my last encounter with Red Hat was back in the 1990s and my comment reflected that the language used to initially avoid adopting this fix came straight out of the IBM cookbook I encountered a few years back.
Re: IBM Processes (Score:1)
Re:IBM Processes (Score:5, Informative)
> service agreements---to not redistribute RHEL binaries or source despite open source licensing allowing this---are in a very grey area of GPL
Sorry, but it's not grey - that's nonsense.
Re: IBM Processes (Score:1)
Re: (Score:2)
Because they are very careful in their language and the fine print. The source *is* available to their customers, just as the licenses require. And customers *can* redistribute binaries and source and RH won't sue them over that. But RH very much can cut off their contracts, leaving them in the cold. Technically the GPL is not violated. Think of it as something similar to what Tivo did. They comply with the letter of the GPL (and other copyleft licenses) but not the spirit.
Also i don't know of any compa
Re: (Score:2)
I'm of the view that what happened is probably a GPL violation, but as for other code, developed under non-copyleft licenses such as BSD or MIT, RH/IBM is within its rights to not distribute their modified versions of code. Nothing in those licenses would prevent it.
Which I think demonstrates that, at least under some circumstances, copyleft licenses are a Good Thing.
Re: (Score:2)
I'd say it's a grey area . . albeit a very, very dark shade of grey . . until it is tested in court.
And, worst case, they've just taught us how to fix the next version of the GPL so this can't ever happen again.
Re: (Score:2)
Most of Red Hat's public comments in TFS sound like boilerplate PR/sales speak, so we're left to read between the lines. But one gets the impression that even the day to day work being done by the actual technical folks is significantly and forcibly constrained by some draconian business policies.
It's probably not a very pleasant place to work... hopefully they're at least well compensated.
Steven Vaughan-Nichols (Score:5, Interesting)
It's important to note when discussing Steven Vaughan-Nichols writing that he works with Cathey Communications, which is a firm that provides PR services to CIQ, the company behind Rocky Linux.
https://twitter.com/GordonMess... [twitter.com]
That's important because it tends to explain why, when he writes about Alma, he emphasizes that they've "moving away from 1:1 compatibility", but when he writes about CIQ developments that are forks of Red Hat products, they're a "Rocky-friendly" fork.
The way he frames the development of forks tends toward praise in CIQ's case, and FUD in Alma's case.
We used to refer to this sort of thing as AstroTurfing. It looks like grassroots support, but it's manufactured.
It also tends to explain why, in this article, Steven uncritically quotes people trolling in Red Hat's GitLab merge requests, extending more legitimacy to their toxic behavior than he does to the actual developers.
Re: Steven Vaughan-Nichols (Score:1)
Re: Steven Vaughan-Nichols (Score:2)
"Fuck You and the patch you rode in on!" (Score:1)
That seems to be Red Hat's response to this. And if I were a Red Hat customer, I'd be really pissed to know that Red Hat would willingly distribute software with ANY known vulnerabilities in it, where a patch exists. Yeah, they have to test/verify it. But waving a finger at the submission is not the same as "OK, we'll fix this."
Re: (Score:3)
I started looking elsewhere when they broke chrooted bind in the middle of a stable cycle and would not take a tested, complete, and verified community patch for ... what was it, 14 months?
It was an abject rejection of the open source ethos. And it's wasn't even a security patch which could break systems, theoretically - they fucking broke everybody's systems and were happy to leave it like that for over a year.
So now all my clients run Debian and they don't buy RHEL licenses anymore.
Thanks for all the Swe
Disappointing (Score:1)
Wait until RedHat starts part 2 of their evil Plan (Score:4, Interesting)
Begining with RHEL 10, all source code for BSD, MIT and Apache licensed code will ONLY be available for Fedora. Said code will not be guaranteed for CentOS, and if you are a RHEL client and want the code specific for your instance, you need to engage your account representative, or follow this (cumbersome?) automated procedure.
Wait until Oracle and Suze (and alma and rocky) have to say to potential users tha want some sort of Redhat compatibility that they are not sure if that, for big portions of their code for OpenSSH(BSD), OpenSSL (ASL2.0), NFS4(BSD), Pyton3 (BSD+ASL2), just to name a few examples, not only they (Oracle, suze, Alma, Rocky) are not sure if the code is the same, but even if the code is the same, they do not know the compiler used, or the compile time parameters used...
Bye bye Oracle Clone. Bye Bye (rumoured) Suze clone, and sorry Alma and rocky, you were collateral damage.
Whanna see how much of RHEL 9 is covered by BSD/MIT/Apache? See here:
https://access.redhat.com/docu... [redhat.com]
PS: Do not hate me for predicting the future, I do not like what Redhat is doing, but I think the license allows them to do it, and I understand why they are doing it.
Re: Wait until RedHat starts part 2 of their evil (Score:1)
Not as unreasonable as it sounds (Score:1)
The claim that CentOS is community driven is definitely a bit of corporate misdirect. CentOS started as a free version of RHEL and that's what it remains, I doubt it took many custom patches before either, meaning the patch would have to go to RHEL.
As for the resistance to patching RHEL, RHEL is a very boring distro by design. It doesn't like updates unless absolutely necessary because customers don't like updates (every update is a risk that things break). And Red Hat doesn't like custom patches because th
Re: Not as unreasonable as it sounds (Score:1)
Let the forking begin (Score:2)
And so it starts. One side accepts patches, and the other does not. Red Hat is going to go down the tubes.
IBM just had to screw up a perfectly fine Linux company.
Red Hat needs to be an example, IMO. (Score:2)
Re: Red Hat needs to be an example, IMO. (Score:1)
Re: (Score:2)
I'm not opposed to companies trying to earn a profit, SO LONG AS they do so lawfully and ethically.
I don't think Red Hat did in this case. In part because the state of ethics and rule of law in our society is dismal, and IBM/RH took advantage of this fact to try to open up a new revenue stream (current CentOS/Alma/Rocky users) which (a) does not exist, and (b) will alienate a great number of its current revenue stream.
This greatly strengthens the case for a genuine community-based OS supported both by the
This seems reasonable to me (Score:4, Insightful)
Getting the code written is only the first step in what Red Hat does with it. We'd have to make sure there aren't regressions, QA, etc.
He's completely right. Every time you "fix" something, there's a risk of creating a new problem in the process. You need to weigh the problem you're fixing against the risk of creating new problems. You can mitigate that risk through testing and code analysis, but that costs time and money. Sometimes the right answer is not to accept a fix. You don't automatically accept every fix that's submitted. You can agree or disagree about whether they made the right call in this case.
If you don't like the decisions they make, that's why we have lots of distros. The managers of each distro decide what to include, and users decide which to use.
Re: This seems reasonable to me (Score:2)
Re: (Score:2)
Sure. It used to be true that RHEL led the RPM distros, but that's likely to be changing in the mid-term with decisions like this. Tempting to call it a false economy, but really it's just them figuring out whether they want to lead a fleet of distros or economise and run a single distro that might not be compatible with others. Sounds like the latter.
Who knows if, in the long run, the costs of keeping third-party software compatible with RHEL will be higher or lower, once it's no longer a leader. I guess w
Re: This seems reasonable to me (Score:1)
Re: (Score:2)
SuSE seems willing. It's possible we'll see opensource efforts take the lead instead, like the Debian project did on their side of the wall.
Re: (Score:2)
My policy is now that when possible I will package:
* Server software in the form of containers, with no dependencies on any specific Linux distro beyond the kernel; and
* UI components as either Web-based, or as Flatpaks or AppImages, likewise not depending on any specific Linux distro.
I will test first and foremost with Debian stable, then also with more recent kernels so as to maximize hardware compatibility.
I recommend that others strongly consider doing the same.
Fool me once
Just like Debian, not bad! (Score:3)
I am beginning to think with a little more funding, Debian outshine Red Hat. They have even made leaps and bounds with their targeted SELinux policies, including implementing the foundation for confining some common desktop apps in the future, something Dan Walsh and Red Hat has clearly given up on for native packages, despite Wayland making it possible.
In any case, the future can only hold good things if Alma is planning to break full compatibility and compete with additional fixes to packages. After all, some knock-offs are simply better than the real thing!
Re: (Score:2)
Systemd was one of the few reasons I did not go all-in on recommending Debian.
It now seems like a relatively minor problem, compared to being forced to change distros because of unethical and at least borderline-unlawful behavior on the part of an OS vendor.
I'm going with Debian + containers + Flatpaks or AppImages (not snaps) for as much stuff as I possibly can going forward, although I use Gentoo at home, which avoids the problem in a different way. I don't recommend it for production deployments because
Why report bug to Red Hat? (Score:2)
Which I why I bailed.... (Score:1)
As a former RHCE (Red Hat Certified Engineer) I have bailed on Red Hat Linux. The death of CentOS forced me into Ubuntu land and I an not happy about it.
Sure, CentOS may have been taking some sales from RHEL but it really made people in the community unhappy.
Nothing ever works in CentOS stream, it is bleeding edge and you can barely use it as a hobbyist.
Trying getting Stabile Diffusion going in CentOS, not happening.
Re: (Score:2)
RHCE here too. I stopped using RHEL or its derivatives the day I couldn't install a RHEL test box without a license.
I don't understand why people haven't just ditched it completely?
RHEL is just for work, use Debian for everything else. Problem fixed. Using Fedora or CentOS whatever just helps Red Hat.
Re: Which I why I bailed.... (Score:1)