Amazon Cloud Traffic Is Suffocating Fedora's Mirrors (phoronix.com) 52
Michael Larabel reports via Phoronix: A massive uptick in traffic to Fedora's package mirrors is causing problems for the Linux distribution. Some five million additional systems have started putting additional strain on Fedora's mirror resources since March and appear to be coming from Amazon's cloud. Stephen Smoogen of Red Hat wrote a blog post today around 5+ million more EPEL-7 systems beginning in March. Fedora hosts the packaging mirrors for Extra Packages For Enterprise Linux (EPEL) to augment the package selection available on RHEL, CentOS, Amazon Linux, etc.
The past three months now there has been a 5+ million surge in Fedora/EPEL traffic and it's placed a strain on the systems. It's about doubling the number of unique IPs connecting to the mirror system. The massive uptick in Fedora/EPEL activity puts additional pressure on Fedora web proxies for mirror data and then the mirrors themselves that tend to be volunteer run. Much of this new traffic is coming from the Amazon/AWS cloud.
The past three months now there has been a 5+ million surge in Fedora/EPEL traffic and it's placed a strain on the systems. It's about doubling the number of unique IPs connecting to the mirror system. The massive uptick in Fedora/EPEL activity puts additional pressure on Fedora web proxies for mirror data and then the mirrors themselves that tend to be volunteer run. Much of this new traffic is coming from the Amazon/AWS cloud.
We're all suffocated by Amazon (Score:5, Insightful)
It's not just Fedora.
Re: (Score:2)
Of course, today, I wouldn't have any mod points.
If only the actual Amazon was as pervasive.
Re: (Score:2)
Yeah, our public time server gets absolutely hammered when Amazon users en masse decide to use it.
You'd think for such a large company they'd be hosting common infrastructure (like mirrors) for the betterment of the internet. That little thing they're making *billions* from.
Re: (Score:2)
Why would they do that?
By letting others provide 3rd party services they don't have to host themselves, they externalize their costs and maximize their profits, like any good amoral for-profit should do by law.
Re: (Score:2)
They also encourage the market share of Amazon Linux, which is Fedora based, but always far enough behind that it's not fully compatible with anything else.
Re: (Score:2)
They also encourage the market share of Amazon Linux, which is Fedora based, ...
Maybe they enabled the EPEL repo by default via an update, and didn't use one of their own mirrors?
Re: (Score:2)
EPEL would be enabled by default on CentOS or Rocky Linux or Alma Linux. Not on RHEL. If a major VM image creator enabled repos such as EPEL, without AWS hosted internal mirrors or proxies, that could increase the burden. It would be interesting to know if the real burden is Fedora itself, or EPEL.
EPEL has been a support problem for years, because the software there gets de;eted as Red Hat policies change. Previous or expired versions are deleted, which makes relying on it for anything important or version
Re: (Score:2)
Why would they do that?
They should want to set up their own mirrors (and default their installs to using them) for the bandwidth savings alone!
Re: (Score:3)
Yeah, our public time server gets absolutely hammered when Amazon users en masse decide to use it.
You'd think for such a large company they'd be hosting common infrastructure (like mirrors) for the betterment of the internet. That little thing they're making *billions* from.
Why would they add a cost with no direct profit benefit? They are a giant behemoth corporation. The motus operandi is "spend as little as possible, take as much as possible, profit as much as possible." Nowhere in there is anything about doing good works for others.
I'd argue it should be a part of the corporate mandate. But I'm not the one that sets the world in motion, so what the hell do I know?
Re: (Score:2)
One wonders what chaos would erupt if you were to check to see if the incoming time request was coming from an Amazon IP, then deliberately offset the time values by a random value. Nothing terribly huge - maybe a few seconds at most.
If
Re: (Score:2)
They do. Why do you think they don't ? https://docs.aws.amazon.com/AW... [amazon.com]
No excuse for Amazon (Score:5, Informative)
Set up your own mirrors and make them preferred for your IP space. It isn't hard, Fedora makes it easy.
Re:No excuse for Amazon (Score:5, Insightful)
First rule of modern business, fuck everyone else.
Re: (Score:2)
Why does Amazon hate capitalism?
Re:No excuse for Amazon (Score:5, Interesting)
This.
If they are hitting the public Fedora mirrors for in-house use, they are suffering right along with everyone else. Set up their own and they get preferred treatment.
Just guessing, but I suspect that someone probably wrote a little script to spin up some VMs. Figuring it was a small project, they fetched straight from the public source. The project grew and they forgot to check their architecture. Or Amazon is full of assholes. Equal probability.
Re: No excuse for Amazon (Score:5, Funny)
Re: (Score:2)
100%
They are essentially holding Fedora hostage. Fedora can always block Amazon IPs (might not be easy), but Amazon will counter by just delisting Fedora as an option from AWS cloud offerings and move them to another distro that they can abuse.
Not sure there are any good moves here.
Maybe asking them nicely?
Re: (Score:3)
Amazon will counter by just delisting Fedora as an option from AWS cloud offerings and move them to another distro that they can abuse.
I'm sorry for the possibly idiotic question, but this would hurt Fedora how? Is Fedora a commercial distribution?
If anything, if I was an Amazon customer and suddenly Fedora disappeared from my list of options, I'd strongly consider moving my business elsewhere.
Re: (Score:2)
They are essentially holding Fedora hostage. Fedora can always block Amazon IPs (might not be easy), but Amazon will counter by just delisting Fedora as an option from AWS cloud offerings and move them to another distro that they can abuse.
I'm not 100% certain, but it seems that the case is:
* "5+ million more EPEL-7 systems" - EPEL is the extra packages, not the main repository.
* "Amazon Linux" is based on Fedora/RHEL/CentOS Stream.
Someone correct me if I'm wrong, but it sure sounds like a ton of Amazon Linux instances enabled the EPEL repository. The Amazon Linux repositories are likely hosted by Amazon already, but the EPEL is extra add on stuff. Dunno what spurred it all of a sudden - maybe some package needed by cloud services got added/m
I swear (Score:1)
CDNs are slower than ever now. I have 300+Mbit connection but they barely break 1/10 that speed
Go buy that IBM license (Score:3, Interesting)
Redhat. It was the most successfull open source company ever. Then IBM, which equals death.
CentOS. Like Redhat only still free. Then RH then IBM then *poof*.
Fedora. Like RH only so new it made the bleeding edge not yet bleed before it bled. *poof*
And now, their choice for where to foist their software is suffering too.
This disease has come full cycle.
Goodbye RH/IBM/COS/FED and hello free and open source software (FOSS).
Downvote me if you think IBM rocks. They have destroyed one third of the Linux ecosystem.
All that is left now is Debian and derivatives (Ubuntu, Mint, etc.) and the sources (Arch, etc.)
We have MORE compilers MORE languages (Rust, e.g.) MORE build systems MORE runtime
systems (not getting into SystemD vs SysV init) and more software.
What will hurt is letting these self-promoters win. Download your FOSS distro and don't pay
these slimeballs anything. Oh yeah, I know what RH contributed, but then I also know IBM
made its value go to zero.
E
P.S. Yeah, I'm on the list of the kernel developer creds. My 15 seconds of fame involved the
BCM driver... but I stood on the shoulders of giants and they worked their asses off and they
don't deserve to have their work poo-pooh'd by suit-wearing asshats with IBM tats on their d****s.
Re: Go buy that IBM license (Score:1)
Re: (Score:2)
I guess the GNU kernel is an option..
Good lord, why? Redhat/IBM haven't yet got their hands on the Linux Kernel. And neither has Mr. Poettering. Users of other Linux distros are safe. For now.
But if you really want to escape the Linux train for some reason, I'd personally recommend you first check the BSDs.
Re: (Score:2)
But if you really want to escape the Linux train for some reason, I'd personally recommend you first check the BSDs.
Though deprecated, Debian GNU/kFreeBSD may be of interest. It replaces the Linux kernel in Debian with the FreeBSD kernel. So if you like the way a Debian distribution looks and works but don't want Linux for some reason, this may fill that need. That said, I don't think unfriendlyLLM knew what they were saying (the kernel is not currently at any risk; many user space things are changing dramatically, so one may want to look at distributions with a more traditional user space - Slackware, Devuan, Free/Open/
Re: (Score:2)
I guess the GNU kernel is an option. But it kind of sort of don't work? I'm a bit scared of it......
As vbdasc said, go BSD before you go HURD. FreeBSD is really nice if you're a Linux user already. Different, but not different enough you feel like you stepped onto another planet the first time you fire it up. I'd recommend that long before heading into the obscurity bucket on moral grounds.
Re: (Score:1)
Downvote me if you think IBM rocks. They have destroyed one third of the Linux ecosystem.
All that is left now is Debian and derivatives (Ubuntu, Mint, etc.) and the sources (Arch, etc.)
Gentoo user here. If Redhat and it's derivatives have actual value, someone will take up the mantle to keep it going. If not, RIP. The closest I ever got to using Redhat was Mandrake Linux. I also think you're glossing over the Rocky and AlmaLinux distros.
Re: (Score:2)
Downvote me if you think IBM rocks. They have destroyed one third of the Linux ecosystem.
IBM once rocked, but this was quarter century ago or more. Not anymore. But anyway, I personally think that Redhat was on its path to destruction even before the IBM love affair.
Re: (Score:2)
Tough decision for me, I want to mod you up and comment, I chose a comment :)
It is not only IBM, but they are the most visible. I think the rot is with the Linux Foundation which is controlled by the likes of IBM, Microsoft, Oracle and other fortune 500 companies. Now they are "raping" the small developers who gave a lot of free labor over the last 30+ years so these companies can pass huge bonuses to their execs and dividends to their stockholders.
I never hear the Foundation calling out hardware vendors
Re: (Score:2)
Linux is about software freedom AND getting shit done.
Re: Go buy that IBM license (Score:1)
People ditching CentOS? (Score:2)
The days of Judgement is coming close.
So to say what many are thinking... (Score:2)
What new onerous changes will IBM make and use this as the excuse for doing so?
Re: (Score:2)
It's difficult to predict since they fired Lennart Pottering and stopped letting him destabilize the kernel with systemd "features".
Apparently it's the repodata (Score:2)
The repodata associated with RPMs is in large, monolithic tarballs which cannot be incrementally updated. So the high churn of Fedora distributions means this data is _constantly_ being refreshed, and has to be completely replicated locally to do any analysis of new package listings whatsoever. It's more of a problem for Fedora because the churn is so large, Just the repodata for Fedora release 40 is nearly half a Gigabyte, and the update of any one package requires a rebuild of that entire repodata. It's n
Re: (Score:3)
You don't have to download the whole file to get to the metadata for a tar file. If the build procedure is defined correctly, you can get to the package metadata by just downloading a few kilobytes of the file.
Re: (Score:2)
>You don't have to download the whole file to get to the metadata for a tar file.
Tar files don't have a centralized index for its constituent files if that's what you mean by metadata, so you effectively do need the whole file. If that's not what you meant, then can you explain?
The metadata for the distro's package manager, dnf/yum. Summaries relating to the packages, dependencies, and their content. Something like that anyway.
Oh comeon! (Score:2)
Re:Oh comeon! (Score:4, Informative)
How hard could it be to cache stuff if you’re s big as AWS?
This is not Amazon (Amazon itself don't use EPEL7), this is some customer (or set of customers) being stupid. And AWS cannot control their customer stupid (but they are willing to accept payment for the fees of customers being stupid).
Re: (Score:3)
Yep, this.
To be fair, AWS could provide a per region mirror of most popular free to their customers with some init script to set that up mostly automatically. They could even charge some small fee after some free tier like they do for other services. Unfortunately each customer needs to do their own setup, which they don't because it means adding a layer of security, which is additional bootstrapping work. It is much simpler to simply have the default behavior to point to the official URLs.
We are going thr
Re: (Score:2)
I've been using RH then CentOS/Fedora since 1995 and I like it.
It was the best distro for a situation when you had to support multiple users who need easy Linux but do not want to be an admin (university/science labs).
Is there any user-friendly alternative to Fedora other than Ubuntu at the moment? I cannot really see one.
I have tried all recent LTS Ubuntus and every case Fedora won for - either due to some stupid installation issues or due to snap...
Re: (Score:2)
I think a lot of folks when to Rocky Linux after Redhat took CentOS and made it part of their dev lifecycle.
https://rockylinux.org/ [rockylinux.org]
Re: (Score:1)
Amazon Linux 2023 does not support EPEL
Re: (Score:2)
I just checked and it does indeed set the fedora repo when using amazon-linux-extras. I always assumed it would use their own repo.
Re: (Score:2)
Duh (Score:4, Informative)
This article weirdly leaves out the WHY. (WTF?) So let me explain:
Amazon has their own version of Linux and when using AWS there are advantages to using Amazon Linux, like built-in commands and libraries that (obviously) integrate with their cloud services.
Previous versions (1 and 2) were more CentOS/RHEL-based. The newer version (AL2023) is more based on Fedora. So of course more updates are pulled from the Fedora-verse.
The number of AWS instances migrating now is because AL2 will be deprecated in about a year. AWS is keen to remind people to upgrade (not a bad thing, of course) and so people are dutifully doing just that.
Re: (Score:2)
What now? You have to go and Pollyanna all this shit. Oh well, this [slashdot.org] just showed up
Clone the repo! Amazon (Score:2)