The Linux Kernel Archives 154
Jeremy Andrews writes "KernelTrap offers an interesting look at the history behind the Linux Kernel Archives, home of the Linux kernel. They start from the beginning in 1997, when kernel.org ran on a generic "white box PC" using a shared T1, to the present where it runs on multiple quad Opterons each with 24 gigabytes of RAM, 10 terabytes of disk space, and a gigabit link to the internet. Much of the article is based on an interview with Peter Anvin, also including quotes from Linus Torvalds, Paul Vixie of Internet Systems Consortium, Inc who donates the bandwidth, and Matt Taggart of Hewlett-Packard who donated the hardware."
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
If we're savvy enough to need a kernel tarball, we're savvy enough to run AdBlock [mozdev.org], no?
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
I know where my wife works they won't let her use USB ports, which is a real bind when it comes to taking files to work sometimes.
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2, Interesting)
"When asked about viewing the actual access logs, Peter explained that although they do occasionally get requests from various sorts of researchers, they generally don't make them available for privacy reasons."
Perhaps the anonymous logs should be sold to pay for some of this juicy bandwidth they're c
Re:Bandwidth and Slashdot Effect on kernel.org (Score:2)
I don't think that the linked article (last time i checked it wasn't) is slashdotted.
So, what they need to do... (Score:3, Funny)
Hmm... (Score:2, Funny)
Re:Hmm... (Score:1, Funny)
We all know that.
Re:Hmm... (Score:1, Offtopic)
Re:Cool (Score:2, Funny)
Nice to see folks getting credit... (Score:4, Interesting)
We were having bandwidth limitations on RubyForge [rubyforge.org]; it was getting up to 80 GB per month [rubyforge.org] at the end of 2004. Mirroring out releases helped get usage back down to 15 GB per month. Many thanks to our mirror [rubyforge.org] providers!
Re:Nice to see folks getting credit... (Score:1)
Re:Nice to see folks getting credit... (Score:2)
Don't get me wrong but that sounds pretty low (nowadays!).
You can get a cheap dedicated server for $30/month with 100GB traffic incl.
A website that I babysit sends 1.5TB/month over the wire.
We pay well under $1000 for 1/2 rack, power *and* traffic.
Re:Nice to see folks getting credit... (Score:1)
Yup, although now with the mirrors in place the system is doing about 200 GB per month.
Has the /. effect met its match? (Score:2, Funny)
Re:Has the /. effect met its match? (Score:2)
Meh, it sounds like a lot until you consider the number of people with broadband. For example, 2000 people with 1Mbit connections could saturate a pair of gigabit links.
Re:Has the /. effect met its match? (Score:1)
With a 1 MBit/s link, you can download 1 GB in roughly 2.5 hours. Thus, 2000 people with 1Mbit connections could saturate a pair of gigabit links for maybe 5 hours when downloading FC.
(Sidenote: I have no idea how large FC 4 is actually going to be.)
The article says they expect the links to be saturated for about 3 to 4 days.
Only 1 Gb/sec? no wonder I can't get new releases (Score:2, Funny)
Mod +5 funny -5 irreverant
Yes but... (Score:5, Funny)
Re:Yes but... (Score:3, Informative)
Yup.
Re:Yes but... (Score:2)
Seriously? I thought they ran Debian. Did they change? If so, it's a bit of a kick in the face for Debian. And the standard kernel from Redhat? Why?
Re:Yes but... (Score:1)
Re:Yes but... (Score:2)
Somebody's smoking a bit too much weed, if they think that RH is easier to upgrade than Debian.
IIRC, though, I think it's simply because the admins know/like/use RH.
Re:Yes but... (Score:2)
Re:Yes but... (Score:2)
RPMs tend to be much less granular than DEBs. So, you need to upgrade or newly install more stuff, even if you don't need/want it..
Interesting they use Fedora and not RH EL (Score:4, Funny)
a) kernel.org doesn't think the 'enterprise readiness' of RH Enterprise Linux is that great, (vs what Fedora offers) even in what should be considered one of the most mission-critical sites in the Linux ecosystem (or that the difference with Fedora is worth paying for)
b) No one at RH is bright enough to be embarassed by this and offer kernel.org some free licenses...
Re:Yes but... (Score:3, Funny)
Re:Yes but... (Score:1, Redundant)
Re:Yes but... (Score:3, Informative)
Slashdot history! (Score:5, Interesting)
This was a great article! I can attest the there is quite a difference with the new hardware, I got a 500KBps download last night while downloading rc3-mm2.
Can we please have the same kind of article about slashdot hardware?
Re:Slashdot history! (Score:5, Funny)
Re:Slashdot history! (Score:2)
Re:Slashdot history! (Score:2)
Re:Slashdot history! (Score:2)
The Multia is Alpha, albeit rather anemic even back then.
http://www.obsolyte.com/dec/multia/ [obsolyte.com]
Re:Slashdot history! (Score:2)
Last I checked a Multia [obsolyte.com] wasn't an Alpha [majix.org] Server [majix.org]
Re:Slashdot history! (Score:2)
Point taken.
Last I checked a Multia wasn't an Alpha Server
Or this one? http://h18002.www1.hp.com/alphaserver/gs320/ [hp.com] We have a couple (running OpenVMS, of course), and they're pretty sweet.
Interesting quote (Score:5, Interesting)
I wonder what would have happened with Transmeta and Linux if they had used the Transmeta domain to host the kernel archives. Would IBM have gotten involved with Linux? Would SCO have sued Transmeta instead of IBM? Would Linus have left Transmeta?
Alternative Resources (Score:1)
Oh my god, it's a diesel!
Slashdotting? (Score:4, Funny)
Do I smell a challenge?
Re:Slashdotting? (Score:2)
After all, 4 years ago, a pimped-out machine had less than 20gig of hd space, 128meg of ram, and 500-800 mhz. DVD burner? forget it, not at $3000 each.
Nowadays even entry-level boxes are better than that.
Re:Slashdotting? (Score:1)
A quick check at HP.com shows a ProLiant DL585 with 2 800-series dual-core Opteron processors starts at $12K. Add in 24GB of RAM and 10TB (two MSA30 each with 14 drives) and you're north of $50K in a hurry...
Re:Slashdotting? (Score:2)
How much would a terrabyte of storage have cost you 4 years ago? You would have needed to stack 50 x 20gig drives, plus the cases, controllers, and power supplies, etc. Today? Under a grand, as an off-the-shelf item. Terabyte drives will be in people's boxes by Christmas of next year, if not this year. You can buy 2 TB of storage today for less than what I paid for an 80 meg hd 15 years ago.
Same thing for cpus. You'll have a hard time
Re:Slashdotting? (Score:2)
it cost around 3 grand in total . Today i could build a system that with an 80GB HDD , an athlon-xp 2500+ , 512MB DDR RAM a better graphics card all for around 1/15th the price , so what cost me then 3000 will today (for something alot better) cost 150-200 (self built
Re:Slashdotting? (Score:2)
Multiple 4 way opterons? 24 GB memory? At least 6 years way, assuming that the average home machine does not simply drop in cost at the expense of specs.
Re:Slashdotting? (Score:2)
1.6GHz AMD CPU, 128meg RAM, CDRW drive, and a DVD drive. Cost about $1500.
Except for the RAM, it's actually quite comparable to today's entry level
boxes. Still runs like a champ, too.
Re:Slashdotting? (Score:2)
It's amazing how quickly we take for granted what was cutting-edge just a few years ago.
Re:Slashdotting? (Score:2)
Re:Slashdotting? (Score:2)
It was way overpriced.
Re:Slashdotting? (Score:2)
For what the average computer cost back then (2-3k), you can get a decently pimped-out box today. Not only have prices come down, the barrier has been lowered as well. And its just going to get better.
Look at how quickly 64-bit cpus went from esoteric to "how many do
Re:Slashdotting? (Score:2)
Even the bargain-basement celerons hadn't broken 1ghz by August.
Re:Slashdotting? (Score:2)
Awesome (Score:2)
* I strongly suspect this not to be true.
Re:Awesome (Score:3, Interesting)
Re:Awesome (Score:1)
--
lds
Re:Awesome (Score:2)
How so? That thing is designed for serious backbone duty, and should thus be built to handle insane amounts of data. That thing makes a business-class router on a GigE line look like a Linksys on a DSL line (not knocking Linksys, I love my WRTs).
Re:Awesome (Score:1)
I'm very excited by having gotten to touch an engineering model of the CRS-1!
--
Phil
load average 1024??!!! (Score:5, Interesting)
Can you even get the server to TELL you what the load is when it's that high?? That's INSANE!
Re:load average 1024??!!! (Score:3, Insightful)
sure;
The proccessors could actualy be doing very little other then waiting on the disks.. I'd be more interested in output from vmstat.
Re:load average 1024??!!! (Score:2)
Would definately look funny. The scale alone would be good for some jawdropping at second glance...
Re:load average 1024??!!! (Score:2)
Re:load average 1024??!!! (Score:1)
Would you care to elaborate on that?
AFAIK load avg is avg number of processes waiting for execution and cpu load can be measured as how much non-idle work the processor is doing.
load average includes IO (Score:2)
processes waiting for in uninterruptable sleep.
That would be disk IO, mostly.
Re:load average 1024??!!! (Score:2)
Of course you can. "Normal people" often get high load values when they run out of mem and the box starts swapping and you can't control the box. Try running some thousands of "cp
Re:load average 1024??!!! (Score:2)
Uptime usually works well into 2k, but "ps" and even ls in the
Re:load average 1024??!!! (Score:2)
Someone needs to check up on what the load average actually means :).
So then... (Score:2)
As we have some figures for the numbers of machines in the early days and surely we have the traffic figures for then as well...
We should be able to make a reasonable guess at the number of machines out there with Linux on them...
Hat's off to HP (Score:5, Insightful)
Re:Hat's off to HP (Score:1)
BTW-- I can attest that the dual Opteron DL145's from HP [hp.com] are rocking boxes for Linux.
Re:Hat's off to HP (Score:3, Informative)
I'll second this. Their support for many things is outstanding. HP really does seem to care about the end users.
I have seen them support networking equipment purchased second (third?) hand off of eBay without asking "Where did you buy it?" or anything like that.
In another instance, one of their supprot people searched through several dozen models of laptops and found one in a different product line that had compatible drivers so that an OS besides Windows XP could be used. No other company has come clos
It's About Marketing, DUH (Score:1)
I believe that was the reaction HP's marketing department also expected. Admittedly, providing the hardware was a very nice gesture, but in reality, it's a brilliant marketing move.
Furthermore, I hope you will take other factors and datapoints into consideration when someone asks you for your advice, though. The servers donated were relatively high-end -
Re:What? (Score:2)
Re:What? (Score:2)
1970???? Try 1829 (Andrew Jackson).
And don't forget that Kennedy put his brother in as Attorney General.
Slashdot effect minor compared to .tar downloads (Score:4, Interesting)
Re:Slashdot effect minor compared to .tar download (Score:2)
Bittorrent as a large-scale software distribution? (Score:1)
However, in this post [iu.edu] from hpa, it looks like the tools are not ready [yahoo.com].
Re:Bittorrent as a large-scale software distributi (Score:2)
Must
noatime interesting (Score:5, Interesting)
Re:noatime interesting (Score:2)
Re:noatime interesting (Score:2)
Do any Linux distros disable noatime by default? If someone cares about noatime, let them manually enable it. Currently, we are all paying the price for those users!
Re:noatime interesting (Score:2)
Re:noatime interesting (Score:2, Informative)
Re:noatime interesting (Score:2)
Re:noatime interesting (Score:2)
(atime hurts performance is because of the extra drag from the collapsing quantum wavefunctions.)
I worked at Globix (Score:1)
Obligatory... (Score:1)
Can you image a cluster of these...no wait...
That's what rsync does (Score:4, Interesting)
I don't have occasion to use rsync, and I'm not too familiar with its design, but I think it synchs directories by checksumming the files in them to see if they differ. So Peter is saying above that the server's bottleneck is checksumming. I would think that on a server like this, checksums could be cached - why checksum a stable file more than once? Once you have a checksum for linux-2.6.0.tar.bz2, why calculate it again?
This would require a bit of bookkeeping when files change, but wouldn't it be worth it on such a busy system? (Or am I confused?)
Re:That's what rsync does (Score:2)
Re:That's what rsync does (Score:4, Informative)
Re:That's what rsync does (Score:2)
Re:That's what rsync does (Score:2)
rsync is more than a simple checksum (Score:1)
What's different about rsync [anu.edu.au] is that it does not ordinarily use a single file checksum (and therefore copy whole files if changed). Instead, to save bandwidth, it uses a more sophisticated system to ensure that only changed parts of a file are transmitted - and it detects changed parts by comparing (many) checksums, I believe. The report [anu.edu.au]sums it up like this:
CVS + p2p (Score:1)
Re:here's an idea (Score:5, Interesting)
This was suggested. The kernel.org people didn't seem to have interest in it. Those light http servers are probably good for lots of small static html files. kernel.org is not like that - it needs to serve + 20 MB files and CD ISOs. Your benchmarks don't measure that. I can bet the kernel.org people knows what they're using and they'd have switched if it'd be really useful.
Re:here's an idea (Score:5, Informative)
Sure, the startup cost of the transaction is higher than for a lightweight HTTP server, but the startup cost of the transaction isn't a big deal for us, and we appreciate the flexibility that Apache offers.
Re:here's an idea (Score:2)
Doesn't apache do some additional stuff per request that others don't?
Things like check whether authentication should be requested and the like?
I might be mistaken but if not - don't these things amount to a significant overhead, esp. when talking.. uh.. "lots" of requests per second?
Re:fine but... (Score:1)
Does it run MSDOS?