Preload Drastically Boosts Linux Performance 144
Nemilar writes "Preload is a Linux daemon that stores commonly-used libraries and binaries in memory to speed up access times, similar to the Windows Vista SuperFetch function. This article examines Preload and gives some insight into how much performance is gained for its total resource cost, and discusses basic installation and configuration to get you started."
LiveCDs do this... (Score:4, Informative)
Re: (Score:1)
Re: (Score:2, Informative)
It felt like I was running XP on a machine with the minimum specs and running bloated software. Even the mouse was jerky.
Something's not right there. Puppy's normally responsive on machines that'd be slow with 98SE.
Re: (Score:1)
Re: (Score:3, Informative)
Re:Puppy (Score:1)
I'll say somethings not right there, I loaded Puppy on a 1 Gig USB drive, and tried it on my 7 year old homebuilt, and it was way slower than even XP.
Jerky mouse, freezing up, crashing, it brought back the days of Win98SE.
I thought it was because the USB load was "experimental".
I'll try DSL, or some other lighter weight distro and see what happens.
Re: (Score:2, Funny)
"Sounds like 7 years ago you had no idea how to build a decent computer..."
Sounds like you're quite an arrogant asshole...
Re: (Score:1)
Anyway, back to the article. This is an expecially good idea because these days you can put in 4 GB of PC5300 on up for around $100 from newegg and probably never use half of it. I ran oblivion while burning a DVD and a lot o
Re:LiveCDs do this... (Score:4, Informative)
is toram parameter really faster? (Score:2)
But I always have wondered does linux cache the compressed blocks, or the uncompressed?
Re: (Score:3, Informative)
Even if your particular LiveCD is set to watch for and automatically use swap partitions, a HD is still significantly faster than an optical drive. If you install Linux permanently to
Re: (Score:1)
Re: (Score:2)
dramatically speeding up loading
Not so much in my experience, it just pushes all the loading and *then some* to the boot process. You may only ever hit, say, 40% of the disk content on a livecd if not tuned to your usage, and therefore incur 40% of the content having to be read painfully slow on demand. By virtue of being on demand, every operation that reads new files will be obviously slow the first go around.
Meanwhile, toram and such cause 100% of the disk content to be hit up during boot process, and can make it excruciatingly slow
Re: (Score:2)
As a side note, from my experience even repeated loadings take a long time. If I open firefox it will often hang waiting for the disk to spin up after it's open and again if I close and reopen it. Even using the terminal can cause this. I pretty much hate using LiveCDs if they aren't cached in ram, but often I don't feel like
Re:LiveCDs do this... (Score:4, Insightful)
Several things come to mind when I read the post.
I thought Linux cached used libraries in RAM already, resulting in the appearance that Linux was always using up all my memory but wasn't really. If true, then this basically does what? Guesses what you want to use and loads them for me? Decides what I use a lot and makes certain they never fall out of memory? In both cases, someone is not using my resources in an optimum manner.
If I use the price of my first desktop computer and use that to purchase a new computer at Dell I am moving up 40 times in speed, 2x in architecture buss, 4x in cores, blah blah blah. Compared to the last computer (2006) I purchased I can still get something easily double in performance from that.
So, Not sure what you need in performance, but between the stupid amount of computing power and Linux already doing a lot of in-memory caching there might be a pretty small margin for improvement. But I guess what I really struggle with is the idea of someone/something trying to proactively determine what I'm going to use and then force my computer into a certain behavioural pattern that is making assumptions about my use. Sure, it screams marketing demographics, but even without a PR department for Linux I still don't think there is sufficient need for something like this.
Can someone elaborate on practical reasons where this is something I would really need
Re:LiveCDs do this... (Score:5, Informative)
The thing that eats all your RAM is nothing Linux specific at all, it is your applications asking for more RAM than they are currently going to use. Why should they do such a thing? Well, what do you think memory management would look like if hundreds of apps, daemons and kernel threads ask for two bytes at a time? It'd paint a pretty fragmented picture, so they ask for gobs of pages at a time. Pages seldom touched get swapped out, but still there's an awesome amount of overallocation - thus your memory seems to be 100% allocated 100% of the time.
So, preloading libs that are frequently used is probably going to use your RAM in a more meaningful way unless you already have a problem with constant swapping.
Re:LiveCDs do this... (Score:4, Insightful)
I disagree with both of you. There should not be a need for this. Linux memory management should be closer to optimal for desktop users, but unfortunately the current strategy just doesn't work. It's optimized for servers, paging out interactive apps whenever there's something going on in the background.
In particular, the locatedb daemon makes everything unresponsive because linux caches every file on your file-system it touches, even though it's pretty much guaranteed nobody else needs those files anytime soon. This may be theoretically "optimal" in the general case, but it certainly doesn't feel that way for desktop users. Most desktop users would be more than happy to have background jobs run slower if it didn't impact responsitivity. Also, I believe many people would prefer predictive response-times; it's better for the disk to churn while loading a huge file, instead of it churning everywhere else to page in libraries that have been paged out because the huge file is in memory.
Adding a daemon to predict shared library usage is a step in the wrong direction. Not because it doesn't fix the problem, unfortunately I haven't tried it, but sure, it might even work fantastic. It's a step in the wrong direction because it's a kludge, and not a proper fix for having memory management strategies in the kernel that the users actually want. Unfortunately, fixes to this problem are hard to do, and every time someone tries to do a proper fix, it is debated to death on the kernel mailing list, and then dies slowly as it ages out of tree. For all I know, it's also the right decision, if it should be in-kernel, it should also be *right*. A daemon might be a better place to experiment, and hell, if it solves the problem for 99% of the users, we might not even need to change the current strategy, which is certainly right for servers. After all, we live with kludges other places, such as the X Window System needing to be root and accessing raw kernel memory.
But yeah, memory management is complicated. I doubt you can solve this on a piece of paper. If it works, I'm all for it! Maybe this is a proper kludge?
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
It doesn't help that a lot of desktop I/O hardware is really, really crappy, and bogs the CPU down doing nothing more than disk platter baby-sitting and checksum calculations. But not until after literally starving for memory and disk bandwidth.
The minute you invest in decent hardware, a lot of the I/O scheduling problems go away.
Re: (Score:2)
At the risk of sounding stupid, I'll have to disagree with both you and the parent.
In particular, Linux has long had an over commit memory system. An app can ask for vast quantities of memory and the kernel will happily give it. But that doesn't change the amount of memory being used nor does it kick things out of the cache. Only when that memory is actually used will the kernel *really* allocate it and make room in RAM.
Also, Linux does not cache every file on the system that it touches. locatedb sc
Re: (Score:3, Informative)
Dammit, this is so easy to demonstrate... (Score:3, Interesting)
slashdot won't seem to let me format the way I want, but run free -m on your own. The cached column is the 379 figure.
Note the 379M number. That is the amount of data read from disk and kept in ram. When an application needs to malloc and no completely free memory, yes it will free up those pages (it ideally pi
Re: (Score:2)
The moment you start running something, preload all libraries this process is known to reference? That way it doesn't matter if the application blocks on each library being loaded.
Though I am pulling this out of my ass.
Re: (Score:3, Informative)
I thought Linux cached used libraries in RAM already, resulting in the appearance that Linux was always using up all my memory but wasn't really
Linux uses a disk cache in RAM to keep from re-hitting the HD for often-accessed files. It actually is using up all your memory that hasn't been allocated to an application (which is good, because unused memory is wasted memory), but it will drop some disk cache space when other applications need more memory.
If true, then this basically does what? Guesses what you want to use and loads them for me?
Essentially, yes. On of the bigger bottlenecks to application startup is disk seek/read times. By performing this action in the background before it is requested, you won't hit that lag time. The
Re: (Score:3, Interesting)
Mixed response... (Score:2)
Not foolproof... (Score:2)
Re: (Score:2)
Good point. I generally just keep applications up and move to a new desktop when I need the real estate.
This is a Microsoft solution for Microsoft software design.
Please don't put it in my distro. If I use something large, like OO, I don't mind the start up time because once it's up, I can just leave it up all day long.
ramdisk (Score:2, Insightful)
http://forums.gentoo.org/viewtopic-t-296892.html [gentoo.org]
I never actually tried it, but I might now that I have 4gb ram! A daemon to help automate this process would be welcome, though.
Re: (Score:3, Informative)
But I thought Vista doing it = RAM hogging? (Score:1, Insightful)
Re: (Score:3, Interesting)
Also, Windows' VM system (IMNSHO) has always sucked and will continue to suck; predictive loading of entire bits of software has nothing to do with it.
Re: (Score:1, Informative)
Re: (Score:1, Interesting)
Re: (Score:1, Informative)
Re: (Score:1)
Re:But I thought Vista doing it = RAM hogging? (Score:5, Informative)
Linux handles having lots of RAM a lot better than Windows (XP) does, because of differences in the way the caching system was designed. Linux (and OS X) was intended to run entirely from RAM and use little swap. I've run, say, OpenOffice once, not used it for several weeks, and the next time I start it it loads almost instantly, because it was still sitting in the cache. My machines have 2GB of RAM, with much less than 500MB actually in use - the remaining 1.5GB is being used as disk cache. Swap usage is either zero, or very close. Of course, performance goes to hell if you do something that flushes the disk cache, or if you try using such a system on a machine with 256MB of RAM.
Windows, on the other hand, was designed to run almost entirely from swap, and tends to drop stuff from the disk cache when it's not been used in a while, as well as moving stuff out to swap rather aggressively. That works great if you barely have enough RAM to run the OS, but it's terribly wasteful if you have more than enough RAM. In this case, SuperFetch is actually useful, allowing it to catch up to and actually surpass Linux, by monitoring which files are actually used and making sure they're already in the disk cache.
That's great, although nothing new. Other OSes have had this for years (this Linux implementation dates back to 2005, Mac OS X has had it for ages, and neither implementation was original) - Microsoft were just the first to brand it.
TFA said nothing about Vista's implementation.
I think the primary problem people have with Microsoft's implementations is that they're typically very complicated, and have a tendency to degrade over time. XP is the typical whipping boy for this - none of the self-maintaining performance stuff (prefetching, or the prelinker) actually works for longer than about six months, meaning that an XP installation starts off fast, gradually gets faster, and then rapidly slows down as the system tries to speed itself up.
Re: (Score:1, Troll)
It's also due to linux (and just about everything else including MS Server 2003) correctly supporting the Pentium Pro and later processors. With the 64 bit versions the 2GB ceiling vanishes - but with 32 bit Vista the ceiling is far too close to the floor in my opinion.
Re: (Score:1)
To use some old cliches, if a soccer-sewing sweatshop company donates some money to orphans, would you be all praise? If a
Re: (Score:2)
Old tech (Score:1, Insightful)
Hope... (Score:4, Funny)
Re: (Score:2)
I'm not sure what your seeing unless your running one an a modern computer and the other on a five year old E-machine. Maybe dua
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
*Ofc if you have session restore on kde this wont be true.
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
And I agree
Apple OS X (Score:2)
Re: (Score:2)
Re: (Score:2)
Back when I used to use Macs fairly regularly (back around OS X's release)
Nice but... (Score:5, Informative)
Re:Nice but... (Score:5, Informative)
Re: (Score:2)
I'm guessing you tried doing something silly like loading your whole
Sure, alltray moves the load to RAM, but at the cost of entire applications. The point of preload is that it just caches the most commonly used files.
Sure, preload caches the commonly used files, but what if your openoffice
Blogspam (Score:5, Informative)
Re: (Score:2, Insightful)
For those of us with our own blogs, how on earth do you get readers without tooting your own horn?
Re:Blogspam (Score:5, Insightful)
Re: (Score:2)
Well... (Score:2)
Similar? (Score:5, Funny)
Vista users respond positively toward the speed boost everytime we "Reload" their Vista. The downtime and data lost as a result of "Reload" might irritate some disgruntled users, but most of them enjoy the free break at the expense of the company.
Nothing in those Linus thingy could beat that user satisfaction. I might be bias though.
Not sure about the gain (Score:4, Insightful)
Difference with readahead? (Score:4, Interesting)
Does anyone knows the difference between the two projects? Does preload have a better algorithm for selecting the files to read? Does it also use this special syscall?
Re: (Score:2)
Oh dear, oh dear, oh dear (Score:2)
Readahead Vs Preload (Score:2)
Has anyone got any insight into what's better, or if they will peacefully work together? I'd prefer faster app times over faster boot times as I hardly ever reboot.
eeepc + preload = less waiting, more performance! (Score:3, Informative)
Re:eeepc + preload = less waiting, more performanc (Score:1)
COOL (Score:1)
I guess being smarter is more beneficial than I had thought
Re:What, what? (Score:5, Informative)
preload 0.2 release: 2005-09-01
And it was there before as it was packaged in Gentoo (back when it was still popular) and Suse 9.3
Re: (Score:2)
Re: (Score:2)
Re:What, what? (Score:5, Informative)
Re:What, what? (Score:5, Funny)
*Preloading commonly used data, libraries, and binaries...
gcc OK
make OK
libc-dev OK
emerge OK
kernel-src OK
*Preload done, 3827K of USE Flags, 2TB of source code, and one compiler, and firefox to surf forums.gentoo.org for better use flags while you compile loaded into main memory
Re: (Score:1)
Wake me up when it reaches 1.0, it's just a pre-beta release for now.
Re: (Score:1, Insightful)
Re:What, what? (Score:5, Funny)
This reminds me of a geek girlfriend I had... she told me she was 29.9.9.12.1 years old. But when I met her in real life, i was suprised she had a daughter... 17.1.25 years old!
Re:What, what? (Score:5, Insightful)
Just let it go. This pissing match over innovation serves no useful purpose.
Re:What, what? (Score:5, Insightful)
invention = first time to do something ever
Note how MS is always careful to point out they innovate.
*flush*
Re: (Score:2)
invention = first time to do something ever
Re:What, what? (Score:5, Funny)
Fulfill your fantasy (Score:1, Funny)
Re: (Score:1)
I call that version 1.0, and preload isn't there yet.
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
it's an old, but good idea.
The first thing that went in though was usually
Re: (Score:3, Interesting)
The Amiga one of the same era was very good. You had a recoverable RAM disk, which functioned the same as a standard RAM disk, but would maintain its contents on restart. That meant reboots were lightning quick, and any data you stored in the RRAM disk was still there.
Shame we haven't got back to that level of functionality.
Re: (Score:3, Interesting)
Except when you were coding something that ran away in memory, corrupted the RAD (the name of the recoverable RAM disk), destroying everything you hadn't saved for an hour or so. :)
(The Amiga didn't have an MMU originally, and even when they got it, the OS didn't support memory protection due to the shared message passing)
Re:What, what? (Score:5, Interesting)
My computer is MUCH faster now that SuperFetch is disabled. Like night and day.
Re:What, what? (Score:5, Insightful)
Re:What, what? (Score:4, Informative)
Not really. Caching policies like this have been around for longer than Windows has even existed. Most of the things that Linux "adopts" from Windows or Macintosh originally came from UNIX or mainframes. Even in 2008, there is hardly an original idea in any of those operating systems. And preload itself is, of course, older than Vista.
You can be mad at Vista for a number of reasons, but SuperFetch is not one of them - I have noticed a decent speed improvement because of it, and look forward to having something similar in Linux.
It's not clear to me why this should be a separate user process; what it's doing is simple enough that whatever is doing can be done directly by the kernel. In fact, I wouldn't be surprised if you could get the same speedups by simply tuning a couple of kernel parameters.
Re; Separate Process (Score:3, Insightful)
Re: (Score:3, Insightful)
The kernel can keep track of files in whatever way it likes, including paths. But I don't see a problem with using inode numbers anyway. I think "preload" is trying a little too hard.
Moreover the act of caching the file is easily accomplished by a low-priority user-space task which speculatively reads the files
These "low-priority user-space tasks" are spreading like a cancer. Linux
Re: (Score:2)
I don't know what Vista you're using, but my Vista is noticably slower than XP for everything I do (on the same computer I had XP on before) even after turning off all fancy graphics and many unused services and background apps. Maybe SuperFetch is helping and it would be even slower without it, I can't really say, all I know as a user is that the system is much slower, and it's beefy hardware I'm talking about too.
Anyway, I would rather OS developers focus on making the system faster, optimising existing
Re:Security Implications? (Score:5, Funny)
You mean like losing the data after a few hours of no power?
Re:Security Implications? (Score:4, Funny)
Re: (Score:1)
Last I heard it was a big deal when someone published that RAM keeps its data for a few minutes of no power..
Re: (Score:2)
Re: (Score:1)