Tuning Linux VM swapping

Lank writes "Kernel developers started discussing the pros and cons of swapping to disk on the Linux Kernel mailing list. KernelTrap has coverage of the story on their homepage. Andrew Morton comments, 'My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful.' Personally, I just try to keep my memory usage below the physical memory in my machine, but I guess that's not always possible..."

It doesn't work that way

[Pharmboy]

Personally, I just try to keep my memory usage below the physical memory in my machine, but I guess that's not always possible..."

I keep my memory usage much below the total ram on the servers, but in real life, the machine still swaps. This is because even tho the machine NEVER needs more ram than is available at any given time, over a period of days, it will use more than the available ram. It caches out the old data that was used 12 hours ago.

Unless you reboot every day (as in a client machine) you will use swap on just about any machine. Using swap is not bad. Using swap for a currently running application is not so good. This isn't a bug, its a feature. Reading data from swap after it has been accessed is still faster than reading new data from the drives, especially if its a network drive.

What's wrong with many resident pages?

[YetAnotherName]

You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine...

Why not? BloatyApp, if it's that bloaty is probably an object oriented program with template instantiation (or is by Micro$oft); these programs are notoriously huge, but also have notoriously poor locality of reference. The user will get better perceived response if you can keep more of BloatyApp resident.

If there's space in memory, I don't see the point of pre-emptively ejecting as many LRU pages of BloatyApp. (Of course, I haven't RTFA, but this is /. so you're not supposed to!)

Re:How fast is swapping really these days?

[Moderation abuser]

Well, disk access speed, say 5ms. RAM access speed 10ns so RAM is approx half a million times faster than disk.

Re:God no...

[petabyte]

Actually, you can change it on the fly with /proc/sys/vm/swappiness Increasing the number will increase the agressiveness of the swapout. Mr. Morton runs with his set at 100 (the max). 0, I believe would turn swap.

My kernel has autoswappiness enabled so it figures out the number on its own. I'm running at 64 ATM on a 256 Meg system (ram donations accepted) :).

Re:God no...

[kinema]

All you need to do is: "echo 0 > /proc/sys/vm/swappiness" and the VM will do it's best to keep from swapping pages to disk.

Re:God no...

[Elm Tree]

If I recall correctly it's runtime tuneable. So us power users with 1+ gig can tune swapping down, and desktop distros can tune it up. That's my favorite part about linux, I can just
cat 0 > /proc/sys/vm/swappiness
and I have instant control as to the performance of my machine. In fact... I could even write wrappers to specific programs so that they can tune the system's swappiness to better suit them. I.E. Programs that use huge ammounts of memory, less swappy, programs with repetetive disk access more swappy... Is swappy even a word?

Re:The big issue

[turgid]

dare I suggest using common sub libraries within libraries, that is people like KDE and GTK get thier heads together and say "are thier functions we include in our libraies that could just as well be linked to an underlying library?"

Well, you see, KDE is written in C++. GTK is C. C++ stuff does not play well with different version of the same compiler let alone different compilers or even different languages.

In theory you're only "supposed" to use either GNOME or KDE and therefore only have one set of libraries installed. In practice you often want to use applications from both systems, and end up installing them both.
The problem is political and religious as well as technical.

My vote....

[tsmithnj]

is to do something like AIX does, where I can use "vmtune" to customize the percentages of memory I devote (hard or soft limit) to filesystem pages or computational pages. This way I can tune for my Bloatware, tune for file copying a la XP, or tune for my DBMS, whatever suits me.... The developers could take it one step further and provide a simple, understandable (as opposed to AIX's) interface for configuration......

Re:The big issue

[alex_tibbles]

In fact, one big problem is the way that the loader performs relocations on C++ libraries. Google it. It's why KDE apps take a few seconds to load (and used to be even worse). IIRC the main problem is that many objects (function, variable etc.) need to be copied into the address space of each application using it, so that the sharing never happens in practise.

Re:How fast is swapping really these days?

[drinkypoo]

Hard drives have seek time and maximum bandwidth. Memory has latency and maximum bandwidth. PC100 SDRAM (for example) has about 10ns latency. As you say, hard drives have much longer seek times than SDRAM has latency; usually between 9 and 20 ms. Hard drives typically transfer between 10 and 30 MB/sec; PC100 SDRAM which is 64 bits (8 bytes) wide has 100 MHz x 8 Bytes = 800 MB/s transfer (peak theoretical.) According to SiSoftware Sandra 2004 Pro, a Via KT133 chipset machine with a fairly fast AMD processor develops 760MB/sec peak transfer. (For comparison, my Athlon XP 2500+ system with dual channel DDR333 and a bunch of crap running in the background cranks out 2309 MB/s on top of whatever memory access is already occurring. It uses an nForce2 chipset.)

Re:How fast is swapping really these days?

[happyfrogcow]

Locality of reference... that's why we should all be using profiling tools. I'd like to take this opportunity to promote Valgrind [kde.org]. Ddon't fear the kde URL, it runs in text mode and the requirements page has no mention of kde libs or even X Windows.

It has awesome memory checking, cache profiling and heap profiling.

I'm just a satisfied user, i have no relationship with the Valgrind developers.

Re:God no...

[The Spoonman]

Right-click My Computer -> Advanced -> Performance -> Advanced -> Memory Usage. Set to Programs. Now, click Change under Virtual Memory. Set your cache size small. For 1G of RAM, you prolly don't need a biggun. I usually set it to 100M for Inital and Max and then up it based on how often the machine swaps.

Then, make the following changes to the registry:

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\ClearPageFileAtShutdown, set to 1. I don't shut my machine down very often, but occasionally XP will increase the size of the pagefile if it absolutely needs to depending on circumstances. This forces it back to the size you want it when you restart.

HKLM\System\CurrentControlSet\Control\FileSystem\N tfsDisable8dot3NameCreation, set to 1 ONLY IF YOU USE NO 16-BIT APPS ON YOUR MACHINE. Speeds up writes.

HKLM\System\CurrentControlSet\Control\FileSystem\N tfsDisableLastAccessUpdate, set to 1 if you don't care when files are accessed. This is rarely needed, and the setting speeds up writes.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Contro l\Session Manager\Memory Management\IoPageLockLimit. Little more complex:

Set to 4096 if you've got more than 32M RAM

Set to 8192 if you've got more than 64M RAM

16384, 128M

32768, 160M

65536, 256M

131072, 512M

This changes the maximum number of bytes that can be locked for I/O operations. The default is 512Kb. While the above are the recommendations, I've found stepping down one level to provide the most performance for my needs, YMMV. (For example, I have 256M, but I set my IO limit to 32768.)

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\DisablePagingExecutive, Set to 1 to disable paging of the kernel.

There, that wasn't so hard, was it? For those who want to flame that statement, keep in mind, that the information above is easier to find than some of the tuning suggestions I've heard for Linux. I've used Linux for 10 years, and only today heard about /proc/sys/vm/swappiness. Oh, and all of the above apply from at least NT4+.

Re:God no...

[5E-0W2]

There's no "copy file" system call, it's just a read and then write loop. However in many cases, the program doing the copying should use O_DIRECT or posix_fadvise DONTNEED.

Re:God no...

[Gilk180]

Duh.

Who thinks that writing a page to disk immediately wipes the ram it came from. This is how things really do work.

The problem is whether this paged but still valid ram should be given to other processes. I would say it is a tuning thing. There is no perfect method that will be best for all situations.

Whoever modded this insightful needs to do a little reading about operating systems.

Two copies ARE kept

[Anonymous Coward]

In fact, this is already done that way since the 2.0 days IIRC (it may be 2.2).
And also the other way round - if a page is paged in (into ram), it is not immediately deleted from swap, so if that page is NOT MODIFIED it doesn't need paged out again if memory is required.

XP paging registry settings

[ScrappyLaptop]

Most of what you are looking for is here:

"DisablePagingExecutive"=dword:00000001

"LargeSystemCache"=dword:00000001

The combo will keep your kernel in RAM (DisablePagingExecutive)and enforce a minimum reserve (4MB) amount of memory for it (LargeSystemCache), although that amount can grow dynamically. The LargeSystemCache may cause "Delayed Write Failed" errors, if it does, reboot in SafeMode and undo the damage.

More goodies here, many adjustments that can affect how your sytem divides between swap, cache and currently-running-apps:

HKLM/System/CurrentControlSet/Control/Session Manager/Memory Management

"ClearPageFileAtShutdown" "DisablePagingExecutive" "IoPageLockLimit" "LargeSystemCache" "NonPagedPoolQuota" "NonPagedPoolSize" "PagedPoolQuota" "PagedPoolSize" "PagingFiles" "SecondLevelDataCache" "SystemPages" "PhysicalAddressExtension" "WriteWatch"

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Se ssion Manager\Memory Management\PrefetchParameters

"VideoInitTime" "AppLaunchMaxNumPages" "AppLaunchMaxNumSections" "AppLaunchTimerPeriod" "BootMaxNumPages" "BootMaxNumSections" "BootTimerPeriod" "MaxNumActiveTraces" "MaxNumSavedTraces" "RootDirPath" "HostingAppList" "EnablePrefetcher"

Re:God no...

[jbayes]

I've used Linux for 10 years, and only today heard about /proc/sys/vm/swappiness.

That's 'cause "swappiness" was just added in the 2.5 series. It hasn't been around that long.

Re:Stupid Windows Kernal Swapping

[Dun Malg]

In the end, I have never ever seen a Windows-system without a partially swapped kernel, even with tons of free RAM available. This is just plain stupid, or is there some sort of "smart" explanation for this?

When you have a bunch of lazy, slacker, multi-megabyte services running in the background, waiting for that once-in-a-blue-moon event that requires their help (yes, I'm talking about YOU spoolsv.exe, you 3.98MB hog!), you might as well shove them into the swap file. Windows can end up with an unGODLY amount of oversized crap just sitting aorund waiting for something to happen. Take a look at the "processes" tab on the task manager-- it's insane! I figure the NT kernel is about the same: a bunch of modular pieces, two thirds of which are used maybe once at startup and then paged out. I guess I understand why they did it-- if I had a 57MB kernel image lolling around memory, I'd look for ways to swap out some too.

Re:Stupid Windows Kernal Swapping

[mr_jrt]

If you have an NT-based system then try this article [microsoft.com] and see if it helps. Works great for me on XP :)

Re:God no...

[Anonymous Coward]

man 2 sendfile

Re:God no...

[Anonymous Coward]

Yes, a program can use the mlock(2) and munlock(2) calls to disable swapping of its memory on a page granularity (usually a multiple of 4096 bytes).

No swap involved

[Handyman]

The funny thing about the whole thread is this: it eventually turned out that the guy who originally complained about swapping didn't have any actual swapping going on at all. What is usually happening is that pages from an *executable* are being dropped, and people incorrectly refer to this as "swapping". In Linux, pages from an executable aren't "swapped out" but are simply dropped and read back from the original executable as needed. The trouble is that once you then access some of those pages again -- say, when you're exercising the repaint code from some bloatware app for the first time in a while, because it's been in the background for a while -- they have to be faulted in one by one, and that takes a lot of time per page.